Semi-automatic tools enhancements (Client-side points minimizer) #3450

bsekachev · 2021-07-22T08:02:28Z

Motivation and context

Related #2936
Related #2515

New setting:

On the fly working:

How has this been tested?

Manual testing

Checklist

I submit my changes into the develop branch
I have added description of my changes into CHANGELOG file
I have updated the documentation accordingly
I have added tests to cover my changes
I have linked related issues (read github docs)
I have increased versions of npm packages if it is necessary (cvat-canvas,
cvat-core, cvat-data and cvat-ui)

License

I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.
I have updated the license header for each file (see an example below)

# Copyright (C) 2021 Intel Corporation
#
# SPDX-License-Identifier: MIT

bsekachev · 2021-07-22T09:39:39Z

@TOsmanov @aschernov

This is a feature you requested, please provide your feedback.
Also, need to update documentation when the PR is merged.

@dvkruchinin
Could you prepare a test for the issue? At least we added a new setting (default accuracy) and need to check this setting can be saved and restored. It also should work for OpenCV intelligent scissors (I am not sure if we have a written test for the feature)

aschernov · 2021-07-22T09:47:03Z

@bsekachev thank you, we'll take a look.

dvkruchinin · 2021-07-22T10:09:26Z

@dvkruchinin
Could you prepare a test for the issue? At least we added a new setting (default accuracy) and need to check this setting can be saved and restored. It also should work for OpenCV intelligent scissors (I am not sure if we have a written test for the feature)

Sure. I'll prepare a test. Unfortunately, it will not be possible to do this for OpenCV yet, since the problem of losing the Content-Length header is not solved if OpenCV is loaded from the browser under the control of Cypress.

bsekachev · 2021-07-23T07:34:59Z

@nmanovic Here you can see errors in tests we discussed last meeting

nmanovic · 2021-07-23T14:10:25Z

@dvkruchinin , could you please look at problems with tests in the PR?

nmanovic · 2021-07-23T14:46:53Z

@bsekachev ,

For an image I got less than 3 points. Probably need to restrict that.

From my point of view the "approximation accuracy" should be visible only after we have a polygon. Now it is visible as soon as we activate an interactor.
Personally I will rename "approximation accuracy" into "Number of points" with 2 visible values: low and high.
Also for small epsilon I would prefer to have more values: 0.5, 1, 1.25 1.5, 2, 3, 4. In some cases the difference between number of points is too big.

...c/components/annotation-page/standard-workspace/controls-side-bar/approximation-accuracy.tsx

nmanovic · 2021-07-23T14:54:36Z

@bsekachev , it will be great to apply the approxPolyDP function for existing polygons somehow in the future. For example, how to reduce the number of points for Mask RCNN?

nmanovic · 2021-07-23T14:55:54Z

@bsekachev , looks like a natural idea to have "done" button directly on the slider. It is more visible. But I'm not sure.

dvkruchinin · 2021-07-26T08:10:02Z

@dvkruchinin , could you please look at problems with tests in the PR?

I can't reproduce the problem locally or in a fork. This could probably be related to internal problems of the github actions virtual environment.
I will soon prepare the necessary test and look at the passing of tests in the current repository.

bsekachev · 2021-07-26T08:18:12Z

For an image I got less than 3 points. Probably need to restrict that.

Looks like OpenCV can return it after approximation (event despite of "closed" flag equals to true).

I am not sure we should strict it because in this case we can only return original points, but it is not clear for a user (users shift slider to less points, but unexpectedly they get all the points).
But I would suggest to mark the shape as RED on the canvas. It's a kind of a signal that something is wrong with the shape. Even if a user submits such the shape, they get an exception from cvat-core (not enough points).

Personally I will rename "approximation accuracy" into "Number of points" with 2 visible values: low and high.
Also for small epsilon I would prefer to have more values: 0.5, 1, 1.25 1.5, 2, 3, 4. In some cases the difference between number of points is too big.

Done

bsekachev · 2021-07-26T08:24:21Z

@bsekachev , looks like a natural idea to have "done" button directly on the slider. It is more visible. But I'm not sure.
@bsekachev , it will be great to apply the approxPolyDP function for existing polygons somehow in the future. For example, how to reduce the number of points for Mask RCNN?

These ideas are bound each other, because in current implementation "Done" button exists when drawing all polyshapes, but the slider does not.

Right now I would suggest leaving as is beacuse I don't have ideas how it should be implemented in UI for all the cases (exactly when it should appear and disappear, how it should work with drawing (when we need Done button, but a user is not assumed to create extra points to be minimized), or with editing (when we could need minimize points after detectors, but done button is not supposed to be used), etc.

bsekachev · 2021-07-26T08:31:27Z

From my point of view the "approximation accuracy" should be visible only after we have a polygon. Now it is visible as soon as we activate an interactor.

@nmanovic Done

nmanovic · 2021-07-27T07:46:12Z

Every interaction with DEXTR leads to the problem below. Probably need to check the threshold value.

Uncaught (in promise) RangeError: offset is out of bounds
    at Float32Array.set (<anonymous>)
    at Object.eval [as matFromArray] (eval at initialize (cvat-ui.b532816919397967c04d.min.js:435), <anonymous>:32:9059162)
    at Object.approxPoly (cvat-ui.b532816919397967c04d.min.js:435)
    at sH.approximateResponsePoints (cvat-ui.b532816919397967c04d.min.js:435)
    at sH.componentDidUpdate (cvat-ui.b532816919397967c04d.min.js:435)
    at os (cvat-ui.b532816919397967c04d.min.js:352)
    at pl (cvat-ui.b532816919397967c04d.min.js:352)
    at t.unstable_runWithPriority (cvat-ui.b532816919397967c04d.min.js:360)
    at Ba (cvat-ui.b532816919397967c04d.min.js:352)
    at fl (cvat-ui.b532816919397967c04d.min.js:352)

nmanovic · 2021-07-27T07:50:08Z

...c/components/annotation-page/standard-workspace/controls-side-bar/approximation-accuracy.tsx

+    if (approxPolyMaxDistance > 0) {
+        if (approxPolyMaxDistance <= 8) {
+            // −2.75x+7y+1=0 linear made from two points (1; 0.25) and (8; 3)
+            threshold = (2.75 * approxPolyMaxDistance - 1) / 7;


Can the threshold be negative?

If approxPolyAccuracy == 13, it will be -1/7

No, approxPolyMaxDistance is an integer >= 1 and <= 8 in this piece of code.

if approxPolyAccuracy == 13, then approxPolyMaxDistance == 0 and the condition in line 35 is false

bsekachev · 2021-07-27T07:54:19Z

@nmanovic

Every interaction with DEXTR leads to the problem below. Probably need to check the threshold value.

Could you please provide more details (image, points, threshold), because I can't reproduce it.

nmanovic · 2021-07-27T08:24:02Z

Use the video file: https://github.com/opencv/opencv/blob/master/samples/data/vtest.avi?raw=true, try to annotate people using DEXTR.

bsekachev · 2021-07-27T08:32:13Z

@nmanovic

Works as expected

bsekachev · 2021-07-27T08:39:40Z

@nmanovic

Maybe just give me points that DEXTR returns to client and be sure you pull the latest changes (I committed all the changes)

nmanovic · 2021-07-28T12:08:16Z

@bsekachev , I will merge. But I cannot pull dots from extreme points (less and more).

Boris Sekachev added 5 commits July 20, 2021 12:44

First stage for points minimizer

2bf427e

Fixed issue with correct opencv initialization status

57fbbe2

Displaying points during interaction

dd63985

Added releasing memory

b6cb02f

Initial version for on-the-fly optimization

367383a

bsekachev added the enhancement New feature or request label Jul 22, 2021

bsekachev requested review from ActiveChooN and nmanovic July 22, 2021 08:02

Boris Sekachev added 2 commits July 22, 2021 12:18

Redesigned accuracy

b2d9548

Updated version & changelog

4636548

bsekachev changed the title ~~[WIP] Semi-automatic tools enhancements (Client-side points minimizer)~~ Semi-automatic tools enhancements (Client-side points minimizer) Jul 22, 2021

bsekachev requested a review from klakhov July 22, 2021 09:26

bsekachev changed the title ~~Semi-automatic tools enhancements (Client-side points minimizer)~~ [WIP] Semi-automatic tools enhancements (Client-side points minimizer) Jul 22, 2021

Boris Sekachev added 2 commits July 22, 2021 15:56

Fixed opencv scissors

725dafa

Clean up some intermediate state

90d69bc

bsekachev changed the title ~~[WIP] Semi-automatic tools enhancements (Client-side points minimizer)~~ Semi-automatic tools enhancements (Client-side points minimizer) Jul 22, 2021

Fixed scss

c111668

ActiveChooN previously approved these changes Jul 23, 2021

View reviewed changes

nmanovic reviewed Jul 23, 2021

View reviewed changes

...c/components/annotation-page/standard-workspace/controls-side-bar/approximation-accuracy.tsx Outdated Show resolved Hide resolved

Redesigned slider a bit

ca42cb5

Added errored shape

1117bf8

bsekachev dismissed ActiveChooN’s stale review via 1117bf8 July 26, 2021 08:19

Keep slider hidden while didn't recieve first points

bbe5c75

Boris Sekachev added 2 commits July 26, 2021 12:16

Adjusted settings slider

b137176

Updated label

b62d6bf

TOsmanov mentioned this pull request Jul 26, 2021

Documentation update #3464

Merged

8 tasks

dvkruchinin mentioned this pull request Jul 26, 2021

Cypress test. Settings. Default number of points in polygon approximation. #3465

Merged

8 tasks

nmanovic reviewed Jul 27, 2021

View reviewed changes

Boris Sekachev added 3 commits July 27, 2021 13:36

A couple of fixes for trackers & detectors

a1be09c

Merged develop

2d9f04f

Updated default value

1197639

nmanovic approved these changes Jul 28, 2021

View reviewed changes

nmanovic merged commit 0ea4897 into develop Jul 28, 2021

nmanovic deleted the bs/semi_enhancements branch July 28, 2021 12:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Semi-automatic tools enhancements (Client-side points minimizer) #3450

Semi-automatic tools enhancements (Client-side points minimizer) #3450

bsekachev commented Jul 22, 2021 •

edited

Loading

bsekachev commented Jul 22, 2021

aschernov commented Jul 22, 2021

dvkruchinin commented Jul 22, 2021

bsekachev commented Jul 23, 2021

nmanovic commented Jul 23, 2021 •

edited

Loading

nmanovic commented Jul 23, 2021

nmanovic commented Jul 23, 2021

nmanovic commented Jul 23, 2021

dvkruchinin commented Jul 26, 2021

bsekachev commented Jul 26, 2021

bsekachev commented Jul 26, 2021

bsekachev commented Jul 26, 2021

nmanovic commented Jul 27, 2021 •

edited

Loading

nmanovic Jul 27, 2021

nmanovic Jul 27, 2021

bsekachev Jul 27, 2021

bsekachev Jul 27, 2021

bsekachev commented Jul 27, 2021

nmanovic commented Jul 27, 2021

bsekachev commented Jul 27, 2021

bsekachev commented Jul 27, 2021

nmanovic commented Jul 28, 2021

Semi-automatic tools enhancements (Client-side points minimizer) #3450

Semi-automatic tools enhancements (Client-side points minimizer) #3450

Conversation

bsekachev commented Jul 22, 2021 • edited Loading

Motivation and context

How has this been tested?

Checklist

License

bsekachev commented Jul 22, 2021

aschernov commented Jul 22, 2021

dvkruchinin commented Jul 22, 2021

bsekachev commented Jul 23, 2021

nmanovic commented Jul 23, 2021 • edited Loading

nmanovic commented Jul 23, 2021

nmanovic commented Jul 23, 2021

nmanovic commented Jul 23, 2021

dvkruchinin commented Jul 26, 2021

bsekachev commented Jul 26, 2021

bsekachev commented Jul 26, 2021

bsekachev commented Jul 26, 2021

nmanovic commented Jul 27, 2021 • edited Loading

nmanovic Jul 27, 2021

Choose a reason for hiding this comment

nmanovic Jul 27, 2021

Choose a reason for hiding this comment

bsekachev Jul 27, 2021

Choose a reason for hiding this comment

bsekachev Jul 27, 2021

Choose a reason for hiding this comment

bsekachev commented Jul 27, 2021

nmanovic commented Jul 27, 2021

bsekachev commented Jul 27, 2021

bsekachev commented Jul 27, 2021

nmanovic commented Jul 28, 2021

bsekachev commented Jul 22, 2021 •

edited

Loading

nmanovic commented Jul 23, 2021 •

edited

Loading

nmanovic commented Jul 27, 2021 •

edited

Loading