Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incremental save of annotation. #120

Merged
merged 13 commits into from
Oct 15, 2018
Merged

Incremental save of annotation. #120

merged 13 commits into from
Oct 15, 2018

Conversation

azhavoro
Copy link
Contributor

@azhavoro azhavoro commented Oct 5, 2018

No description provided.

Copy link
Member

@bsekachev bsekachev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++: /* exported createExportContainer ExportType */
Also need add these symbols into eslintrc.conf.js global variables

@@ -3150,6 +3147,25 @@ class PointsView extends PolyShapeView {
}
}

function createExportContainer() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe it is better to move this function into base.js
The same for ExportType object

@@ -970,13 +1090,13 @@ def _save_paths_to_db(self):
# for Postgres bulk_create will return objects with ids even ids
# are auto incremented. Thus we will not be inside the 'if'.
if shape_type == 'polygon_paths':
db_paths = list(self.db_job.objectpath_set.filter(shapes="polygons"))
db_paths = list(self.db_job.objectpath_set.exclude(id__in=db_paths_ids).filter(shapes="polygons"))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we use exclude() with indexes which have existed before bulk_create(), we need only one row:
db_paths = list(self.db_job.objectpath_set.exclude(id__in=db_paths_ids)
Right?

cvat/apps/engine/annotation.py Show resolved Hide resolved
@@ -145,6 +145,7 @@ class Meta:
class Shape(models.Model):
occluded = models.BooleanField(default=False)
z_order = models.IntegerField(default=0)
client_id = models.IntegerField(default=-1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably better add client_id into Annotation class instead of Shape and ObjectPath

self._save_shapes_to_db()
self._save_paths_to_db()
def _get_relaited_obj_field_name(self, shape_type):
'polygon_paths', 'polyline_paths', 'points_paths', 'box_paths'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it extra line?

@@ -51,6 +57,8 @@ class ShapeCollectionModel extends Listener {
this._colorIdx = 0;
this._filter = new FilterModel(() => this.update());
this._splitter = new ShapeSplitter();
this._erased = false;
this._initialShapes = [];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This field is list, but in this.updateHash() the same name is used as dictionary

@@ -221,49 +229,84 @@ class ShapeCollectionModel extends Listener {
return this;
}

_getExportTargetContainer(export_type, shape_type, container) {
Copy link
Member

@bsekachev bsekachev Oct 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure it is should be a method? It don't uses any "this" context.

@@ -801,6 +878,10 @@ class ShapeCollectionModel extends Listener {
get shapes() {
return this._shapes;
}

get erased() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unusable getter

for (const shape of this._shapes) {
let target_export_container = undefined;
if (!shape._removed) {
if (!(shape.id in this._initialShapes)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if (!(shape.id in this._initialShapes) || this._erased) {

continue;
}
}
else if (shape.id in this._initialShapes) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(shape.id in this._initialShapes && !this._erased)

@@ -77,7 +79,8 @@ class AnnotationParser {

let occluded = +shape.getAttribute('occluded');
let z_order = shape.getAttribute('z_order') || '0';
return [points, occluded, +z_order];
let client_id = box.getAttribute('client_id') || '-1'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@azhavoro ,

Need to check somehow that client_id is valid. For example, we don't have the same client id for different boxes or all of them are -1. The check should be on client side when we parse annotation and on server side when the user presses down key.


const data = {
annotation: exportedData,
annotation: JSON.stringify(exportedData),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the reason to send it as string?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think cause of issue was hash function, that used early. I've not changed request format for backward capability and suggest to change it in new MR.

shape_container_target = export_action_container.polyline_paths;
}

if (!shape_container_target) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does switch has 'default'? Does it make sense to move the statement into default?

this._erased = false;
}

export() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How long does the function works on big tasks like FP/FN?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

About 600ms for 90k objects

client_ids = set()
def extract_and_check_clinet_id(obj):
if 'client_id' not in box:
raise Exception('No client_id field in received data')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please fix indentation

if 'client_id' not in box:
raise Exception('No client_id field in received data')
client_id = obj['client_id']
if obj['client_id'] in client_ids:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

client_id in client_ids

client_ids.add(client_id)
return client_id

for action in ['create', 'update', 'delete']:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's roll the loop

@@ -1707,7 +2065,8 @@ def _flip_shape(shape, im_w, im_h):
("ytl", "{:.2f}".format(shape.ytl)),
("xbr", "{:.2f}".format(shape.xbr)),
("ybr", "{:.2f}".format(shape.ybr)),
("occluded", str(int(shape.occluded)))
("occluded", str(int(shape.occluded))),
("client_id", "{}".format(shape.client_id)),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

str(shape.client_id)

@@ -1726,7 +2085,8 @@ def _flip_shape(shape, im_w, im_h):
"{:.2f}".format(float(p.split(',')[1]))
)) for p in shape.points.split(' '))
)),
("occluded", str(int(shape.occluded)))
("occluded", str(int(shape.occluded))),
("client_id", "{}".format(shape.client_id)),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use str

@@ -1773,7 +2133,8 @@ def _flip_shape(shape, im_w, im_h):
for path in path_list:
dump_dict = OrderedDict([
("id", str(path_idx)),
("label", path.label.name)
("label", path.label.name),
("client_id", "{}".format(path.client_id)),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use str

bsekachev
bsekachev previously approved these changes Oct 12, 2018
@bsekachev
Copy link
Member

bsekachev commented Oct 12, 2018

@azhavoro
I found a bug in this patch. I tried next scenario:

  1. I opened a annotation task.
  2. I remove all annotation by click "Remove annotation" button
  3. I created any bounding box
  4. I saved it and got 400 error.
  5. I saved it again and it was successful.
  6. I updated page and I saw old annotation together with new annotation.

@bsekachev
Copy link
Member

@azhavoro
Also after client parsed an annotation file without client_id field, we have situation when all objects have index -1. I think it isn't right.

@bsekachev
Copy link
Member

bsekachev commented Oct 12, 2018

@azhavoro
Other bug in a scenario:

  1. I opened task (it was a interpolation task, but I think it doesn't matter)
  2. I uploaded annotation file without client_id fields.
  3. I saved job. It was successful.
  4. I moved any box. (UPD: actually it isn't necessary step)
  5. I saved it again and I got 400 error.
  6. Each next save also returned 400 code.

@bsekachev
Copy link
Member

@azhavoro
ReID doesn't work with this patch. I checked it.

@nmanovic nmanovic dismissed bsekachev’s stale review October 14, 2018 18:35

The patch has critical bugs

@nmanovic nmanovic merged commit 7badd24 into develop Oct 15, 2018
@nmanovic nmanovic deleted the az/anno_synchronization branch October 15, 2018 13:19
@nmanovic nmanovic mentioned this pull request Oct 15, 2018
nmanovic added a commit that referenced this pull request Dec 29, 2018
* Bug has been fixed: impossible to lock/occlude object in AAM
* Bug has been fixed: invisible points actually are visible
* Bug has been fixed: impossible to close points after editing (#98)
* doc: grammatical cleanup of README.md (#107)
* Add info about development environment into CONTRIBUTING.md (#110)
* Now we store virtual URL instead of update it in the browser address bar (#112)
* Copy URL, Frame URL and object URL functionality in a context menu
* Bug has been fixed: label UIs don't update after changelabel (#109)
* Common escape button for exit from creating/groupping/merging/pasting/aam
* Switch outside/keyframe shortkeys
* Fix django vulnerability (#121)
* Add analytics component (#118)
* Incremental save of annotations (#120)
* Create task timeout 1h -> 4h. (#136)
* OpenVino integration (#134)
* Update README.md (#138)
* Add an extra field into meta section of a dump file (#149)
* Job status was implemented (#153)
* Back link to task from annotation view (#156)
* Change a task with labels and attributes in admin panel (#157)
* Permissions per tasks and jobs (#185)
* Fix context menu, text visibility for small images (#202)
* Fixed: both context menu are opened simultaneously
* Fixed: shape can be unavailable behind text
* Fixed: invisible text outside frame
* Fix upload big xml files for tasks (#199)
* Add Questions section to Readme.md (#226)
* Fixed labels order (#242)
* Propagate behaviour has been updated in cases with a different resolution (#246)
* Updated the guide and images (#241)
* Fix number attribute for float numbers. (#258)
TOsmanov pushed a commit to TOsmanov/cvat that referenced this pull request Aug 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants