Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend COCO format to skeletons #4821

Merged
merged 313 commits into from
Sep 1, 2022
Merged

Extend COCO format to skeletons #4821

merged 313 commits into from
Sep 1, 2022

Conversation

yasakova-anastasia
Copy link
Contributor

@yasakova-anastasia yasakova-anastasia commented Aug 24, 2022

Motivation and context

  • Added COCO Keypoints 1.0 for skeleton import/export
  • Fixed an issue with moving tasks between projects

How has this been tested?

Checklist

License

  • I submit my code changes under the same MIT License that covers the project.
    Feel free to contact the maintainers if that's a concern.

@zhiltsov-max
Copy link
Contributor

zhiltsov-max commented Aug 26, 2022

  1. Tried to export task inside project with 3 skeletons in COCO and then import back. It seems that visibility properties of points are not preserved - neither occluded, nor outside. After import I've got all points visible. The exported files contain this info.
  2. Failed to export this task in CVAT format, backups for the task and project also failed
# task export
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/rq/worker.py", line 936, in perform_job
    rv = job.perform()
  File "/opt/venv/lib/python3.8/site-packages/rq/job.py", line 684, in perform
    self._result = self._execute()
  File "/opt/venv/lib/python3.8/site-packages/rq/job.py", line 690, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/home/django/cvat/apps/dataset_manager/views.py", line 114, in export_task_as_dataset
    return export(dst_format, task_id=task_id, server_url=server_url, save_images=True)
  File "/home/django/cvat/apps/dataset_manager/views.py", line 81, in export
    export_fn(db_instance.id, temp_file, dst_format,
  File "/home/django/cvat/apps/dataset_manager/task.py", line 750, in export_task
    task.init_from_db()
  File "/home/django/cvat/apps/dataset_manager/task.py", line 622, in init_from_db
    annotation.init_from_db()
  File "/home/django/cvat/apps/dataset_manager/task.py", line 535, in init_from_db
    self._init_shapes_from_db()
  File "/home/django/cvat/apps/dataset_manager/task.py", line 437, in _init_shapes_from_db
    shapes[db_shape.parent].elements.append(db_shape)
KeyError: 45

# task backup
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/rq/worker.py", line 936, in perform_job
    rv = job.perform()
  File "/opt/venv/lib/python3.8/site-packages/rq/job.py", line 684, in perform
    self._result = self._execute()
  File "/opt/venv/lib/python3.8/site-packages/rq/job.py", line 690, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/home/django/cvat/apps/engine/backup.py", line 710, in _create_backup
    exporter.export_to(temp_file)
  File "/home/django/cvat/apps/engine/backup.py", line 386, in export_to
    self._export_task(zip_obj=zf, target_dir=target_dir)
  File "/home/django/cvat/apps/engine/backup.py", line 377, in _export_task
    self._write_annotations(zip_obj, target_dir)
  File "/home/django/cvat/apps/engine/backup.py", line 369, in _write_annotations
    annotations = serialize_annotations()
  File "/home/django/cvat/apps/engine/backup.py", line 362, in serialize_annotations
    annotations = dm.task.get_job_data(db_job_id)
  File "/home/django/cvat/apps/profiler.py", line 11, in wrapped
    return f(*args, **kwargs)
  File "/usr/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/home/django/cvat/apps/dataset_manager/task.py", line 659, in get_job_data
    annotation.init_from_db()
  File "/home/django/cvat/apps/dataset_manager/task.py", line 535, in init_from_db
    self._init_shapes_from_db()
  File "/home/django/cvat/apps/dataset_manager/task.py", line 437, in _init_shapes_from_db
    shapes[db_shape.parent].elements.append(db_shape)
KeyError: 45
  1. It seems that I can't import skeletons from a dataset. When I try to import the exported task data as a dataset for an empty project, I get this error:
Error: Request failed with status code 500. "cvat.apps.dataset_manager.bindings.CvatImportError: Image img3: can't import annotation #0 (points): Label '1' is not registered for this task\n".
  1. Failed to move task to another project with the same skeleton count, but the skeletons are different (including different point count). The error is:
.3:60250] Traceback (most recent call last):
[Fri Aug 26 17:00:40.001177 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
[Fri Aug 26 17:00:40.001178 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     response = get_response(request)
[Fri Aug 26 17:00:40.001179 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
[Fri Aug 26 17:00:40.001181 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     response = wrapped_callback(request, *callback_args, **callback_kwargs)
[Fri Aug 26 17:00:40.001182 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
[Fri Aug 26 17:00:40.001183 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     return view_func(*args, **kwargs)
[Fri Aug 26 17:00:40.001183 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/rest_framework/viewsets.py", line 125, in view
[Fri Aug 26 17:00:40.001184 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     return self.dispatch(request, *args, **kwargs)
[Fri Aug 26 17:00:40.001185 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
[Fri Aug 26 17:00:40.001186 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     response = self.handle_exception(exc)
[Fri Aug 26 17:00:40.001187 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
[Fri Aug 26 17:00:40.001188 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     self.raise_uncaught_exception(exc)
[Fri Aug 26 17:00:40.001189 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
[Fri Aug 26 17:00:40.001190 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     raise exc
[Fri Aug 26 17:00:40.001191 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
[Fri Aug 26 17:00:40.001192 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     response = handler(request, *args, **kwargs)
[Fri Aug 26 17:00:40.001193 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/drf_spectacular/drainage.py", line 156, in wrapped_method
[Fri Aug 26 17:00:40.001194 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     return method(self, request, *args, **kwargs)
[Fri Aug 26 17:00:40.001195 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/rest_framework/mixins.py", line 82, in partial_update
[Fri Aug 26 17:00:40.001196 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     return self.update(request, *args, **kwargs)
[Fri Aug 26 17:00:40.001197 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/drf_spectacular/drainage.py", line 156, in wrapped_method
[Fri Aug 26 17:00:40.001198 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     return method(self, request, *args, **kwargs)
[Fri Aug 26 17:00:40.001199 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/rest_framework/mixins.py", line 68, in update
[Fri Aug 26 17:00:40.001200 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     self.perform_update(serializer)
[Fri Aug 26 17:00:40.001200 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/home/django/cvat/apps/engine/views.py", line 773, in perform_update
[Fri Aug 26 17:00:40.001201 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     updated_instance = serializer.save()
[Fri Aug 26 17:00:40.001202 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/rest_framework/serializers.py", line 200, in save
[Fri Aug 26 17:00:40.001203 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     self.instance = self.update(self.instance, validated_data)
[Fri Aug 26 17:00:40.001204 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/home/django/cvat/apps/engine/serializers.py", line 632, in update
[Fri Aug 26 17:00:40.001205 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     model.objects.filter(job__segment__task=instance, label=old_label).update(
[Fri Aug 26 17:00:40.001206 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/db/models/query.py", line 783, in update
[Fri Aug 26 17:00:40.001207 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     rows = query.get_compiler(self.db).execute_sql(CURSOR)
[Fri Aug 26 17:00:40.001208 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1559, in execute_sql
[Fri Aug 26 17:00:40.001209 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     cursor = super().execute_sql(result_type)
[Fri Aug 26 17:00:40.001209 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
[Fri Aug 26 17:00:40.001210 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     cursor.execute(sql, params)
[Fri Aug 26 17:00:40.001211 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 98, in execute
[Fri Aug 26 17:00:40.001213 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     return super().execute(sql, params)
[Fri Aug 26 17:00:40.001214 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
[Fri Aug 26 17:00:40.001215 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
[Fri Aug 26 17:00:40.001216 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
[Fri Aug 26 17:00:40.001217 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     return executor(sql, params, many, context)
[Fri Aug 26 17:00:40.001218 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
[Fri Aug 26 17:00:40.001218 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     return self.cursor.execute(sql, params)
[Fri Aug 26 17:00:40.001219 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
[Fri Aug 26 17:00:40.001220 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     raise dj_exc_value.with_traceback(traceback) from exc_value
[Fri Aug 26 17:00:40.001221 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
[Fri Aug 26 17:00:40.001222 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     return self.cursor.execute(sql, params)
[Fri Aug 26 17:00:40.001223 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250] django.db.utils.IntegrityError: null value in column "label_id" violates not-null constraint
[Fri Aug 26 17:00:40.001224 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250] DETAIL:  Failing row contains (51, 0, 0, points, f, 0, 469.69,307.0, 7, null, manual, 0, 42, f).
  1. Can't open a job in the task after failed move

@yasakova-anastasia
Copy link
Contributor Author

@zhiltsov-max, thank you for the review.

  1. Tried to export task inside project with 3 skeletons in COCO and then import back. It seems that visibility properties of points are not preserved - neither occluded, nor outside. After import I've got all points visible. The exported files contain this info.

Fixed.

  1. Failed to export this task in CVAT format, backups for the task and project also failed

I cannot reproduce this problem.

  1. It seems that I can't import skeletons from a dataset. When I try to import the exported task data as a dataset for an empty project, I get this error:
Error: Request failed with status code 500. "cvat.apps.dataset_manager.bindings.CvatImportError: Image img3: can't import annotation #0 (points): Label '1' is not registered for this task\n".

I'm aware of this issue. This is not related to COCO format. I will fix it in another PR.

  1. Failed to move task to another project with the same skeleton count, but the skeletons are different (including different point count).

Fixed.

  1. Can't open a job in the task after failed move

Fixed.

@sizov-kirill
Copy link
Contributor

sizov-kirill commented Aug 29, 2022

Question: Am I understand correctly that for moving task from one project to another I should to setup skeleton label twice: ones for source project and once for target project and these skeletons must be identical?

Error: I put skeleton on an image and marked one point as outside, after that bounding box of my skeleton changed, I saved, and after that I tried to export annotations to the COCO Keypoints 1.0 format, but I got an error Error: Request failed with status code 500. "KeyError: 127\n".

@yasakova-anastasia
Copy link
Contributor Author

Question: Am I understand correctly that for moving task from one project to another I should to setup skeleton label twice: ones for source project and once for target project and these skeletons must be identical?

Yes, you're right

@cvat-ai cvat-ai deleted a comment from github-actions bot Aug 31, 2022
@sizov-kirill
Copy link
Contributor

Could you please merge develop branch, it has some important changes that necessary for running all tests for this PR?

@cvat-ai cvat-ai deleted a comment from github-actions bot Aug 31, 2022
@cvat-ai cvat-ai deleted a comment from github-actions bot Aug 31, 2022
@yasakova-anastasia
Copy link
Contributor Author

/check

@github-actions
Copy link
Contributor

github-actions bot commented Aug 31, 2022

❌ Some checks failed
📄 See logs here

@sizov-kirill
Copy link
Contributor

/check

@github-actions
Copy link
Contributor

github-actions bot commented Sep 1, 2022

❌ Some checks failed
📄 See logs here

Copy link
Contributor

@sizov-kirill sizov-kirill left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but need to add documentation for new format.

@nmanovic nmanovic merged commit 8b3e247 into develop Sep 1, 2022
@nmanovic nmanovic deleted the ay/skeletons-to-coco branch September 1, 2022 08:51
@nmanovic nmanovic mentioned this pull request Sep 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants