[Datumaro] Add generic accuracy checker model launcher

[zhiltsov-max]

Merge after #1714

• Added generic model launcher with Accuracy Checker backend.
• Added a PyTorch model example (requires some path adjustments for testing)

Motivation and context

Model support in Datumaro is extended from OpenVINO to OpenVINO, TensorFlow, PyTorch, ONNX, MxNet. Such models now can be used for auto annotation and explainable AI.

How has this been tested?

Manual test

# create a project
# copy and adjust paths in the pytorch model example
datum model add -l ac -- -c <path/to/config.yaml>
datum model run -m model-1

Checklist

• I submit my changes into the develop branch
• I have added description of my changes into CHANGELOG file
• I have updated the documentation accordingly
• I have added tests to cover my changes
• I have linked related issues (read github docs)
• I have increased versions of npm packages if it is necessary (cvat-canvas,
  cvat-core, cvat-data and cvat-ui)

License

• I submit my code changes under the same MIT License that covers the project.
  Feel free to contact the maintainers if that's a concern.
• I have updated the license header for each file (see an example below)

# Copyright (C) 2020 Intel Corporation
#
# SPDX-License-Identifier: MIT

[zhiltsov-max]

@eaidova, please, take a look at AC use.

[zhiltsov-max]

@nmanovic, can we put it here?

[coveralls]

Pull Request Test Coverage Report for Build 5861

• 11 of 91 (12.09%) changed or added relevant lines in 3 files are covered.
• 1 unchanged line in 1 file lost coverage.
• Overall coverage decreased (-0.3%) to 65.527%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
datumaro/datumaro/util/annotation_tools.py	2	3	66.67%
datumaro/datumaro/plugins/accuracy_checker_plugin/launcher.py	5	19	26.32%
datumaro/datumaro/plugins/accuracy_checker_plugin/details/ac.py	4	69	5.8%

Files with Coverage Reduction	New Missed Lines	%
datumaro/datumaro/components/extractor.py	1	91.01%

Totals
Change from base Build 5855:	-0.3%
Covered Lines:	10927
Relevant Lines:	16272

---

💛 - Coveralls

[azhavoro]

@zhiltsov-max could you update documentation and add a couple of examples for non openvino frameworks?

[zhiltsov-max]

@azhavoro, here is an example of a pytorch model. I can add examples with other frameworks in another PR.

[zhiltsov-max]

@azhavoro, the pytorch example.

[eaidova]

not sure that it is good approach, at least it will require update on each new representation and extend code with a lot of elif statements...
Probably, condition can be converted to dict format where keys are type and value is functor for conversion or something similar.
Another approach which I can suggest to have to_dm_prediction method inside AC prediction, if it is possible (but it will require to update AC code and make AC dependent on DM...now we are working on dep reducion due to necessary to work on ARM...)

so here is something to think about...

[azhavoro]

Issues:

1. Where I can find the dependencies list? Without manual installation yamlloader and nibabel packages python3 datum.py model add -h doesn't display ac as a launcher, also there are no warnings or exception messages.

   List of builtin launchers: open_vino
   

2. python3 datum.py model add -p test -l ac
   2020-06-08 16:47:22,089 INFO: Checking the model
   2020-06-08 16:47:22,089 ERROR: join() argument must be str or bytes, not 'NoneType'
   Traceback (most recent call last):
     File "datum.py", line 8, in <module>
       sys.exit(main())
     File "/home/django/datumaro/datumaro/cli/__main__.py", line 113, in main
       return args.command(args)
     File "/home/django/datumaro/datumaro/cli/contexts/model/__init__.py", line 89, in add_command
       project.make_executable_model(args.name)
     File "/home/django/datumaro/datumaro/components/project.py", line 823, in 
   make_executable_model
       **model.options, model_dir=self.local_model_dir(name))
     File "/home/django/datumaro/datumaro/components/project.py", line 292, in make_launcher
       return self.launchers.get(name)(*args, **kwargs)
     File "/home/django/datumaro/datumaro/plugins/accuracy_checker_plugin/launcher.py", line 29, in __init__
       with open(osp.join(model_dir, config), 'r') as f:
     File "/usr/lib/python3.5/posixpath.py", line 89, in join
       genericpath._check_arg_types('join', a, *p)
     File "/usr/lib/python3.5/genericpath.py", line 143, in _check_arg_types
       (funcname, s.__class__.__name__)) from None
   TypeError: join() argument must be str or bytes, not 'NoneType'
   

3. python3 datum.py model add -p test -l ac -- -c tests/assets/pytorch_launcher/model_config.yml
   log1.txt

4. cd tests/assets/pytorch_launcher && python3 ~/datumaro/datum.py model add -p ~/datumaro/test -l ac -- -c model_config.yml
   log2.txt

[zhiltsov-max]

Thanks for the comments.

1. Yes, there is some problem with listing plugin dependencies. Here you only need to install Accuracy checker. Actually, it is an unfortunate coincidence that it was found in your OpenVINO env.
2. Should be fixed.
3. Right, in this version it is impossible to launch a model with arbitrary location without absolute path in the config.
4. Opened a PR with fix AC: Add data mapping to CPU for PyTorch launcher when no CUDA available openvinotoolkit/open_model_zoo#1209

[nmanovic]

Here is an overview of what got changed by this pull request:

Complexity increasing per file
==============================
- datumaro/datumaro/plugins/accuracy_checker_plugin/details/representation.py  9
- datumaro/datumaro/plugins/accuracy_checker_plugin/launcher.py  3
- datumaro/datumaro/plugins/accuracy_checker_plugin/details/ac.py  9
- datumaro/tests/assets/pytorch_launcher/samplenet.py  1
         

See the complete overview on Codacy