[WIP] Serverless tutorial

.vscode/launch.json

@@ -44,6 +44,9 @@
             "justMyCode": false,
             "pythonPath": "${command:python.interpreterPath}",
             "program": "${workspaceRoot}/manage.py",
+            "env": {
+                "CVAT_SERVERLESS":  "1",
+            },
             "args": [
                 "runserver",
                 "--noreload",

README.md

@@ -63,18 +63,22 @@ For more information about supported formats look at the
 | [ImageNet](http://www.image-net.org)                                          | X      | X      |
 | [CamVid](http://mi.eng.cam.ac.uk/research/projects/VideoRec/CamVid/)          | X      | X      |
 
-## Deep learning models for automatic labeling
+## Deep learning serverless functions for automatic labeling
 
 | Name                                                                                                    | Type       | Framework  | CPU | GPU |
 | ------------------------------------------------------------------------------------------------------- | ---------- | ---------- | --- | --- |
-| [Deep Extreme Cut](/serverless/openvino/dextr/nuclio)                                                   | interactor | OpenVINO   | X   |
+| [Deep Extreme Cut](/serverless/openvino/dextr/nuclio)                                                   | interactor | OpenVINO   | X   |     |
+| [Faster RCNN](/serverless/openvino/omz/public/faster_rcnn_inception_v2_coco/nuclio)                     | detector   | OpenVINO   | X   |     |
+| [Mask RCNN](/serverless/openvino/omz/public/mask_rcnn_inception_resnet_v2_atrous_coco/nuclio)           | detector   | OpenVINO   | X   |     |
+| [YOLO v3](/serverless/openvino/omz/public/yolo-v3-tf/nuclio)                                            | detector   | OpenVINO   | X   |     |
+| [Object reidentification](/serverless/openvino/omz/intel/person-reidentification-retail-300/nuclio)     | reid       | OpenVINO   | X   |     |
+| [Semantic segmentation for ADAS](/serverless/openvino/omz/intel/semantic-segmentation-adas-0001/nuclio) | detector   | OpenVINO   | X   |     |
+| [Text detection v4](/serverless/openvino/omz/intel/text-detection-0004/nuclio)                          | detector   | OpenVINO   | X   |     |
+| [SiamMask](/serverless/pytorch/foolwood/siammask/nuclio)                                                | tracker    | PyTorch    | X   |     |
+| [f-BRS](/serverless/pytorch/saic-vul/fbrs/nuclio)                                                       | interactor | PyTorch    | X   |     |
+| [Inside-Outside Guidance](/serverless/pytorch/shiyinzhang/iog/nuclio)                                   | interactor | PyTorch    | X   |     |
 | [Faster RCNN](/serverless/tensorflow/faster_rcnn_inception_v2_coco/nuclio)                              | detector   | TensorFlow | X   | X   |
-| [Mask RCNN](/serverless/openvino/omz/public/mask_rcnn_inception_resnet_v2_atrous_coco/nuclio)           | detector   | OpenVINO   | X   |
-| [YOLO v3](/serverless/openvino/omz/public/yolo-v3-tf/nuclio)                                            | detector   | OpenVINO   | X   |
-| [Text detection v4](/serverless/openvino/omz/intel/text-detection-0004/nuclio)                          | detector   | OpenVINO   | X   |
-| [Semantic segmentation for ADAS](/serverless/openvino/omz/intel/semantic-segmentation-adas-0001/nuclio) | detector   | OpenVINO   | X   |
-| [Mask RCNN](/serverless/tensorflow/matterport/mask_rcnn/nuclio)                                         | detector   | TensorFlow | X   |
-| [Object reidentification](/serverless/openvino/omz/intel/person-reidentification-retail-300/nuclio)     | reid       | OpenVINO   | X   |
+| [Mask RCNN](/serverless/tensorflow/matterport/mask_rcnn/nuclio)                                         | detector   | TensorFlow | X   |     |
 
 ## Online demo: [cvat.org](https://cvat.org)
 

components/serverless/README.md

@@ -2,7 +2,104 @@
 
 ### Run docker container
 
+To use automatic and semi-automatic annotation models you need to install extra
+components which are used by CVAT to implement the functionality. You can do that
+by the command below:
+
 ```bash
 # From project root directory
 docker-compose -f docker-compose.yml -f components/serverless/docker-compose.serverless.yml up -d
 ```
+
+It will run [nuclio](https://github.com/nuclio/nuclio). Nuclio is a
+high-performance `serverless` framework focused on data, I/O, and compute
+intensive workloads.
+
+But it isn't enough. You have to deploy one or several serverless functions
+which you can use in UI to annotate images. Basically a serverless function
+for us is a docker container with HTTP interface which accepts some input in
+json format, executes some code to process the input, and
+
+
+### Tutorial how to add your own DL model for automatic annotation
+
+Let's try to integration [IOG algorithms for interactive segmentation](https://github.com/shiyinzhang/Inside-Outside-Guidance).
+
+First of all let's run the model on your local machine. The repo doesn't have good instructions and look
+like uses outdated versions of packages. The building process is going to be funny. For old version of
+pytorch packages it is better to use conda. See below a possible instructions how to run the model on a
+local machine.
+
+```bash
+git clone https://github.com/shiyinzhang/Inside-Outside-Guidance
+cd Inside-Outside-Guidance/
+conda create --name iog python=3.6
+conda activate iog
+conda install pytorch=0.4 torchvision=0.2  -c pytorch
+conda install -c conda-forge pycocotools
+conda install -c conda-forge opencv
+conda install -c conda-forge scipy
+```
+
+Download weights from google drive: https://github.com/shiyinzhang/Inside-Outside-Guidance#pretrained-models
+Also we will need `VOCtrainval_11-May-2012.tar` dataset for evaluation: http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar
+
+Modify `mypath.py` in accordance with instructions inside the repo. In my case `git diff` below:
+
+```python
+diff --git a/mypath.py b/mypath.py
+index 0df1565..cd0fa3f 100644
+--- a/mypath.py
++++ b/mypath.py
+@@ -3,15 +3,15 @@ class Path(object):
+     @staticmethod
+     def db_root_dir(database):
+         if database == 'pascal':
+-            return '/path/to/PASCAL/VOC2012'  # folder that contains VOCdevkit/.
++            return '/Users/nmanovic/Workspace/datasets/VOCtrainval_11-May-2012/'  # folder that contains VOCdevkit/.
+
+         elif database == 'sbd':
+-            return '/path/to/SBD/'  # folder with img/, inst/, cls/, etc.
++            return '/Users/nmanovic/Workspace/datasets/SBD/dataset/'  # folder with img/, inst/, cls/, etc.
+         else:
+             print('Database {} not available.'.format(database))
+             raise NotImplementedError
+
+     @staticmethod
+     def models_dir():
+-        return '/path/to/models/resnet101-5d3b4d8f.pth'
++        return '/Users/nmanovic/Workspace/Inside-Outside-Guidance/IOG_PASCAL_SBD.pth'
+         #'resnet101-5d3b4d8f.pth' #resnet50-19c8e357.pth'
+```
+
+It looks like need to update `test.py` to run it without `train.py` script.
+
+```python
+diff --git a/test.py b/test.py
+index f85969a..8e481d0 100644
+--- a/test.py
++++ b/test.py
+@@ -51,9 +51,10 @@ net = Network(nInputChannels=nInputChannels,num_classes=1,
+                 freeze_bn=False)
+
+ # load pretrain_dict
+-pretrain_dict = torch.load(os.path.join(save_dir, 'models', modelName + '_epoch-' + str(resume_epoch - 1) + '.pth'))
+-print("Initializing weights from: {}".format(
+-    os.path.join(save_dir, 'models', modelName + '_epoch-' + str(resume_epoch - 1) + '.pth')))
++#pretrain_dict = torch.load(os.path.join(save_dir, 'models', modelName + '_epoch-' + str(resume_epoch - 1) + '.pth'))
++#print("Initializing weights from: {}".format(
++#    os.path.join(save_dir, 'models', modelName + '_epoch-' + str(resume_epoch - 1) + '.pth')))
++pretrain_dict = torch.load('/Users/nmanovic/Workspace/Inside-Outside-Guidance/IOG_PASCAL_SBD.pth')
+ net.load_state_dict(pretrain_dict)
+ net.to(device)
+ ```
+
+Now it is possible to run `test.py` and it will generate results inside `./run_0/Results` directory.
+It is already a great progress. We can run the pretrained model and get results. Next step is to
+implement a simple script which will accept an image with a bounding box and generate a mask for the
+object. Let's do that.
+
+```bash
+cp test.py model_handler.py
+```
+

cvat-ui/src/components/annotation-page/standard-workspace/controls-side-bar/tools-control.tsx

@@ -88,7 +88,7 @@ const mapDispatchToProps = {
     createAnnotations: createAnnotationsAsync,
 };
 
-function convertShapesForInteractor(shapes: InteractionResult[]): number[][] {
+function convertShapesForInteractor(shapes: InteractionResult[], button: number): number[][] {
     const reducer = (acc: number[][], _: number, index: number, array: number[]): number[][] => {
         if (!(index % 2)) {
             // 0, 2, 4
@@ -97,8 +97,7 @@ function convertShapesForInteractor(shapes: InteractionResult[]): number[][] {
         return acc;
     };
 
-    return shapes
-        .filter((shape: InteractionResult): boolean => shape.shapeType === 'points' && shape.button === 0)
+    return shapes.filter((shape: InteractionResult): boolean => shape.shapeType === 'points' && shape.button === button)
         .map((shape: InteractionResult): number[] => shape.points)
         .flat()
         .reduce(reducer, []);
@@ -235,7 +234,8 @@ export class ToolsControlComponent extends React.PureComponent<Props, State> {
                 try {
                     result = await core.lambda.call(jobInstance.task, interactor, {
                         frame,
-                        points: convertShapesForInteractor((e as CustomEvent).detail.shapes),
+                        pos_points: convertShapesForInteractor((e as CustomEvent).detail.shapes, 0),
+                        neg_points: convertShapesForInteractor((e as CustomEvent).detail.shapes, 2),
                     });
 
                     if (this.interactionIsAborted) {

cvat/apps/documentation/installation.md

@@ -285,7 +285,8 @@ Please see the [Docker documentation](https://docs.docker.com/network/proxy/) fo
 
 ```bash
 # Build and run containers with Analytics component support:
-docker-compose -f docker-compose.yml -f components/analytics/docker-compose.analytics.yml up -d --build
+docker-compose -f docker-compose.yml \
+  -f components/analytics/docker-compose.analytics.yml up -d --build
 ```
 
 ### Semi-automatic and automatic annotation

cvat/apps/lambda_manager/views.py

@@ -86,13 +86,14 @@ def __init__(self, gateway, data):
         # ID of the function (e.g. omz.public.yolo-v3)
         self.id = data['metadata']['name']
         # type of the function (e.g. detector, interactor)
-        kind = data['metadata']['annotations'].get('type')
+        meta_anno = data['metadata']['annotations']
+        kind = meta_anno.get('type')
         try:
             self.kind = LambdaType(kind)
         except ValueError:
             self.kind = LambdaType.UNKNOWN
         # dictionary of labels for the function (e.g. car, person)
-        spec = json.loads(data['metadata']['annotations'].get('spec') or '[]')
+        spec = json.loads(meta_anno.get('spec') or '[]')
         labels = [item['name'] for item in spec]
         if len(labels) != len(set(labels)):
             raise ValidationError(
@@ -106,24 +107,34 @@ def __init__(self, gateway, data):
         # http port to access the serverless function
         self.port = data["status"].get("httpPort")
         # framework which is used for the function (e.g. tensorflow, openvino)
-        self.framework = data['metadata']['annotations'].get('framework')
+        self.framework = meta_anno.get('framework')
         # display name for the function
-        self.name = data['metadata']['annotations'].get('name', self.id)
-        self.min_pos_points = int(data['metadata']['annotations'].get('min_pos_points', 1))
+        self.name = meta_anno.get('name', self.id)
+        self.min_pos_points = int(meta_anno.get('min_pos_points', 1))
+        self.startswith_box = bool(meta_anno.get('startswith_box', False))
         self.gateway = gateway
 
     def to_dict(self):
         response = {
             'id': self.id,
             'kind': str(self.kind),
             'labels': self.labels,
-            'state': self.state,
             'description': self.description,
             'framework': self.framework,
-            'name': self.name,
-            'min_pos_points': self.min_pos_points
+            'name': self.name
         }
 
+        if self.kind is LambdaType.INTERACTOR:
+            response.update({
+                'min_pos_points': self.min_pos_points,
+                'startswith_box': self.startswith_box
+            })
+
+        if self.kind is LambdaType.TRACKER:
+            response.update({
+                'state': self.state
+            })
+
         return response
 
     def invoke(self, db_task, data):
@@ -153,7 +164,8 @@ def invoke(self, db_task, data):
             elif self.kind == LambdaType.INTERACTOR:
                 payload.update({
                     "image": self._get_image(db_task, data["frame"], quality),
-                    "points": data["points"],
+                    "pos_points": data["pos_points"],
+                    "neg_points": data["neg_points"]
                 })
             elif self.kind == LambdaType.REID:
                 payload.update({

serverless/openvino/common/python3 → serverless/common/openvino/python3

@@ -3,5 +3,5 @@
 args=$@
 
 . /opt/intel/openvino/bin/setupvars.sh
-PYTHONPATH=/opt/nuclio/common:$PYTHONPATH
+PYTHONPATH=/opt/nuclio/common/openvino:$PYTHONPATH
 /usr/bin/python3 $args

serverless/deploy_cpu.sh

@@ -2,57 +2,19 @@
 # Sample commands to deploy nuclio functions on CPU
 
 SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+FUNCTIONS_DIR=${1:-$SCRIPT_DIR}
 
 nuctl create project cvat
-nuctl deploy --project-name cvat \
-    --path "$SCRIPT_DIR/openvino/omz/public/faster_rcnn_inception_v2_coco/nuclio" \
-    --volume "$SCRIPT_DIR/openvino/common:/opt/nuclio/common" \
-    --platform local
 
-nuctl deploy --project-name cvat \
-    --path "$SCRIPT_DIR/openvino/omz/public/mask_rcnn_inception_resnet_v2_atrous_coco/nuclio" \
-    --volume "$SCRIPT_DIR/openvino/common:/opt/nuclio/common" \
-    --platform local
+for func_config in $(find "$FUNCTIONS_DIR" -name "function.yaml")
+do
+    func_root=$(dirname "$func_config")
+    echo Deploying $(dirname "$func_root") function...
+    nuctl deploy --project-name cvat --path "$func_root" \
+        --volume "$SCRIPT_DIR/common:/opt/nuclio/common" \
+        --platform local
+done
 
-nuctl deploy --project-name cvat \
-    --path "$SCRIPT_DIR/openvino/omz/public/yolo-v3-tf/nuclio" \
-    --volume "$SCRIPT_DIR/openvino/common:/opt/nuclio/common" \
-    --platform local
-
-nuctl deploy --project-name cvat \
-    --path "$SCRIPT_DIR/openvino/omz/intel/text-detection-0004/nuclio" \
-    --volume "$SCRIPT_DIR/openvino/common:/opt/nuclio/common" \
-    --platform local
-
-nuctl deploy --project-name cvat \
-    --path "$SCRIPT_DIR/openvino/omz/intel/semantic-segmentation-adas-0001/nuclio" \
-    --volume "$SCRIPT_DIR/openvino/common:/opt/nuclio/common" \
-    --platform local
-
-nuctl deploy --project-name cvat \
-    --path "$SCRIPT_DIR/openvino/omz/intel/person-reidentification-retail-300/nuclio" \
-    --volume "$SCRIPT_DIR/openvino/common:/opt/nuclio/common" \
-    --platform local
-
-nuctl deploy --project-name cvat \
-    --path "$SCRIPT_DIR/openvino/dextr/nuclio" \
-    --volume "$SCRIPT_DIR/openvino/common:/opt/nuclio/common" \
-    --platform local
-
-nuctl deploy --project-name cvat \
-    --path "$SCRIPT_DIR/tensorflow/matterport/mask_rcnn/nuclio" \
-    --platform local
-
-nuctl deploy --project-name cvat \
-    --path "$SCRIPT_DIR/tensorflow/faster_rcnn_inception_v2_coco/nuclio" \
-    --platform local
-
-nuctl deploy --project-name cvat \
-    --path "$SCRIPT_DIR/pytorch/foolwood/siammask/nuclio" \
-    --platform local
+nuctl get function
 
-nuctl deploy --project-name cvat \
-    --path "$SCRIPT_DIR/pytorch/saic-vul/fbrs/nuclio" \
-    --platform local
 
-nuctl get function

serverless/openvino/dextr/nuclio/function.yaml

@@ -15,7 +15,7 @@ spec:
   eventTimeout: 30s
   env:
     - name: NUCLIO_PYTHON_EXE_PATH
-      value: /opt/nuclio/common/python3
+      value: /opt/nuclio/common/openvino/python3
 
   build:
     image: cvat/openvino.dextr
@@ -48,6 +48,7 @@ spec:
 
   platform:
     attributes:
+      processorMountMode: volume
       restartPolicy:
         name: always
         maximumRetryCount: 3

serverless/openvino/dextr/nuclio/main.py

@@ -15,7 +15,7 @@ def init_context(context):
 def handler(context, event):
     context.logger.info("call handler")
     data = event.body
-    points = data["points"]
+    points = data["pos_points"]
     buf = io.BytesIO(base64.b64decode(data["image"].encode('utf-8')))
     image = Image.open(buf)
 

serverless/openvino/omz/intel/person-reidentification-retail-300/nuclio/function.yaml

@@ -14,7 +14,7 @@ spec:
   eventTimeout: 30s
   env:
     - name: NUCLIO_PYTHON_EXE_PATH
-      value: /opt/nuclio/common/python3
+      value: /opt/nuclio/common/openvino/python3
 
   build:
     image: cvat/openvino.omz.intel.person-reidentification-retail-0300
@@ -43,6 +43,7 @@ spec:
 
   platform:
     attributes:
+      processorMountMode: volume
       restartPolicy:
         name: always
         maximumRetryCount: 3

serverless/openvino/omz/intel/semantic-segmentation-adas-0001/nuclio/function.yaml

@@ -37,7 +37,7 @@ spec:
   eventTimeout: 30s
   env:
     - name: NUCLIO_PYTHON_EXE_PATH
-      value: /opt/nuclio/common/python3
+      value: /opt/nuclio/common/openvino/python3
 
   build:
     image: cvat/openvino.omz.intel.semantic-segmentation-adas-0001
@@ -70,6 +70,7 @@ spec:
 
   platform:
     attributes:
+      processorMountMode: volume
       restartPolicy:
         name: always
         maximumRetryCount: 3