Skip to content

Latest commit

 

History

History
123 lines (83 loc) · 5.47 KB

host.md

File metadata and controls

123 lines (83 loc) · 5.47 KB
Raw

Landing OpenVINO™ Model Server on Bare Metal Hosts and Virtual Machines

Requirements

OpenVINO™ model server installation is fully tested on Ubuntu16.04, however there are no anticipated issues on other Linux distributions like CentoOS*, RedHat*, ClearLinux* or SUSE Linux*.

Download the installer from: OpenVINO™ toolkit

Installation of Dependencies

Generally, you need to repeat the steps listed in the Dockerfile

The steps include installation of Linux packages of cpio, make, wget, python3, python setup tools and virtualenv.

Later, OpenVINO™ toolkit need to be installed - at least the inference engine is required.

It is important to set correct environment variables PYTHONPATH and LD_LIBRARY_PATH pointing to Python modules of OpenVINO™ and compiled libraries.

Model Server Installation

It is recommended to run OpenVINO™ model server from Python virtual environment. This will avoid all conflicts between Python dependencies from other applications running on the same host.

Note All python dependencies are included in requirements.txt file.

OpenVINO™ Model server can be installed using commands:

make install

or if you don't want to create a virtual environment:

pip install .

or if you want to modify the source code and easily test changes:

pip install -e .

Starting the Serving Service

Process of starting the model server and preparation of the models is similar to docker containers.

When working inside the activated virtual environment or with all Python dependencies installed, the server can be started using ie_serving command:

ie_serving --help
usage: ie_serving [-h] {config,model} ...

positional arguments:
  {config,model}  sub-command help
    config        Allows you to share multiple models using a configuration file
    model         Allows you to share one type of model

optional arguments:
  -h, --help      show this help message and exit

The server can be started in interactive mode, as a background process or a daemon initiated by systemctl/initd depending on the Linux distribution and specific hosting requirements.

Refer to docker_container.md to get more details.

Using Neural Compute Sticks

OpenVINO Model Server can employ AI accelerators Intel® Neural Compute Stick and Intel® Neural Compute Stick 2.

To use Movidus Neural Compute Sticks with OpenVINO Model Server you need to have OpenVINO Toolkit with Movidius VPU support installed. In order to do that follow OpenVINO installation instruction. Don't forget about additional steps for NCS.

On server startup, you need to specify that you want to load model on Neural Compute Stick for inference execution. You can do that by setting target_device parameter to MYRIAD. If it's not specified, OpenVINO will try to load model on CPU.

Example:

ie_serving model --model_path /opt/model --model_name my_model --port 9001 --target_device MYRIAD

You can also run it in Docker container

Note: Checkout supported configurations. Look at VPU Plugins to see if your model is supported. If not, take a look at OpenVINO Model Optimizer and convert your model to desired format.

Using HDDL accelerators

OpenVINO Model Server can employ High-Density Deep Learning (HDDL) accelerators based on Intel Movidius Myriad VPUs. To use HDDL accelerators with OpenVINO Model Server you need to have OpenVINO Toolkit with Intel® Vision Accelerator Design with Intel® Movidius™ VPU support installed. In order to do that follow OpenVINO installation instruction. Don't forget about additional steps for Intel® Movidius™ VPU.

On server startup, you need to specify that you want to load model on HDDL card for inference execution. You can do that by setting target_device parameter to HDDL. If it's not specified, OpenVINO will try to load model on CPU.

Example:

ie_serving model --model_path /opt/model --model_name my_model --port 9001 --target_device HDDL

Check out our recommendation for throughput optimization on HDDL

You can also run it in Docker container

Note: Check out supported configurations. Look at VPU Plugins to see if your model is supported. If not, take a look at OpenVINO Model Optimizer and convert your model to desired format.

Using Multi-Device Plugin

See Multi-Device Plugin overview In order to use Multi-Device Plugin on bare host, simply apply instructions linked above without Docker specific steps.