OpenVINO™ toolkit provides the extgen tool that facilitates creating Model Optimizer and Inference Engine extensions. The tool generates extension source files with stubs for the core functions. To get the workable extension, you should only add your implementation of these functions to the generated files.
To generate extension files, run the extgen tool in one of the available modes:
new command line option. For example: extgen tool directory: <INSTALL_DIR>/deployment_tools/extension_generator/config.extgen.json.example.To run the tool in the interactive mode, specify the following parameters:
mo-op to generate a Model Optimizer operationmo-caffe-ext to generate a Model Optimizer Caffe* extractormo-tf-ext to generate a Model Optimizer TensorFlow* extractormo-mxnet-ext to generates a Model Optimizer MXNet* extractorie-cpu-ext to generate an Inference Engine CPU extensionie-gpu-ext to generate an Inference Engine GPU extensionoutput_dir to set an output directory. If not specified, the current directory is used by default.You can use any combination of the parameters to generate Model Optimizer and/or Inference Engine extension files. For example:
To generate Model Optimizer extension files, run the tool in the interactive mode with necessary parameters or in the silent mode with a configuration file. For example, to generate operation and extractor files for a Caffe model in the <output_directory> in the interactive mode, run the following command:
The extension stub files are generated in the <output_dir>/user_mo_extensions directory, which has the following structure:
/front/caffe - Folder with Caffe extractors/mxnet - Folder with MXNet extractors/tf - Folder with TensorFlow extractors/ops - Folder with operation filesSpecific paths to the generated files appear on the screen. For example, for the Caffe* Proposal layer, the files are <output_dir>/user_mo_extensions/front/caffe/proposal_ext.py and <output_dir>/user_mo_extensions/ops/proposal.py.
Usually, you can use an extractor file without changes. Exceptions are the cases when you want to transform parameters from a input file in the IR. In this case, you should add these transformations to the extract method. Do not forget to add parameter names to the supported_attrs and backend_attrs methods to the operation file.
An operation file can be used without changes if your layer does not change the shape. Otherwise, you should implement the shape calculation in the <op_name>_infer method. Also, you can add default values to the __init__ method. You can find more details in the Extending Model Optimizer with New Primitives.
To generate stub files for GPU and CPU Inference Engine extensions, run the tool and provide input information interactively or in the configuration file. For example, to generate an Inference Engine CPU extension files in the <output_directory> in the interactive mode:
The extension stub files are generated in the <output_dir>/user_ie_extensions directory.
For CPU, several files are generated in the cpu subdirectory. You must change only <output_dir>/user_ie_extensions/cpu/ext_<op_name>.cpp with adding inference implementation.
For GPU, <op_name>.cl and <op_name>.xml are generated in the gpu subdirectory. You must update both:
<op_name>.cl, implement an OpenCL™ kernel to infer the model.<op_name>.xml, fill information about input/output buffers and worksize for your kernel.More details about implementing Inference Engine extensions see in Inference Engine Kernels Extensibility.
This section provides step-by-step examples of extension generation for conversion Caffe* and TensorFlow* models. The Caffe* example describes the Inference Engine extension creation. The TensorFlow* example uses existing Inference Engine operation. If you need Inference Engine extension to infer a TensorFlow-based model, look at steps 6-7 in Caffe* example, because Inference Engine extension generation does not depend on the framework is it based on.
This section provides a sample for generating and implementing Model Optimizer and Inference Engine custom layer extensions for the Proposal layer of a Caffe* example model. The model (.prototxt and .caffemodel) used in the example is described in the Extending Model Optimizer with New Primitives chapter.
extgen: extgen.py file with the following parameters to generate extension stub files: ./user_mo_extensions and ./user_ie_extensions directories, which have the following structure:/user_mo_extensions__init__.py/front/caffe__init__.pyproposallayer_ext.py/mxnet__init__.py/ops__init__.pyproposal.py/user_ie_extensions/cpuCMakeLists.txtext_base.cppext_base.hppext_lists.cppext_lists.hppext_proposal.cpp/gpuImplement extension functions in the generated files:
a. Extractor proposallayer_ext.py can be used without changes.
b. Add the shape calculation logic to the operation file proposal.py. According to IR catalog, the 'Proposal' layer shape dynamically depends on the post_nms_topn parameter.
Add this parameter with the default value in __init__ method:
Then add supported attributes in the method supported_attrs:
Now add shape calculation in infer function:
Proposal layer inference to the execute method in the ext_proposal.cpp file. You can find sample code for this extension in the <INSTALL_DIR>/deployment_tools/inference_engine/samples/extension/ext_proposal.cpp file. For more information about implementation of Inference Engine extensions, refer to Inference Engine Kernels Extensibility.Build a library with CPU extension to use it with the Inference Engine:
a. Create a new build directory:
b. Go to the created build directory:
c. Set the environment variables:
d. Run CMake to generate the Make files:
e. Build the library:
This section provides an example for generating and implementing Model Optimizer extension on TensorFlow* example model.
If you already have a model with unrecognized operation, you can omit Model Preparation and go to Extension Generation chapter.
In example the Pooling layer will be used to illustrate extension generation. ModelOptimizer already supports this layer but we will remove it and show how it can be created with extgen tool. This process described in Model Preparation chapter.
Operation and Inference engine extension generation does not depend on framework and was demonstrated already in Caffe example, so here only TensorFlow extractor generation will be done.
ResNet-50 from TensorFlow* Model Zoo. Follow the instructions in Convert Model From TensorFlow* to prepare the model for converting.ResNet-50 model, it will be converted successfully. To demonstrate extension generation, remove the existing implementation of Pooling layer from the Model Optimizer: Now the sample model is ready for extension generation.
extgen.py file with the following parameters to generate extension stub files : user_mo_extensions directory, which has the following structure:/user_mo_extensions__init__.py/front/caffe__init__.py/mxnet__init__.py/tf__init__.pypooling_ext.py/ops__init__.pyImplement extension functions in the generated files:
a. The extractor pooling_ext.py requires additional attribute conversion. Several attributes should be initialized by constants, real values will be calculated during inference. These changes are needed because we use existing operation that was written for several frameworks.
Conversion should finish successfully.