DOCSEARCHbyAlgolia
Submit Docs

OpenVINO Toolkit Documentation

repository·master·Indexed 27 days ago

https://github.com/openvinotoolkit/openvino

An open-source software toolkit for optimizing and deploying deep learning models across CPUs, GPUs, and NPUs. It supports frameworks including PyTorch, TensorFlow, and ONNX, providing tools for efficient inference from edge to cloud. The toolkit includes the OpenVINO Runtime, a C API for inference and preprocessing, the benchmark_app for performance estimation, and the physicalai package for Physical AI workflows.

Tokens
239K
Snippets
469
Records
1.3K
Agent score
92%

What's inside OpenVINO

  2. Overview of OpenVINO ONNX Frontend

    master

    The OpenVINO ONNX Frontend is responsible for importing ONNX models and converting them into the ov::Model representation.

    Key capabilities include:

    • Modifying tensor properties (e.g., data type and shapes).
    • Changing model topology (e.g., cutting subgraphs, inserting additional inputs and outputs).
    • Searching models using tensor and operator names.

    The component is implemented in C++ and implements the common interface defined in the Frontends API.

  4. Overview of OpenVINO™ Security Add-on components

    master

    The OpenVINO™ Security Add-on works with the OpenVINO™ Model Server on Intel® architecture to provide secure packaging, secure model execution, and access control. It consists of three primary components running in Kernel-based Virtual Machines (KVMs):

    1. OpenVINO™ Security Add-on Tool (ovsatool): Used by Model Developers or Independent Software Vendors to:
      • Generate access-controlled models (output file archives) from OpenVINO™ IR files.
      • Manage cryptographic keys and collateral within a secure VM environment.
      • Generate user-specific licenses in JSON format, allowing for policies like time limits or execution counts.
    2. OpenVINO™ Security Add-on License Service: Hosted by the Independent Software Vendor to validate licenses. When a user attempts to load an access-controlled model, the runtime contacts this service over the Internet to verify the license parameters.
    3. OpenVINO™ Security Add-on Runtime: Installed by the User in a Guest VM. It validates the user's license against the License Service during the model loading process in the OpenVINO™ Model Server. Once validated, the model is loaded and inference requests are serviced.

  5. Overview of OpenVINO Python API

    master

    The OpenVINO Python API provides bindings to both basic and advanced APIs from the OpenVINO Runtime, allowing users to integrate OpenVINO functionality directly into Python code.

    Beyond simple C++ translations, the Python API includes 'pythonic' enhancements such as:

    • NumPy Support: Specific extensions to handle NumPy-based data structures.
    • Framework Integration: Support for inputs from external frameworks.
    • Pythonic Helpers: Shortcuts and helpers designed for Pythonic workflows.
    • Advanced Concepts: Support for features like shared memory to optimize performance.

  6. Overview of OpenVINO TensorFlow Frontend

    master

    The TensorFlow Frontend (TF FE) is a C++ component that reads and converts TensorFlow models into an ov::Model object. This object can then be serialized into the OpenVINO Intermediate Representation (IR) format.

    Note: This is an internal API. Regular users should not use the frontend directly. Instead, use user-facing tools like the OVC tool (OpenVINO Model Converter), the Model Conversion API, or the OpenVINO Runtime read_model function to handle TensorFlow models.

  8. Overview of NNCF Optimization Methods

    master

    NNCF supports models in OpenVINO IR, PyTorch, and ONNX formats. The main optimization methods are:

    • Weight Compression: Used for footprint reduction and inference acceleration in Large Language Models (LLMs) and Transformer-based models.
    • Post-training Quantization: Applies 8-bit integer quantization. It is the easiest method as it requires no retraining, but may impact accuracy.
    • Training-time Optimization: Includes advanced methods like Sparsity and Quantization-aware Training (QAT). This requires the PyTorch framework and is used if post-training quantization results in unacceptable accuracy loss.

  9. Overview of OpenVINO™ Paddle Frontend

    master

    The OpenVINO Paddle Frontend is a specialized library for the Baidu PaddlePaddle™ framework. It facilitates the transition of PaddlePaddle models into the OpenVINO ecosystem through two primary components:

    1. Paddle Reader: Parses PaddlePaddle protobuf models and converts them into the OpenVINO InputModel format.
    2. Paddle Converter: Decodes PaddlePaddle models and operators, mapping them semantically to the OpenVINO opset.

    Developers looking to extend or debug the frontend can refer to the component's internal documentation and API headers located in the docs/ and include/ directories respectively.