TI Deep Learning (TIDL) Tools

The edgeai-tidl-tools repository provides a complete environment for model compilation and inference. Key components include:

• Model Compilation: Tools for compiling models on x86 systems (downloaded to the tools folder during setup).
• Wrapper APIs: Ready-to-use Python and C++ Wrapper APIs for easy integration into custom applications.
• Reference Examples: Python and C++ examples for implementation guidance.
• Model Tools: Utility scripts in model-tools for offline optimization and modification of models.
• Testing Framework: A Pytest-based TIDL Unit Test Framework to validate models across different runtimes and devices.
• Deployment Support: Support for both x86 (host emulation) and TI SOC (native execution).

TIDL (TI Deep Learning) is a software product designed to accelerate Deep Neural Networks (DNNs) on TI embedded devices. It enables heterogeneous execution across Cortex-A based MPUs, C7x DSPs, and TI's DNN accelerator (MMA).

Supported TI Devices:

• AM62A
• J722S | TDA4AEN | AM67A
• J721E | TDA4VM
• J721S2 | TDA4VL | AM68A
• J784S4 | TDA4VH | AM69A
• AM62 (Note: AM62 without hardware acceleration is supported but lacks hardware acceleration features)

For the most current list, refer to the README.md.

The TIDL (Texas Instruments Deep Learning) Runtime Wrapper Libraries provide high-level interfaces for accelerating deep learning model inference on Texas Instruments devices. These wrappers allow developers to interact with different runtime environments while leveraging TIDL acceleration.

There are two primary implementation paths:

1. C++ Wrappers: Focused on high-level interfaces for inference execution.
2. Python Wrappers: Focused on high-level interfaces for both model compilation and inference execution.

OSRT Model Tools is a utility collection designed to optimize and modify ONNX and TFLite models for improved compatibility and performance with TI Deep Learning (TIDL). It is divided into two main functional components:

1. onnx_tools: Utilities for the ONNX model format. Capabilities include model extraction, adding intermediate outputs, and converting input formats (e.g., RGB to YUV).
2. tflite_tools: Utilities for the TensorFlow Lite model format. Capabilities include optimizing model inputs and converting input formats (e.g., RGB to YUV).

The runtimes/examples directory provides practical implementations for model compilation and inference using TIDL (TI Deep Learning) runtime libraries on Texas Instruments hardware. It covers both Python and C++ implementations, supporting various runtimes including ONNX Runtime, TFLite Runtime, and TVM Runtime.

Key Workflow

1. Model Compilation (Python): Use Python APIs to compile deep learning models into TIDL-compatible artifacts.
2. Model Loading & Inference (C++/Python): Load the compiled artifacts to perform hardware-accelerated inference.

Note that models compiled via Python are stored in a shared model-artifacts/ directory, making them immediately available for use in C++ examples without requiring recompilation.

The tidl_wrapper/cpp directory provides several specialized modules for deep learning inference:

• Common: Contains shared utilities like DlTensor, a unified tensor structure for handling input/output information.
• ONNXRT: Wraps ONNX Runtime with a TIDL provider. Use this to load and run ONNX models with hardware acceleration and automatic tensor type conversion.
• TFLiteRT: Integrates TensorFlow Lite with a TIDL delegate. Use this for TFLite models to enable optimized execution on TI devices.
• TIDLRT: Provides direct access to the TIDL interface without an intermediate framework (like ONNX or TFLite). It is compatible with artifacts from ONNXRT and TFLiteRT if all nodes are offloaded to TIDL. It features shared memory management for efficient data transfer.

The osrt-model-tools package provides a collection of utilities for handling ONNX and TensorFlow Lite models. Key capabilities include:

• Format conversion: Performing operations such as converting color spaces (e.g., RGB to YUV).
• Preprocessing utilities: Tools to prepare model inputs for inference.

This component is intended for post-optimization or pre-inference workflows involving ONNX or TFLite formats.

The tidl_onnx_model_utils package provides utility functions for modifying and analyzing ONNX models. Key capabilities include:

• Graph Analysis: Retrieve nodes located between two specific nodes.
• Model Cleaning: Prune names within the ONNX model.
• Subgraph Extraction: Extract specific subgraphs from a larger model.
• Output Modification: Add intermediate outputs to the model for inspection or further processing.
• Format Conversion: Convert input formats from RGB to YUV.
• Optimization: Optimize model inputs.

The runtimes/ directory provides wrapper libraries and examples for executing deep learning models on Texas Instruments devices. The structure is organized by language and purpose:

• tidl_wrapper/: Contains the core wrapper libraries.
  • python/: Python wrappers for ONNX Runtime, TFLite Runtime, and TVM Runtime.
  • cpp/: C++ wrappers for ONNX Runtime, TFLite Runtime, and direct TIDL Runtime.
• examples/: Contains practical application demonstrations.
  • python/: Python-based examples for model compilation and inference.
  • cpp/: C++-based examples for inference using TIDL acceleration.
• cmake/: Contains CMake configuration files required for building the C++ wrappers and examples.

The tidl-onnx-model-optimizer is a tool used to optimize ONNX models for TIDL (TI Deep Learning) inference compatibility. It applies over 60 transformation rules to the model, including:

• Attention blocks: Advanced optimizations for transformer-based architectures.
• Quantization: Preparing models for reduced precision inference.
• Layout conversions: Adjusting data layouts to match TIDL requirements.

Use this component when you need to ensure an ONNX model is correctly formatted and optimized for execution on TI hardware.