ONE (On-device Neural Engine)

ONERT supports model training in both Host and On-device environments. It is specifically optimized for on-device training where storage and computational resources are limited.

Key features include:

• On-device training: Based on existing trained models.
• Transfer Learning: Learning from new data added to a previously trained model.
• Supervised Fine-tuning: Improving the accuracy of a trained model using less data.
• Resource Optimization: Reuses memory space during training and uses optimized kernels for each operation to improve accuracy quickly with minimal resource usage.
• Model Export: Trained models can be saved and redeployed via the ONERT API for inference or further retraining in other environments.

Stab is a tool designed to schedule backends for each operation within an nnpackage by using profiled execution data. It identifies the fastest backend for each individual operation to optimize overall inference time.

Supported backends include:

• cpu
• ruy
• xnnpack

Note: Other backends will be supported once Stab can measure and utilize permutation time between backends. The final nnpackage containing the optimized backend configuration is generated at ./tools/stab/nnpkg_sched.

ONE is a high-performance, on-device neural network (NN) inference framework designed to execute NN models on various processors, including CPUs, GPUs, DSPs, and NPUs.

It consists of:

• A runtime that operates on Linux kernel-based OS platforms (such as Ubuntu, Tizen, or Android).
• A compiler toolchain that unifies models from different training frameworks (like TensorFlow or PyTorch) into a single format for runtime execution.

The tensorflow_model_freezer tool automates the generation of several model artifacts by defining Test Cases for various operand ranks. The tool generates:

1. Tensorflow model files: in *.pb and *.pbtxt formats.
2. Frozen Tensorflow models: in *.pb and *.pbtxt formats.
3. Tensorboard log files: for visual inspection of the models.
4. TFLITE files: generated after running TOCO.

The one-import-onnx-ext tool acts as a wrapper for the onnx2circle tool, facilitating the conversion of ONNX models into Circle models.

Note: This tool is currently marked as disabled. Since onnx2circle is becoming the default ONNX conversion tool, one-import-onnx-ext is no longer used as the primary driver for onnx2circle conversions.

gen_golden is a tool used to generate golden data from model files in either GraphDef (.pb) or TFLite (.tflite) formats. The tool generates random inputs, executes the model using TensorFlow, and saves both the inputs and the resulting outputs into the project's specific .h5 format.

Prerequisites

• TensorFlow: Requires tensorflow >= 1.12. It has been verified with versions 1.13, 1.14, and 2.0.

# Example: Generate input.h5 and expected.h5 in the current directory
./gen_golden.py Add_000.pb

# Example: Generate input.h5 and expected.h5 in a specific directory
./gen_golden.py -o ~/tmp Add_000.tflite

An IExecutor is the execution engine for a Subgraph, produced as the result of a Subgraph compilation. It acts as an interpreter that executes the inference logic defined within a subgraph.

Models in this system are treated as a set of tasks with dependencies (a Directed Acyclic Graph or DAG), following a dataflow programming model where operations are executed in topological order once their input tensors are ready.

When defining pkg-inputs, pkg-outputs, or model-connect in the MANIFEST, use the triple notation to identify specific connection points:

model_index : subgraph_index : io_index

• model_index: The position of the model in the models array (0-indexed).
• subgraph_index: The sub-graph identifier inside the model file (e.g., for Circle or TensorFlow Lite).
• io_index: The input or output slot number within that specific sub-graph.

Example Connection Logic:

• "from": "0:0:0", "to": [ "1:0:0", "1:0:1" ]: Connects the output at index 0:0:0 of the first model (index 0) to two different inputs of the second model (index 1).