RKNN-Toolkit2 Documentation

The examples/ directory contains reference implementations for converting models from various frameworks to the RKNN format. Examples are organized by the source framework:

• Caffe: Includes mobilenet_v2 and vgg-ssd float models.
• ONNX: Includes resnet50v2 and yolov5 float models.
• PyTorch: Includes resnet18, resnet18_qat (Quantization Aware Training), resnet18_export_onnx (demonstrating ONNX export from PyTorch), and yolov5 float models.
• TensorFlow: Includes ssd_mobilenet_v1 and inception_v3_qat (QAT) models.
• TFLite: Includes mobilenet_v1 and mobilenet_v1_qat models.
• Darknet: Includes yolov3_416x416 float models.

After running the MMSE quantization example, the script outputs an accuracy analysis table and the top classification results.

Accuracy Table Columns:

• layer_name: The specific layer being analyzed.
• simulator_error: Error metrics comparing the simulator to the actual hardware.
  • entire: Error metrics for the entire model.
  • single: Error metrics for a single layer.
  • cos: Cosine similarity metric.
  • euc: Euclidean distance metric.

Classification Output: The script prints the -----TOP 5----- labels, showing the index, confidence score, and class name for the image classification results.

There are two ways to interface with the RKNN API on Android, and your choice depends on your compliance requirements:

1. librknnrt.so: Link directly to this library for better performance. It has a shorter interface calling process. Use this if your device does not need to pass CTS/VTS testing.
2. librknn_api_android.so: This implementation is based on the Android platform HIDL. Use this if your device must pass CTS/VTS testing. This library is part of the Android system SDK located in vendor/rockchip/hardware/interfaces/neuralnetworks.

When reviewing the output of an accuracy analysis, the results are categorized into two error calculation modes:

Each mode provides two mathematical comparisons:

• cos: Cosine similarity.
• euc: Euclidean distance.

The accuracy analysis compares the output of the simulator and the runtime against a 'golden' value. The results are categorized by error type and accumulation method:

Error Types

• simulator_error: The output error of each layer of the simulator compared to the 'golden' value.
• runtime_error: The output error of each layer of the runtime compared to the 'golden' value.

Accumulation Methods

• entire: The error of each layer as it accumulates layer by layer (cumulative error).
• single: The error of a single layer (e.g., single_sim compares the simulator directly to the runtime), which better reflects the accuracy of that specific component.

Metrics

Errors are typically reported using:

• cos: Cosine similarity.
• euc: Euclidean distance.

When using the YOLOv5 demo, keep the following requirements and constraints in mind:

• RKNN-Toolkit2 Version: Requires version 1.4.0 or higher.
• Custom Models: If using a custom trained model, you must align anchor and post-processing parameters to avoid parsing errors. Specifically, update include/postprocess.h with your model's OBJ_CLASS_NUM, NMS_THRESH, and BOX_THRESH.
• Hardware/Performance: Due to hardware limitations, the post-processing part of the YOLOv5 model is implemented on the CPU by default. The provided models use relu activation instead of silu to significantly increase performance at a slight cost to accuracy.
• Dependencies: The demo requires librga.so. Ensure the correct librga library for your system is available and included in LD_LIBRARY_PATH.

For improved inference performance, this demo sets the RKNN model output format to RKNN_QUERY_NATIVE_NHWC_OUTPUT_ATTR.

When using this format, the output buffer is arranged in NHWC order. For example, if the original model output shape is [1, 255, 80, 80], the RKNN output shape will be [1, 80, 80, 255]. The post-processing logic in this demo is specifically optimized for this NHWC layout.

To use the RKNN Toolkit2 remote board debugging (连板) feature, you must ensure the board is running the correct rknn_server and runtime libraries. rknn_server is a background proxy service on the board that receives protocols from the PC via USB and executes the corresponding RKNPU runtime interfaces.

Key Components:

• rknn_server: The background proxy service.
• librknnrt.so: The standard RKNPU runtime library.
• librknnmrt.so: The runtime library specifically for RV1103/RV1106 platforms.

Version Requirement: If you are using models with dynamic shape input, both rknn_server and the RKNPU Runtime library must be version >= 1.5.0.

The demo is split into two main functional layers:

1. JAVA (com.rockchip.gpadc.demo): Responsible for reading camera input, calling JNI for inference, and displaying the results.
2. JNI: Responsible for calling rknnrt (RKNN Runtime) to perform the actual NPU inference.

RKNN-Toolkit2 and its associated runtime components support the following Rockchip series:

• RK3566/RK3568 Series
• RK3588 Series
• RK3562 Series
• RV1103/RV1106

Note: For older hardware such as RK1808, RV1109, RV1126, or RK3399Pro, you must use the original rknn-toolkit and rknpu repositories instead of this toolkit.

To perform inference with dynamic shapes using the RKNN C API, the application follows this lifecycle:

1. Initialize Context: Call rknn_init() to initialize the RKNN context.
2. Set Dynamic Shapes: Use rknn_set_input_shapes() to define the specific shape and layout for all model inputs.
3. Query Model Info: Use rknn_query() to retrieve current input/output information (shape, data type, and size).
4. Set Input Data: Use rknn_inputs_set() to provide the data pointer and size for the model inputs.
5. Run Inference: Execute the model using rknn_run().
6. Retrieve Outputs: Use rknn_outputs_get() to fetch the results (requesting float-type results where applicable).
7. Post-process: Process the output data to extract classification results and probabilities.
8. Cleanup: Release the RKNN context using rknn_release().