LiteRT Documentation

The LiteRt Accelerator Test Suite (ats) is a tool for validating the functional correctness and measuring the performance of custom accelerator implementations in LiteRt.

It performs:

• Numeric Validation: Compares output tensors (activations) from the accelerator against the LiteRt standard CPU backend to ensure precision and correctness.
• Performance Metrics: Automatically captures and records metrics like latency.
• Automated Execution: Typically runs on target devices (e.g., Android) using a shell script wrapper that manages file transfers via adb.

Post-training quantization reduces model size and improves latency for CPU and hardware accelerators with minimal accuracy loss. Choose a method based on your target hardware:

Note: Requires TensorFlow 1.15 or higher.

LiteRT Tensor is a lightweight, Tensor-centric C++ library designed for high-performance tensor manipulation on mobile devices. It provides an expressive API for building computation graphs, simplifying pre- and post-processing, and authoring models that can be compiled into LiteRT (TFLite) models.

Key Capabilities:

• Expressive C++ API: Uses fluent interfaces and operator overloading.
• Backend Agnostic: Uses template mixins to switch between execution backends like TfLite.
• Zero-Copy Execution: Uses shared Buffer and TensorBuffer objects to avoid data duplication between pipeline stages (e.g., between CPU and GPU).
• LiteRT Integration: Allows compiling custom C++ graphs directly into LiteRT models.
• Dynamic Support: Supports dynamic shapes and multiple model entry points (signatures).

The LiteRT Objective-C API provides bindings (prefixed with LRT) for the LiteRT framework.

WARNING

This API is experimental and subject to change or removal in future releases without notice.

The @litertjs/wasm-utils package is a specialized utility designed to handle WebAssembly (Wasm) file loading specifically for @litertjs/core.

Note: This package is an internal dependency for @litertjs/core and is not intended for direct use by end-users.

The TensorFlow Lite Task Library provides optimized, task-specific APIs for common machine learning tasks like image classification, object detection, and natural language processing. It is designed for ease of use, allowing developers to implement inference with minimal code while handling complex data processing (e.g., converting raw data to model-ready formats) automatically.

Supported platforms include Java, C++, and Swift.

The example application utilizes the BertQuestionAnswerer API from the LiteRT Task library for natural language (NL). It uses a Mobile BERT model trained on the SQuAD dataset. By default, the application runs on the CPU, but it supports hardware acceleration via GPU or NNAPI delegates.

Key files in the implementation:

• BertQaHelper.kt: Initializes the question answerer and manages model and delegate selection.
• QaFragment.kt: Handles and formats the question-answering results.
• MainActivity.kt: Contains the primary organizing logic for the application.

The Sample Stable Delegate is an example implementation used for testing stable delegate APIs. It is designed to support a limited set of operations: addition, subtraction, multiplication, equality check, and while loops.

It utilizes the TfLiteOpaqueDelegate API, which allows for the delegation of nodes to alternative backends without exposing implementation details. This enables dynamic loading of delegates.

Critical Requirement: Because open-source TF Lite does not yet provide a binary stable interface between delegates and the runtime, any dynamically loaded opaque delegate must be built against the exact same version and commit as the TF Lite runtime to avoid undefined behavior.

The TensorFlow Lite Delegate Performance Benchmark (DPB) Android app is a wrapper around the TensorFlow Lite benchmark tool and MiniBenchmark. It is designed specifically for testing TensorFlow Lite delegates that implement a stable delegate ABI.

Developers use this app to evaluate both accuracy and latency to catch performance regressions. Unlike running binaries via adb shell, which can be affected by Android's scheduler behavior for background processes, this app provides a more faithful view of the runtime performance (accuracy and latency) that developers can expect when running delegates within a foreground Android Activity/Application.