DOCSEARCHbyAlgolia
Submit Docs

ExecuTorch

repository·main·Indexed 11 days ago

https://github.com/pytorch/executorch

A unified solution for deploying PyTorch models on-device across mobile, embedded, and edge devices, ranging from microcontrollers to high-end smartphones. It provides a lightweight runtime and a workflow for exporting, optimizing, and partitioning models for specialized hardware backends, including Arm Cortex-M, Ethos-U, Apple Core ML, and Metal.

Tokens
362.3K
Snippets
898
Records
1.4K
Agent score
88%

What's inside ExecuTorch

  1. Overview of ExecuTorch WebGPU Backend

    main

    The ExecuTorch WebGPU backend allows running ExecuTorch models on GPUs via the WebGPU standard. It compiles delegated subgraphs into WGSL compute shaders and executes them using Dawn and the Tint WGSL compiler.

    Key Capabilities:

    • Native Execution: Targets Metal and Vulkan in native builds.
    • Browser Execution: Uses Emscripten and emdawnwebgpu to compile the same execution path to WebAssembly for browsers.
    • Dynamic Shapes: Supports runtime dimensions using SymInt arithmetic and per-operator resize hooks.
    • High Performance: Optimized for LLMs with techniques like shared-memory tiled GEMM, FlashDecoding, and 4-bit quantization.

  3. Overview of ExecuTorch

    main
    ExecuTorch is PyTorch's solution for efficient AI inference on edge devices, ranging from high-end mobile phones to constrained microcontrollers. It provides a lightweight runtime designed for portability and performance across diverse platforms, supporting hardware acceleration via CPU, GPU, NPU, and DSP. Developers can use familiar PyTorch tools throughout the entire workflow, from model authoring to deployment.

  4. What is React Native Llama (rnllama)?

    main

    rnllama is a demonstration application that allows developers to run LLaMA language models directly on mobile devices using the ExecuTorch runtime. It provides a React Native UI layer for interacting with the model.

    Key characteristics:

    • On-device execution: Runs LLaMA models locally on the device.
    • Platform: Currently optimized for iOS.
    • Inspiration: The setup is based on the etLLM app architecture.

  5. Overview of ExecuTorch Benchmarking Infrastructure

    main

    The ExecuTorch benchmarking infrastructure is an experimental system designed to measure model performance on Android and iOS devices. It is built on the Nova reusable mobile workflow and allows for reproducible performance measurements across different hardware and backends.

    Key Capabilities:

    • Model Support: Currently supports models like MobileNetV2 (Hugging Face integration is planned).
    • Device Support: Benchmarking is performed on devices such as Apple iPhone, Google Pixel, and Samsung Galaxy.
    • Backend Delegates: Supports delegates including XNNPACK, Apple CoreML, MPS, and Qualcomm QNN.
    • Benchmark Apps: Provides generic Android (android/benchmark/) and iOS (apple/Benchmark/) apps for offline performance measurement.
    • Performance Monitoring: Results are stored in a database and visualized via a dashboard to track trends and detect regressions.

  6. Overview of the Arm Ethos-U Image Classification Example

    main

    This example demonstrates an end-to-end workflow for using the Arm backend in ExecuTorch, supporting both ahead-of-time (AoT) and runtime flows. It is designed to show how to take a model from fine-tuning to deployment on hardware like Corstone-320 (Ethos-U85-256).

    The workflow is split into two primary phases:

    1. Model Export: Fine-tuning a DeiT-Tiny model on the Oxford-IIIT Pet dataset, performing INT8 quantization, and lowering the model to Ethos-U to generate a .pte ExecuTorch program.
    2. Runtime Deployment: Building a bare-metal image-classification application that embeds the exported .pte program and a sample image, then running it on a Corstone-320 Fixed Virtual Platform (FVP).

  7. Overview of Cadence Xtensa Backend for ExecuTorch

    main

    The Cadence backend allows ExecuTorch to run on Cadence Xtensa DSPs, which are optimized for power-limited embedded devices. It supports several DSP families:

    • HiFi Audio DSPs (HiFi4/HiFi5): Optimized for audio processing, speech recognition, and wake word detection.
    • Fusion G3 DSPs: General-purpose AI acceleration.
    • Vision P-Series DSPs: Specialized for computer vision and CNN workloads.

    The integration uses custom operators, compiler passes, and a custom quantizer to represent activations and weights as uint8 instead of float. It also utilizes optimized kernels with Xtensa intrinsics and the HiFi4 Neural Network Library (nnlib) for runtime acceleration.

  8. Overview of the LLM Runner Framework

    main
    The LLM Runner framework for ExecuTorch provides high-level C++ APIs designed for running Large Language Models (LLMs). It is built on a modular architecture using dependency injection to ensure efficient resource management and a clean separation of concerns. The framework is divided into two primary runner types depending on the model's capabilities:

  9. Overview of NXP eIQ Neutron Backend

    main

    The NXP eIQ Neutron Backend is an ExecuTorch implementation designed for the eIQ® Neutron Neural Processing Unit (NPU). It provides ML acceleration for edge AI tasks including anomaly detection, speech recognition, image classification, object detection, facial recognition, image segmentation, and generative AI (LLMs, SLMs, and TTS).

    It supports various neural network types such as CNN, RNN, TCN, and Transformer networks. The backend is currently at Prototype Quality and is subject to significant changes.

  10. Overview of Swin2SR Super-Resolution Example (VGF)

    main

    This example demonstrates the workflow for exporting a Swin2SR image super-resolution model specifically for the Arm VGF backend. The workflow is designed for host-only execution, using the generic executor_runner binary to run the exported model on a host machine rather than a specific device-side VGF runtime application.

    The workflow is divided into two main phases:

    1. Model Export: Preparing assets, performing FP/INT8 export, PTQ (Post-Training Quantization) calibration, and generating the .pte file.
    2. Runtime: Running the .pte file on the host using executor_runner and post-processing the output tensor back into an image.

  11. Overview of ExecuTorch Arm® Backend

    main

    The ExecuTorch Arm® Backend provides an ahead-of-time (AOT) flow to deploy PyTorch models to Arm hardware by lowering them to a TOSA (Tensor Operator Set Architecture) representation.

    Supported targets include:

    • Arm® Ethos™-U55/65/85: Compiled using the Ethos-U Vela compiler.
    • VKML (Vulkan ML extensions): Uses a format containing SPIR-V™ ML operators for Vulkan Machine Learning (VKML) devices.

    The AOT flow supports development on Linux aarch64, Linux x86_64, and macOS with Apple Silicon. Deployment paths include bare metal builds for Arm® Cortex®-M with Ethos-U acceleration or Linux target support for VGF-capable targets.

  12. Available hardware acceleration backends for ExecuTorch

    main

    ExecuTorch supports various hardware acceleration backends for embedded systems, categorized by the type of processor being used:

    DSP Acceleration

    • Cadence Xtensa DSP: Use the embedded-cadence backend for Cadence Xtensa DSP processors.

    CPU Acceleration

    • Arm Cortex-M: Use the embedded-arm-cortex-m backend for Arm Cortex-M processors utilizing CMSIS-NN acceleration.

    NPU Acceleration

    • Arm Ethos-U NPU: Use the embedded-arm-ethos-u backend for Arm Ethos-U NPU acceleration.
    • NXP eIQ Neutron: Use the embedded-nxp backend for NXP eIQ Neutron acceleration.