repository·main·Indexed 23 days ago
https://github.com/software-mansion/react-native-executorchA declarative framework for running on-device AI models (LLMs, Computer Vision, Speech) in React Native using ExecuTorch. It supports the New React Native architecture and provides specialized resource fetchers for Expo and bare React Native projects, as well as hooks like useLLM for chat completions and useBackgroundBlur for WebRTC applications.
.pte file and run it instantly.When using Large Language Models (LLMs), the following terms describe the text processing lifecycle:
While the library provides several ready-to-use models, you can run your own AI models by exporting them to the .pte format.
To do this, you should follow the instructions provided in the PyTorch ExecuTorch Python API or use the optimum-executorch guide.
The SpeechToTextModule provides a direct interface to speech-to-text (STT) capabilities. While the useSpeechToText hook is recommended for React components, use this module for full control over the model's lifecycle in non-React contexts or advanced use cases. You can perform one-shot transcription (for files/clips) or streaming transcription (for live microphone input).
import { SpeechToTextModule, models } from 'react-native-executorch';
// Initialize the model
const model = await SpeechToTextModule.fromModelName(
models.speech_to_text.whisper_tiny_en(),
models.vad.fsmn_vad(),
(progress) => {
console.log(`Loading: ${progress * 100}%`);
}
);
// 1. One-shot transcription
const result = await model.transcribe(waveform);
// 2. Live streaming
model.streamInsert(audioChunk);
const stream = model.stream({ useVAD: true });
for await (const { committed, nonCommitted } of stream) {
// Handle results
}Once you have obtained an array of SegmentedInstance objects via forward(), you can use selector functions to pick specific instances based on user interaction without re-running the model.
selectByPoint): Finds the smallest instance whose mask covers the provided (x, y) coordinates. Ideal for tap-to-select.selectByBox): Finds the instance with the highest IoU (Intersection over Union) with a provided bounding box { x1, y1, x2, y2 }. Ideal for drag-to-outline.selectByText): Finds the instance with the highest cosine similarity between provided instanceEmbeddings and a textEmbedding. Ideal for search-by-description.useInstanceSegmentation.model.forward(image) once.forward again unless the image changes).import {
models,
useInstanceSegmentation,
selectByPoint,
selectByBox,
selectByText,
} from 'react-native-executorch';
const model = useInstanceSegmentation({
model: models.instance_segmentation.fastsam_x(),
});
try {
const instances = await model.forward(imageUri);
// Point: the smallest instance whose mask covers (x, y).
const pointMatch = selectByPoint(instances, x, y);
// Box: the instance with highest IoU with the prompt box.
const boxMatch = selectByBox(instances, { x1, y1, x2, y2 });
// Text: highest cosine similarity between text and per-instance image embeddings.
const textMatch = selectByText(instances, instanceEmbeddings, textEmbedding);
} catch (error) {
console.error(error);
}To track the progress of image generation (e.g., to drive a progress bar UI), provide an inferenceCallback within the model configuration object passed to fromModelName.
The callback is invoked at every denoising step, totaling numSteps + 1 calls. The callback yields the current step index.
The StyleTransferModule is a standard JavaScript object managed by the garbage collector. However, if you need to release the memory occupied by the module immediately rather than waiting for garbage collection, call the delete() method on the module instance.
Warning: Once delete() is called, you cannot use the forward() method on that instance again unless you reload the module.
Vision tasks (classification, detection, segmentation, etc.) are implemented using specialized hooks. All models are selected via a typed models registry using the pattern models.<category>.<model>({ quant?, backend? }).
Key Concepts:
backend that the model doesn't support results in a compile-time error.https://...), a local file URI (file://...), a base64 string, or a bundled asset via require('../assets/img.jpg'). Remote images are cached automatically.models object to access specific model architectures for each category.When working with models in React Native ExecuTorch, you will encounter these core concepts:
forward()) that defines the computation logic. In ExecuTorch, this logic is exported and compiled. Running inference in React Native involves invoking this compiled function with new inputs.To avoid garbled audio or incorrect playback speeds, you must adhere to these sample rate and channel requirements:
AudioContext with { sampleRate: 24000 } to avoid 'chipmunked' or slow audio.React Native ExecuTorch provides specialized hooks for different AI tasks. Use the following decision guide to select the appropriate hook:
useLLM (supports plain chat, Vision-Language Models with LFM2_VL_*, tool calling, and structured JSON output).useClassification (what is in the image).useObjectDetection (bounding boxes).useSemanticSegmentation (per-pixel class) or useInstanceSegmentation (per-instance).usePoseEstimation (keypoints).useOCR (horizontal) or useVerticalOCR (vertical/CJK).useStyleTransfer (artistic filters) or useTextToImage (Stable Diffusion).useImageEmbeddings (CLIP vectors).useSpeechToText (Whisper).useTextToSpeech (Kokoro).useVAD (detecting speech segments).useTextEmbeddings (sentence vectors).useTokenizer (HuggingFace-compatible).usePrivacyFilter (PII redaction)..pte files: useExecutorchModule (for models not covered by dedicated hooks).These terms describe the data structures and optimization techniques used in the library:
[3, 224, 224].