llama.rn

The RNLlamaExample project demonstrates several core LLM capabilities. You can find the implementation details for each feature in the following files:

• Simple Chat: Basic text generation interface (SimpleChatScreen.tsx)
• Multimodal: Image/audio analysis and visual/audio question answering (MultimodalScreen.tsx)
• Tool Calling & MCP: Function calling with custom tools and Model Context Protocol integration (ToolCallsScreen.tsx)
• Embedding: Vector embeddings, semantic search, and reranking (EmbeddingScreen.tsx)
• Parallel Decoding: Concurrent request processing using multiple parallel slots (ParallelDecodingScreen.tsx)
• Text-to-Speech: Local voice synthesis using OuteTTS (TTSScreen.tsx)
• Model Info: Diagnostics and system information (ModelInfoScreen.tsx)

Model configurations and constants are located in src/utils/constants.ts.

Multimodal support allows models to process images and audio alongside text.

Workflow:

1. Initialize Context: Create the base model context. Crucial: You must set ctx_shift: false to maintain media token positioning.
2. Initialize Multimodal: Call context.initMultimodal with the path to your mmproj (projector) file.
3. Verify: Use context.isMultimodalEnabled() and context.getMultimodalSupport() to check capabilities.
4. Inference: Pass media via structured messages (e.g., image_url or input_audio).
5. Cleanup: Call context.releaseMultimodal() when finished.

Supported Formats:

• Images: JPEG, PNG, BMP, GIF, TGA, HDR, PIC, PNM, or Base64 data URLs. (Local file paths or Base64; HTTP URLs not yet supported).
• Audio: WAV, MP3, or Base64 data URLs. (Local file paths or Base64; HTTP URLs not yet supported).

const context = await initLlama({
  model: 'path/to/model.gguf',
  ctx_shift: false, // Required for multimodal
})

const success = await context.initMultimodal({
  path: 'path/to/mmproj.gguf',
  use_gpu: true,
})

if (success) {
  // Use context.completion with media content...
}

The parallel property provides a namespace for performing non-blocking operations using a queue. This allows you to trigger multiple requests (like completions, embeddings, or reranking) without blocking the main thread.

Key operations include:

• enable(config?): Enables parallel processing. config can include n_batch and n_parallel.
• disable(): Disables parallel processing.
• getStatus(): Returns the current ParallelStatus.
• subscribeToStatus(callback): Subscribes to status updates. Returns a remove function to unsubscribe.
• completion(params, onToken?): Queues a completion request. Returns a Promise containing a requestId and a stop function to cancel the specific request.
• embedding(text, params?): Queues an embedding request.
• rerank(query, documents, params?): Queues a reranking request.

// Example: Enabling parallel processing and running a completion
await context.parallel.enable({ n_parallel: 2 });

const { requestId, stop, promise } = await context.parallel.completion(
  { prompt: 'Hello, how are you?' },
  (id, token) => console.log(`Token received: ${token.text}`)
);

const result = await promise;
// To cancel this specific request:
await stop();

To prevent malicious users from injecting special tokens (like <|end|>) into a chat template, the engine uses jinja::string to wrap std::string and preserve origin metadata via an is_input flag.

When is_input is true, downstream applications (like llama-server) can identify that a string originated from user input and treat it differently than template-generated tokens.

How to enable Input Marking

You can activate this feature in two ways:

1. Via JSON conversion: Call global_from_json with the parameter mark_input = true.
2. Manually: Manually invoke value.val_str.mark_input() when creating string values.

Flag Propagation Rules

• One-to-one transformations (e.g., uppercase, lowercase): The is_input flag is preserved.
• One-to-many transformations (e.g., split): The resulting parts are marked is_input = true only if ALL input parts were marked is_input.
• Many-to-one transformations (e.g., join): Follows the same logic as one-to-many.
• Concatenation: String parts are appended as-is, and the new string preserves the is_input flag based on its components.

iOS

After installation, run npx pod-install. To build from source instead of using pre-built binaries, set RNLLAMA_BUILD_FROM_SOURCE to 1 in your Podfile.

Android

If Proguard is enabled, add the following rule to android/app/proguard-rules.pro:

-keep class com.rnllama.** { *; }

To build from source instead of using pre-built libraries, set rnllamaBuildFromSource to true in android/gradle.properties.

Android GPU/NPU Acceleration

OpenCL (GPU)

• Target devices with OpenCL-capable GPUs (e.g., Qualcomm Adreno 700+).
• Add <uses-native-library android:name="libOpenCL.so" android:required="false" /> to your app manifest.
• Set n_gpu_layers > 0 in initLlama.

Hexagon (NPU - Experimental)

• Target devices with HTP (Qualcomm SM8450+ / Snapdragon 8 Gen 1 or newer).
• Add <uses-native-library android:name="libcdsprpc.so" android:required="false" /> to your app manifest.
• Pass devices: ['HTP0'] (or HTP*) to initLlama.
• Set n_gpu_layers > 0 in initLlama.