DOCSEARCHbyAlgolia
Submit Docs

flutter_gemma

repository·main·Indexed 20 days ago

https://github.com/denisovav/flutter_gemma

A modular plugin for integrating on-device Large Language Models (LLMs) into Flutter applications, supporting models like Gemma, Qwen, and Phi. It provides specialized packages for agentic skills (flutter_gemma_agent), system-provided AI models via AICore and Apple Foundation Models (flutter_gemma_builtin_ai), and text embeddings using LiteRT (flutter_gemma_embeddings) for local RAG implementations.

Tokens
96.6K
Snippets
240
Records
352
Agent score
64%

What's inside flutter_gemma

  1. Overview of Flutter Gemma

    main

    Flutter Gemma is a plugin that brings Google's lightweight Gemma language models and other on-device LLMs directly to Flutter applications. It enables advanced AI capabilities—such as text generation, multimodal vision, and audio processing—to run locally on user devices without relying on external servers. This ensures enhanced privacy and offline functionality.

    Key Capabilities:

    • Local Execution: Run models like Gemma, Qwen, DeepSeek, Phi, and SmolLM directly on the device.
    • Multimodal Support: Process text and image inputs (depending on the model).
    • Audio Support: Includes on-device Speech-to-Text (STT), Text-to-Speech (TTS), and full speech-to-speech loops via opt-in packages.
    • Cross-Platform: Supports iOS, Android, Web, macOS, Windows, and Linux.
    • Advanced AI Features: Supports function calling, on-device agents, RAG (Retrieval-Augmented Generation), and 'Thinking Mode' for reasoning models.

  2. Compare platform feature support

    main

    Flutter Gemma provides varying levels of support across Android, iOS, Web, and Desktop. Key differences include:

    • Text Generation & Function Calling: Fully supported on all platforms.
    • Multimodal (Image/Audio): Fully supported on Android, iOS, and Web (via MediaPipe .task). Desktop supports audio via .litertlm only.
    • Thinking Mode: Supported on Android, iOS, and Desktop (Gemma 4, DeepSeek, Qwen3, etc.), but not supported on Web.
    • NPU Acceleration: Supported on Android (.litertlm) and Windows (Intel LunarLake/PantherLake).
    • LoRA Support: Supported on Android, iOS, and Web; not supported on Desktop due to LiteRT-LM limitations.
    • Web Specifics: Web support is primarily via MediaPipe .task. The newer .litertlm web path is an early preview and lacks vision, audio, thinking, and function calling support.

  3. Platform support and GPU limitations for LiteRT-LM

    main

    LiteRT-LM supports various platforms with different backend capabilities:

    • Android: FFI (GPU via OpenCL, NPU via .litertlm on Qualcomm).
    • iOS: FFI (GPU via Metal on device; CPU on simulator).
    • macOS / Linux: FFI (GPU via Metal / Vulkan).
    • Windows: FFI (CPU + Intel NPU).
      • ⚠️ Warning: Discrete GPUs on Windows currently cause crashes due to upstream WebGPU/Dawn issues. Use PreferredBackend.cpu or .npu on Windows until fixed.
    • Web: via @litert-lm/core (CDN, early preview, text-only).

    Native platforms do not require manual setup; the native library is fetched at build time via Native Assets.

  4. Overview of genkit_hybrid architecture

    main

    The genkit_hybrid package is a provider-agnostic Dart package designed to combine multiple Genkit Model actions behind a single routing policy.

    Instead of calling different models (e.g., an on-device model vs. a cloud model) manually in your application logic, you use a single hybridModel factory. This factory returns an ordinary Model that your app can call via a standard ai.generate invocation.

    Core Mechanism:

    • Combinator Factory: hybridModel(branches, strategy) delegates requests to specific branches based on a RoutingStrategy.
    • Routing Strategy: The RoutingStrategy.route() method returns an ordered list of branch keys.
      • A single-element list performs a direct pick (pre-routing).
      • A multi-element list implements a fallback mechanism (try the first, then the next).
    • Delegation: The hybrid model calls the child model's public fn(request, context) directly.
    • Streaming Support: Streaming fallbacks are handled by wrapping the sendChunk callback, ensuring fallbacks only occur before the first token is emitted.

  5. Understand per-token sampler performance and behavior

    main

    When using PreferredBackend.gpu, the forward pass (prefill and decode) is accelerated by the GPU (Metal, DX12, or Vulkan). However, the per-token sampler (responsible for top-k, top-p, or argmax logic) runs on the CPU across all desktop platforms:

    • macOS & Windows: The sampler falls back to the CPU because the upstream libLiteRtTopKMetalSampler and libLiteRtTopKWebGpuSampler have incomplete C ABI exports.
    • Linux: The plugin falls back to the CPU to avoid issues with the process-static wgpu::Instance in the prebuilt sampler, which would reject subsequent engine_create calls.

    This fallback typically adds only 1–5 ms per token, which is negligible compared to the LLM generation time.

  6. Understand Genkit Hybrid error handling and fallback triggers

    main

    Genkit Hybrid distinguishes between transient failures (which trigger a fallback to the next branch) and permanent failures (which propagate immediately to the caller to avoid silent-failure anti-patterns).

    Fallback Triggers (Transient/Availability)

    Fallback occurs when a branch fails due to availability issues. This includes:

    • Any non-GenkitException throwable (e.g., network errors, timeouts, OOM).
    • A GenkitException with one of these statuses:
      • UNAVAILABLE
      • DEADLINE_EXCEEDED
      • RESOURCE_EXHAUSTED

    Immediate Propagation (Permanent)

    If an error is classified as permanent, the system will not attempt a fallback. This prevents wasting resources on requests that are guaranteed to fail (e.g., bad credentials). This includes GenkitException with these statuses:

    • INVALID_ARGUMENT
    • PERMISSION_DENIED
    • UNAUTHENTICATED
    • FAILED_PRECONDITION
    • NOT_FOUND

    Note: An empty or low-quality response is not treated as a failure in v1. Fallback is strictly based on availability, not output quality.

  7. Implement a custom RoutingStrategy

    main

    To control how requests are distributed across branches, implement the RoutingStrategy interface. The core method to implement is route(RoutingContext c), which must return a List<String> representing the ordered keys of the branches to be attempted.

    RoutingContext provides access to:

    • request: The current ModelRequest.
    • branchKeys: A set of available keys in the branches map.
    • isStreaming: A boolean indicating if the request is a streaming request.
    class MyStrategy implements RoutingStrategy {
  @override
  List<String> route(RoutingContext c) {
    // Example: Always prefer cloud if it's a complex request
    if (c.request.messages.length > 5) {
      return ['cloud'];
    }
    return ['onDevice', 'cloud'];
  }
}

  8. Performance and Memory Considerations

    main

    When using flutter_gemma, keep the following performance characteristics in mind:

    • Background Downloads: Network installations use the background_downloader package, supporting resume on interruption, background execution, and network change handling.
    • Progress Tracking: Real-time progress (0-100%) is available for NetworkSource. For Asset, Bundled, and File sources, a single 100% event is emitted.
    • Memory Management: Downloads are streamed to prevent loading full files into memory. The system uses lazy service initialization and a protected file registry to prevent accidental cleanup of external files.

  10. Understand ID handling in Qdrant Edge

    main

    The underlying qdrant-edge engine only supports PointId::NumId(u64) or PointId::Uuid. It does not support arbitrary strings directly.

    The shim uses ExtendedPointId::FromStr to parse strings into these types:

    • Numeric strings (e.g., "42") are converted to NumId(42).
    • UUID strings (e.g., "6ba7b810-9dad-11d1-80b4-00c04fd430c8") are converted to Uuid(...).
    • Any other string format will result in an error.

    Note for Flutter developers: The Dart-side wrapper QdrantVectorStoreRepository automatically handles arbitrary String id inputs by hashing them into UUIDv5 under a fixed namespace, allowing you to use standard strings in addDocument(...).

  11. Understand Flutter Gemma model file types and formatting

    main

    Flutter Gemma supports two main types of model files, distinguished by how chat templates are handled:

    Type 1: MediaPipe-Managed Templates

    These formats handle chat templates internally via MediaPipe.

    • .task files: Optimized for mobile (Android/iOS).
    • .litertlm files: LiteRT-LM format for Android, iOS, and Desktop.
    • Code usage: Use ModelFileType.task for both.

    Type 2: Manual Template Formatting

    These formats require you to manually format chat templates in your code.

    • .bin files: Standard binary format.
    • .tflite files: LiteRT format (formerly TensorFlow Lite).
    • Code usage: Use ModelFileType.binary for both.

    Platform Support Matrix

    FormatAndroidiOSWebDesktopUse Case
    .task✅✅✅❌Older models (Gemma3n, Gemma 3, DeepSeek, Qwen 2.5, Phi-4)
    .litertlm✅✅ ¹❌✅Newer models (Gemma 4, Qwen3, FastVLM + desktop)
    -web.task❌❌✅❌Web-specific builds (e.g. Gemma 4, Gemma3n)
    .bin✅✅✅❌Manual chat template formatting required
    .tflite✅✅✅✅Embeddings only (EmbeddingGemma, Gecko)

    ¹ Note: iOS .litertlm runs on the FFI engine. The Simulator is CPU-only due to Metal allocation limits.

  12. Configure the output format for a skill

    main

    To ensure the agent's responses are predictable and easy to parse or display in a UI, define a strict Output Format section within your skill file. This section should specify the exact headers and sentence constraints the model must follow.

    In the kitchen-adventure example, the required structure is:

    1. [Current Location Name]: A single sensory sentence.
    2. The Situation: 1-2 sentences describing NPCs or threats.
    3. What do you do?: A prompt or 3 suggested actions.
    ### [Current Location Name]

*A vivid, sensory description of the area...*

---

**The Situation:**
(Describe the immediate scene...)

**What do you do?**
(Provide a brief prompt or 3 suggested actions...)