DOCSEARCHbyAlgolia
Submit Docs

QVAC Local-First AI SDK

repository·main·Indexed 43 days ago

https://github.com/tetherto/qvac

A local-first AI SDK and model provider for running LLMs, speech, vision, and generative models on Linux, macOS, Windows, Android, and iOS without cloud dependencies. Includes the @qvac/ai-sdk-provider for integration with the Vercel AI SDK, supporting both external and managed runtime modes for OpenAI-compatible local inference.

Tokens
784.5K
Snippets
2.2K
Records
3.2K
Agent score
56%

What's inside QVAC

  1. Overview of @qvac/diffusion-cpp

    main

    The @qvac/diffusion-cpp package (v0.6.0) provides a cross-platform JavaScript API for diffusion model inference using stable-diffusion.cpp. It is designed for Bare runtime applications and enables local image generation (txt2img and img2img) on desktop and mobile platforms with CPU/GPU acceleration (Metal, Vulkan, OpenCL).

    Key Capabilities:

    • Cross-platform support: macOS, iOS, Linux, Android, and Windows.
    • Local Model Management: Uses disk-local model files. Users must provide absolute file paths for models (e.g., model, clipL, vae) via the constructor.
    • Progress Tracking: Provides step-by-step generation progress via callbacks.
    • Quantized Model Support: Works with GGUF, safetensors, and checkpoint formats.
    • Supported Models: SD2.x, SDXL, SD3, and FLUX.2 [klein].

  2. Overview of @qvac/asr-ggml

    main

    The @qvac/asr-ggml package provides a cross-platform JavaScript API for speech-to-text (STT) inference using Whisper and Parakeet engines. It is designed to run on the Bare Runtime and supports high-performance audio processing with GPU acceleration.

    Key Capabilities

    • Streaming Transcription: Real-time audio chunk processing with segment emission.
    • Silero VAD: Integrated Voice Activity Detection (required for runStreaming()) via the files.vadModel configuration.
    • Hot-Reload: Update configuration (like language or VAD parameters) at runtime without destroying the instance, unless a full model reload is triggered by context-level changes.
    • GPU Acceleration: Supports Metal (macOS/iOS) and Vulkan (Linux/Android/Windows) with automatic CPU fallback.

    Supported Platforms

    PlatformArchitectureMin VersionGPU Support
    macOSarm64, x6414.0+Metal
    iOSarm6417.0+Metal
    Linuxarm64, x64Ubuntu-22+Vulkan (CPU fallback)
    Androidarm6412+Vulkan (CPU fallback)
    Windowsx6410+Vulkan (CPU fallback)

  3. Overview of @qvac/translation-nmtcpp

    main

    The @qvac/translation-nmtcpp package provides offline neural machine translation (NMT) capabilities for QVAC-powered mobile and desktop applications. It allows for text translation between language pairs using multiple optimized backends.

    Key Capabilities:

    • Multi-backend support: Uses GGML-based custom NMT (IndicTrans2) and Mozilla Bergamot.
    • High-level JS API: Provides an ergonomic interface for NMT inference.
    • Local Model Management: Users supply their own local model files via the constructor.
    • Performance Features: Supports batch translation (Bergamot), GPU acceleration (Metal on Apple, Vulkan/OpenCL), and beam search decoding.
    • Cross-platform: Supports macOS, iOS, Linux, Android, and Windows.

  4. Overview of QVAC AI capabilities

    main

    QVAC provides a wide range of local AI capabilities through a unified interface. Key capabilities include:

    • Text Generation & Embeddings: LLM inference and vector embedding generation via Fabric LLM.
    • RAG: Out-of-the-box retrieval-augmented generation workflows.
    • Fine-tuning: Adapting LLMs via LoRA.
    • Multimodal: LLM inference over text, images, and other media.
    • Generative Media:
      • Image: Text-to-image and image-to-image via a customized Diffusion backend.
      • Video: Text-to-video and image-to-video via a customized Diffusion backend.
      • Music: Text/lyrics-to-music via ACE-Step or MiniMax-Music3.
    • Audio & Speech:
      • Transcription (ASR): Via customized Whisper or NVIDIA Parakeet.
      • Text-to-Speech (TTS): Via a customized GGML backend.
      • Voice Assistant: A real-time pipeline combining transcription, text generation, and speech synthesis.
    • Specialized Tasks:
      • Translation: Neural machine translation (NMT) via Fabric LLM and Bergamot.
      • BCI: Brain–computer interface transcription via customized Whisper.
      • VLA: Vision-language-action for robot control via customized GGML.
      • OCR: Optical character recognition via ONNX Runtime.
      • Image Classification: Labeling images with confidence scores via customized GGML.

  5. Overview of @qvac/llm-llamacpp

    main

    @qvac/llm-llamacpp is a cross-platform npm package designed for Large Language Model (LLM) inference within the Bare runtime. It provides a high-level JavaScript API that wraps llama.cpp, enabling local LLM execution on desktop and mobile platforms with support for CPU/GPU acceleration.

    Key Capabilities

    • Cross-platform support: macOS, Linux, Windows, iOS, and Android.
    • Streaming responses: Supports token-by-token output via async iterators or callbacks.
    • Model Types: Supports text-only and multimodal (vision + text) models.
    • Flexible Model Ownership: The caller is responsible for providing absolute file paths to model files (filesystem, P2P, HTTP, etc.); the package does not perform automatic downloads.
    • GPU Acceleration: Supports Metal, Vulkan, and OpenCL.
    • Sharded Loading: Supports loading sharded GGUF models by passing the .tensors.txt companion and all shards in order.
    • Multi-GPU: Supports pipeline ('layer') and experimental tensor ('tensor') parallelism via split-mode/tensor-split configurations.

  6. Overview of Transcription capabilities

    main

    QVAC provides Automatic Speech Recognition (ASR) using two primary inference engines:

    1. qvac-fabric-speech.cpp: Uses whisper.cpp-compatible models. Set modelType: "whisper" to load.
    2. NVIDIA Parakeet: Uses the GGML-based parakeet-cpp engine. Set modelType: "parakeet-transcription" to load.

    Key Features:

    • Batch Transcription: Use transcribe() to get a single string containing the full transcription.
    • Streaming Transcription: Use transcribeStream() to receive real-time text chunks via a duplex session.
    • Audio Input: Accepts audioChunk as either a file path (string) or an in-memory audio buffer.
    • Parakeet Variants: Supports multilingual (TDT), English-only (CTC), Unified (batch + streaming), Indic-language, speaker diarization (Sortformer), and end-of-utterance detection (EOU).

  7. Overview of diffusion-cpp entry points

    main

    The diffusion-cpp package is a native C++ addon for image, video, interactive world-walk, and ESRGAN inference, built for the Bare Runtime. It provides four distinct JS entry points depending on your use case:

    • @qvac/diffusion-cpp: Use ImgStableDiffusion for text-to-image, image-to-image, FLUX.2 reference fusion, and optional ESRGAN upscaling.
    • @qvac/diffusion-cpp/video (or named export): Use VideoStableDiffusion for Wan and LTX text-to-video or image-to-video.
    • @qvac/diffusion-cpp/world: Use WorldStableDiffusion for ABot-World interactive walks (block-by-block generation via keyboard input).
    • @qvac/diffusion-cpp (named export): Use EsrganUpscaler for standalone PNG/JPEG upscaling.

  8. Overview of inference-addon-cpp

    main

    What is inference-addon-cpp?

    inference-addon-cpp is a header-only C++20 library designed to provide the foundational infrastructure for building high-performance inference addons on the Bare runtime. It abstracts the complexities of bridging JavaScript and C++, managing asynchronous job processing, model lifecycles, and streaming model weight loading.

    Core Purpose

    It allows developers to implement native inference logic by fulfilling a specific interface (model::IModel) while the framework handles:

    • Job Execution: Submitting jobs from JavaScript without blocking the calling thread.
    • Asynchronicity: Delivering output via callbacks to ensure non-blocking behavior.
    • Threading & Memory: Managing multi-threading, event loop integration (via libuv), and memory management.
    • Weight Loading: Efficiently streaming large model files (e.g., sharded GGUF models).

    Ecosystem Position

    The library acts as a bridging layer between the Bare runtime and specific Inference Addons (like @qvac/llm-llamacpp or @qvac/transcription-whispercpp).

    Data Flow: JavaScript Application $\rightarrow$ Bare Runtime $\rightarrow$ inference-addon-cpp $\rightarrow$ Specific Addon $\rightarrow$ CPU/GPU Backend

  9. Overview of @qvac/tts-ggml

    main

    The @qvac/tts-ggml module provides text-to-speech (TTS) capabilities for QVAC, powered by the qvac-tts.cpp GGML library. It operates in-process with a persistent native engine, meaning GGUF models, S3Gen preloads, and voice-conditioning tensors are loaded once and reused across multiple synthesis calls for efficiency.

    GPU Acceleration GPU acceleration is opt-in. By default, the engine uses the CPU. To enable GPU support, set config: { useGPU: true } in the constructor options. Supported backends include:

    • Metal (macOS/iOS)
    • CUDA (NVIDIA Linux x64)
    • Vulkan / OpenCL (Linux/Windows/Android)

    If multiple backends are available, you can pin a specific one using the TTS_CPP_GPU_BACKEND environment variable with values: cuda, vulkan, metal, or opencl.

  10. Overview of @qvac/model-fit

    main

    The @qvac/model-fit addon is a memory-fit preflight tool for QVAC. It wraps the llama.cpp common_fit_params C API to project whether a GGUF model will fit within available device memory without actually loading any weights.

    It works by simulating allocations (no_alloc) using GGUF metadata to build a worst-case compute graph. This allows the fitter to estimate the size of the model, KV/context, and compute buffers. It then iteratively calculates an offload plan by reducing context or moving tensors off the GPU until the projection fits within a specified per-device free-memory margin.

  11. Overview of @qvac/embed-llamacpp

    main

    @qvac/embed-llamacpp is a cross-platform npm package designed for the Bare runtime that provides text embedding generation. It wraps llama.cpp in a JavaScript-friendly API, allowing for local embedding model execution on desktop and mobile devices with support for CPU and GPU acceleration (Metal, Vulkan, OpenCL).

    Key Capabilities:

    • High-level JS API: Easy integration for embedding generation.
    • Direct File Streaming: Uses bare-fs to stream model files directly from disk, avoiding unnecessary download or loader layers.
    • Batch Processing: Optimized for high-throughput by processing multiple texts in a single forward pass.
    • Quantized Model Support: Supports GGUF format (Q2-Q8, 1-bit variants).
    • Sharded Loading: Supports loading sharded models by passing every shard path and the .tensors.txt companion file.
    • Multi-GPU Support: Supports pipeline parallelism using 'layer' split mode (Note: 'row' mode is deprecated and rejected).

  12. Overview of supported AI tasks

    main

    The QVAC SDK supports a wide range of AI capabilities across different modalities:

    • Text: Generation, embeddings, RAG (Retrieval-Augmented Generation), fine-tuning (via LoRA), and translation.
    • Audio: Transcription (ASR), Text-to-Speech (TTS), and real-time Voice Assistant pipelines.
    • Vision/Multimodal: Multimodal LLM inference, OCR (Optical Character Recognition), Image Classification, and VLA (Vision-Language-Action) for robotics.
    • Media Generation: Image generation, Video generation, and Music generation.
    • Specialized: BCI (Brain-Computer Interface) transcription for decoding neural signals.