DOCSEARCHbyAlgolia
Submit Docs

NobodyWho Documentation

repository·main·Indexed 20 days ago

https://github.com/nobodywho-ooo/nobodywho

An on-device AI inference engine for running Large Language Models (LLMs) locally and efficiently across multiple platforms. It provides offline-capable AI capabilities, including text-to-speech (TTS) using Kokoro, Pocket TTS, and Supertonic architectures, as well as multimodal input. The project includes an LM Evaluation Harness for running benchmarks on GGUF models and provides bindings for Kotlin, Python, Swift, React Native, Flutter, and Godot.

Tokens
187.5K
Snippets
644
Records
823
Agent score
75%

What's inside NobodyWho

  1. What is NobodyWho?

    main

    NobodyWho is a lightweight, open-source inference engine designed to run open-weights Large Language Models (LLMs) locally within your software. It is built on top of Llama.cpp and provides a simplified API for various tasks without requiring external infrastructure, Docker containers, or GPU servers.

    Key Capabilities:

    • LLM Inference: Run local models directly in your application.
    • Tool Calling: Integrate LLMs with external tools.
    • Output Configuration: Control and format model responses.
    • Real-time Streaming: Enable token-by-token streaming.
    • Embeddings: Generate vector embeddings from text.
    • Speech Synthesis: Perform Text-to-Speech (TTS) tasks.

  2. Overview of NobodyWho features

    main

    NobodyWho is a lightweight, open-source inference engine built on llama.cpp designed for running open-weights language models locally.

    Key capabilities include:

    • Local & Offline: No API keys or cloud infrastructure required.
    • Tool Calling: Supports passing normal Python functions for tool execution, automatically deriving grammar from function signatures.
    • Context Management: Uses conversation-aware preemptive context shifting to prevent crashes during long conversations.
    • Performance: GPU accelerated via Vulkan.
    • Model Compatibility: Works with any LLM in GGUF format.

  3. What is GBNF and Structured Output

    main

    Structured Output is a system that constrains a Large Language Model's (LLM) vocabulary to a specific format you define. This prevents common issues like models adding conversational filler (e.g., "Sure! Here is your JSON...") or failing to follow JSON syntax (e.g., missing quotes).

    GBNF (GGML Backus-Naur Form) is the language used to define these strict rules. A Grammar is the set of GBNF rules that dictate what valid output looks like. By using a grammar, you make it mathematically impossible for the model to generate anything outside of your defined constraints.

  4. What is RAG and how does it work in NobodyWho

    main

    Retrieval-Augmented Generation (RAG) is a technique used to provide an LLM with 'long-term memory' by allowing it to search through a database of information before generating a response. This prevents the model from hallucinating or forgetting details when the context window is limited.

    NobodyWho implements RAG using two primary methods:

    1. Embeddings: Converting sentences into vectors and using cosine similarity to find the closest matches. This is fast and cheap, suitable for large datasets.
    2. Reranking (Cross-encoders): A more accurate but expensive method where a model reads both the query and the document together to score relevance. This is used to sort and filter documents to ensure the most helpful information is provided to the LLM.

    In a typical RAG workflow, embeddings are often used for an initial fast pass to narrow down a large database, followed by a reranker to select the most precise documents for the final context.

  5. Understand the public API package structure

    main

    All consumer-facing types are located in the ai.nobodywho package.

    Important: Do not import types from uniffi.nobodywho directly; these are the raw generated bindings. Instead, use the high-level wrappers provided in ai.nobodywho.

    Public API Components:

    • Wrapper Classes: Chat, Model, Tool, Encoder, etc.
    • Type Aliases (in Exports.kt): SamplerConfig, Asset, ToolCall.
    • Sealed Class: Message (a hand-written wrapper in ai.nobodywho that mirrors uniffi.nobodywho.Message to avoid naming collisions with ai.nobodywho.Tool).

  6. Understand the NobodyWho Swift Architecture

    main

    The Swift package is organized into four distinct layers. Users should only interact with the top layer.

    1. NobodyWho (Public API): Hand-written Swift wrappers providing an idiomatic API (e.g., Chat, Model, Tool, Prompt). This is what you should import NobodyWho.
    2. NobodyWhoGenerated: Auto-generated Swift bindings from UniFFI (nobodywho.swift) containing raw FFI types like RustChat and RustModel. This layer is not directly importable.
    3. NobodyWhoNative: A prebuilt xcframework containing the compiled Rust static libraries for each platform.
    4. NobodyWhoMacros: A Swift compiler plugin providing the @DeclareTool macro.

    Mental Model: You use the high-level NobodyWho wrappers, which internally call the NobodyWhoGenerated bindings, which in turn call the NobodyWhoNative binary layer.

  7. Considerations for tool calling: Model support and context

    main

    When implementing tool calling, keep the following in mind:

    1. Model Compatibility: Not all models support tool calling. For reliable results, the Qwen family of models is recommended.
    2. Context Window: Tool calling fills the conversation context faster than standard chat. You may need to allocate a larger context size than usual to accommodate the tool definitions, parameter descriptions, and tool outputs.

  8. How samplers and the NobodyWhoSamplerBuilder work

    main

    A sampler determines how the model picks the next token from a probability distribution. You can use built-in presets or build a custom sampler using NobodyWhoSamplerBuilder for fine-grained control.

    NobodyWhoSamplerBuilder uses a chainable pattern consisting of two types of methods:

    1. Shift steps: These transform the probability distribution (e.g., .top_k(), .temperature()). You can chain as many as you want.
    2. Terminal steps: These finalize the chain into a NobodyWhoSamplerConfig. You must end your chain with exactly one terminal step.

    To ensure reproducible output, use the .seed(int) shift step. The seed is consumed by all random samplers in the chain (dist, mirostat_v1, mirostat_v2, and xtc), but is ignored by greedy.

    var cfg = NobodyWhoSamplerBuilder.new() \
    .top_k(40) \
    .temperature(0.8) \
    .seed(42) \
    .dist() # Terminal step
chat.set_sampler_config(cfg)

  9. Local Model Folder Format

    main

    When using a local directory as the source, point to the top-level model folder and ensure you provide the matching architecture. The folder structure should mirror the layout of the corresponding Hugging Face repository.

    Supertonic Requirement: The top-level folder for Supertonic must include both the onnx/ and voice_styles/ directories. Download the model files using the same relative paths as the official repository.

  10. Use arguments in tools with automatic schema generation

    main

    NobodyWho can automatically generate a JSON schema for your tools based on Godot type hints. This allows the model to pass arguments to your functions.

    Supported Primitive Types:

    • int
    • float
    • bool
    • String/string
    • Array/string[]

    Requirements:

    • You must ensure all parameters are listed in the function signature.
    • The return type must be explicitly defined (e.g., -> String).

    Note: Descriptions for individual arguments cannot be extracted automatically; for maximum precision, you may need to provide a manual schema.

    func heal_player(amount: int) -> String:
    GameManager.get_local_player().heal(amount)
    return "Healed %d HP" % amount

add_tool(heal_player, "Heals the local player by a number of hit-points")

  11. Supported model path formats in Python

    main

    The model_path argument used by Chat, download_model, and related functions supports three formats for specifying GGUF models:

    1. HuggingFace reference: Uses the prefix hf: or huggingface: (case-insensitive, // is optional). These are downloaded and cached on first use. Examples: hf:owner/repo/file.gguf, huggingface://owner/repo/file.gguf.
    2. HTTPS URL: A direct link to a .gguf file. These are downloaded and cached on first use. Example: https://example.com/model.gguf.
    3. Local path: A path to a file already on your disk. This is used as-is. Example: ./model.gguf.

  12. Understand Embeddings and Semantic Search concepts

    main

    Embeddings allow for semantic text comparison, meaning the system can understand the intent of a sentence even if the exact words do not match (e.g., "Hand me the red potion" vs "Give me the scarlet flask").

    Key terminology:

    • Embedding Model (GGUF): A *.gguf file trained to convert text into numerical vectors.
    • Embedding / Vector: A list of numbers representing the meaning of a piece of text.
    • Cosine Similarity: A mathematical comparison between two embeddings that returns a value between 0 (completely different) and 1 (identical meaning).
    • Semantic Search: The process of finding text based on meaning rather than keyword matching.