DOCSEARCHbyAlgolia
Submit Docs

LLM for Unity Documentation

repository·main·Indexed 22 days ago

https://github.com/undreamai/llmunity

A package (ai.undream.llm) that enables the integration and distribution of Large Language Models within the Unity engine. It features a local-first approach using LlamaLib (based on llama.cpp), support for .gguf models, and a RAG system for semantic search using the usearch library. Key components include the LLM and LLMAgent for AI character interaction, as well as support for remote LLM servers, grammar-based output restriction, and mobile deployment for iOS and Android.

Tokens
9.8K
Snippets
24
Records
40
Agent score
81%

What's inside LLM for Unity

  1. What is USearch?

    main

    USearch is a high-performance similarity search engine designed for vectors and text. It is a compact, single-file implementation that uses the HNSW algorithm. Key features include:

    • High Performance: Significantly faster indexing and search times compared to FAISS.
    • Memory Efficiency: Supports hardware-agnostic f16 and i8 quantization, and allows viewing large indexes from disk without loading them entirely into RAM.
    • Extensibility: Supports user-defined metrics with JIT compilation.
    • Versatility: Supports variable dimensionality, heterogeneous lookups, and near-real-time clustering.
    • Portability: Lightweight with minimal dependencies and native bindings for low latency.

  2. Restrict LLM output using Grammar

    main

    You can constrain the LLM's output using grammars (e.g., GBNF or JSON schema). This is useful for enforcing specific formats like JSON or for implementing function calling by restricting output to valid function names.

    Grammars can be:

    1. Edited directly in the Grammar field of the LLMAgent in the Inspector.
    2. Loaded via the Load Grammar button (Advanced options) using .gbnf or .json files.
    3. Set programmatically via the llmAgent.grammar property.
    ```c#nllmAgent.grammar = "your grammar here";

  3. Compare USearch with FAISS

    main

    USearch is designed to be a more maintainable, portable, and faster alternative to FAISS. While both use the HNSW algorithm, USearch offers several advantages:

    FeatureFAISSUSearch
    Indexing TimeSlowerUp to ~10x faster
    CodebaseLarge (~84K SLOC)Compact (~3K SLOC)
    Metrics9 fixed metricsAny metric (extensible)
    DependenciesRequires BLAS, OpenMPNone (light-weight)
    ID Types32-bit, 64-bit32-bit, 40-bit, 64-bit
    Python Binding Size~10 MB< 1 MB

  4. Use USearch for GIS and Mobile Applications

    main
    USearch provides Objective-C and Swift iOS bindings, making it suitable for mobile applications. A common use case is Geographic Information Systems (GIS), where 2-dimensional vectors representing latitude and longitude are used to find the closest Points of Interest (POIs) on a map. While the Haversine distance is common for these coordinates, the index can be extended to support high-dimensional vectors.

  5. Use User-Defined Functions (UDF) for custom metrics

    main

    USearch supports User-Defined Metrics, allowing you to go beyond standard Euclidean or Inner Product distances. You can implement custom metrics for specialized tasks like geospatial distance (e.g., Haversine) or composite embeddings from multiple models. Because USearch uses HNSW, vectors do not need to be identical in length; they only need to be comparable via the metric.

    For Python users, detailed implementation details for JIT and UDF can be found in the USearch Python SDK documentation.

  6. How semantic search and RAG work in LLM for Unity

    main

    LLM for Unity implements a Retrieval-Augmented Generation (RAG) system for fast similarity search using LLM embeddings and Approximate Nearest Neighbors (ANN) via the usearch library.

    The Workflow:

    1. Building the data: You provide text (phrases, paragraphs, or documents). The system optionally splits these into chunks and encodes them into embeddings using an LLM.
    2. Searching: When you provide a query, the system encodes the query and retrieves the most similar text inputs or chunks from the stored data.

    Key Components:

    • Search Methods: SimpleSearch (brute-force) or DBSearch (fast ANN method, recommended for most cases).
    • Chunking Methods: Methods for splitting inputs into smaller parts (e.g., by tokens, words, or sentences) to maintain consistent meaning within data parts.
    // Example of initializing RAG via code
RAG rag = gameObject.AddComponent<RAG>();
rag.Init(SearchMethods.DBSearch, ChunkingMethods.SentenceSplitter, llm);

  7. Set up a RAG system via the Unity Inspector

    main

    To set up semantic search without writing initialization code:

    1. Create a GameObject for your LLM (as described in the LLM setup guides).
    2. Create an empty GameObject and add the RAG script component.
    3. In the RAG component Inspector:
      • Set Search Type to your preferred method (DBSearch is recommended).
      • Set Chunking Type to your preferred splitting method (e.g., tokens, words, or sentences).
    4. Download a RAG model or load your own via the LLM GameObject.

  8. Build mobile apps for iOS and Android

    main

    When targeting mobile devices, use "Tiny models" (1-2 billion parameters) from the LLM model manager to ensure compatibility with mobile hardware.

    iOS Configuration Use default player settings.

    Android Configuration In Edit > Project Settings > Player > Other Settings, you must set:

    • Scripting Backend: IL2CPP
    • Target Architecture: ARM64

    Model Downloading To reduce initial app size, enable the Download on Build option to download models on the first launch. Use the following methods to manage the download process:

    • await LLM.WaitUntilModelSetup(): Waits until the model is fully ready.
    • await LLM.WaitUntilModelSetup(SetProgress): Accepts a callback to track progress (0.0 to 1.0).
    await LLM.WaitUntilModelSetup();

// Track progress
await LLM.WaitUntilModelSetup(SetProgress);

void SetProgress(float progress){
  string progressPercent = ((int)(progress * 100)).ToString() + "%";
  Debug.Log($"Download progress: {progressPercent}");
}

  9. Install and run LLM for Unity Samples

    main

    The package includes several sample projects (e.g., SimpleInteraction, FunctionCalling, RAG, ChatBot).

    To install a sample:

    1. Open the Unity Package Manager: Window > Package Manager.
    2. Select the LLM for Unity package.
    3. Navigate to the Samples tab.
    4. Click Import next to the desired sample.

    To run a sample:

    1. Open the Scene.unity file located within the sample's folder.
    2. Select the LLM GameObject in the scene.
    3. Assign your preferred LLM model in the Inspector.
    4. Save the scene and enter Play mode.

  10. Summary of LLMUnity v3 Migration Steps

    main

    Follow these steps to upgrade your project to v3:

    1. Update Class Inheritance: Change public class MyNPC : LLMCharacter to public class MyNPC : LLMAgent.
    2. Update Property Names: Replace agent.prompt with agent.systemPrompt. Remove usage of playerName and AIName.
    3. Update Method Calls: Update history methods to await agent.ClearHistory(), await agent.AddUserMessage(), and await agent.AddAssistantMessage().
    4. Update Grammar: Use agent.grammar for strings and agent.LoadGrammar() for files.
    5. Update Save/Load: Set the path via agent.save = "path" and use await agent.SaveHistory() / await agent.LoadHistory().
    6. Remove Chat Templates: Delete any calls to llm.SetTemplate() as templates are now auto-detected.

  11. Quick start: Set up an LLM and AI Character

    main

    To get an AI character working in your Unity scene, follow these steps:

    1. Setup the LLM Backend

    • Create an empty GameObject.
    • Add the LLM component to it.
    • Use the Download Model button to download a default model, or use the Load model button to load your own .gguf model.

    2. Create an AI Character

    • Create another empty GameObject for your character.
    • Add the LLMAgent component to it.
    • Define the character's behavior in the System Prompt field.
    • (Optional) If you have multiple LLM GameObjects, assign the specific one to the LLM field on the LLMAgent component.

  12. Migrate from LLMUnity v2 to v3

    main
    When upgrading to LLMUnity v3, note that the LLM backend (LlamaLib) has been rewritten. Most LLM functionality has moved to the backend. Key changes include the renaming of LLMCharacter to LLMAgent, the removal of manual chat template management (now auto-detected), and changes to how history and grammars are handled.