DOCSEARCHbyAlgolia
Submit Docs

yzma

repository·main·Indexed 18 days ago

https://github.com/hybridgroup/yzma

A Go library providing direct integration with llama.cpp for local LLM and VLM inference. It enables high-performance AI applications with hardware acceleration (CUDA, Metal, Vulkan, CPU) without requiring CGo or external model servers. The package includes a CLI for managing llama.cpp libraries and downloading GGUF models, as well as a Go API for text inference, embeddings, and multimodal vision tasks.

Tokens
10.8K
Snippets
53
Records
67
Agent score
67%

What's inside yzma

  1. Overview of yzma's llama.cpp coverage

    main
    yzma is a Go wrapper for llama.cpp that currently supports over 96% of its functionality. The project provides wrappers for various functional groups including backend management, model manipulation, vocabulary handling, context management, sampling, and multimodal (mtmd) capabilities. This high level of coverage allows developers to use llama.cpp features directly within Go applications.

  2. Explore yzma-based applications

    main

    Many specialized applications leverage yzma for local AI capabilities:

    • ImmyGo: An AI-first Go UI framework (built on Gio) that uses yzma, Ollama, and Anthropic for local AI capabilities.
    • Pingo: A Raspberry Pi Pico pin selector desktop app built with ImmyGo, featuring a chat interface powered by multiple LLM backends including yzma.
    • NornicDB: A high-performance graph database designed specifically for AI agents and knowledge systems.
    • wasmVision: A high-performance computer vision processing engine using vision models for machine learning.
    • Crush: An agentic coding tool built on Fantasy.
    • floop: Provides persistent memory for AI coding agents.
    • unch: A tool for semantic code search for code symbols and documentation.
    • Terminal agent: An LLM agent designed to assist users within the terminal environment.
    • Talking Heads From The Year 2053: Demonstrates physical AI running locally on the Arduino UNO Q.

  3. Explore the yzma ecosystem of tools and frameworks

    main

    The yzma ecosystem includes several high-level tools and frameworks designed to simplify AI integration in Go:

    • Kronk: Provides a high-level API that is compatible with the OpenAI API style, making it easy to swap models or providers.
    • Fantasy: A framework for building AI agents in Go. It is multi-provider and multi-model, and it supports Kronk (which is built on yzma).

    Use these tools if you want to avoid low-level yzma implementation details and prefer a more standardized or agent-centric development experience.

  4. Run Vision Language Action Models (VLA)

    main

    Vision Language Action Models (VLA) integrate vision, language, and actions. They are typically used to output structured data (like JSON) containing descriptions, bounding boxes, and distances based on an image and instruction. They use the same execution pattern as VLMs but are prompted for specific action-oriented outputs.

    go run ./examples/vlm/ -model ~/models/Pelican1.0-VL-3B.i1-Q4_K_M.gguf --mmproj ~/models/Pelican1.0-VL-3B.mmproj-Q8_0.gguf -p "What is in this picture? Provide a description, bounding box, and estimated distance for the llama in json format." -sys "You are a helpful robotic drone camera currently in flight." -image ./images/domestic_llama.jpg

  5. Install llama.cpp libraries with yzma install

    main

    Use the yzma install command to download pre-built llama.cpp binaries for your current operating system. You can specify the installation path, the version, and the hardware processor to use (e.g., CUDA for NVIDIA GPUs).

    # Install with default settings (uses YZMA_LIB env var)
yzma install

# Install to specific path
yzma install --lib /path/to/lib

# Install specific version with CUDA
yzma install --lib /path/to/lib --version b1234 --processor cuda

# Upgrade existing installation
yzma install --lib /path/to/lib --upgrade

# Using short flags
yzma install -l /path/to/lib -v b1234 -p cuda -u

  7. Install yzma and download models

    main

    To use yzma, first install the package following the platform-specific instructions in INSTALL.md. Once installed, you can use the yzma CLI tool to download GGUF models directly from URLs (e.g., Hugging Face).

    To download a specific model, use the yzma model get command with the -u flag followed by the direct download URL.

    $ yzma model get -u https://huggingface.co/QuantFactory/SmolLM2-135M-GGUF/resolve/main/SmolLM2-135M.Q4_K_M.gguf

  8. Install `llama.cpp` for Linux with Vulkan acceleration

    main

    To use Vulkan on Linux, install the Vulkan drivers (e.g., mesa-vulkan-drivers and vulkan-tools) and then run:

    yzma install --lib /path/to/lib --processor vulkan
    sudo apt install -y mesa-vulkan-drivers vulkan-tools
yzma install --lib /path/to/lib --processor vulkan

  10. Manual installation of `llama.cpp` libraries

    main

    If you prefer not to use the yzma installer, you can download prebuilt llama.cpp binaries manually from the llama.cpp releases or llama-cpp-builder releases (for Ubuntu CUDA/Vulkan arm64).

    Crucial Step: Set the YZMA_LIB environment variable Regardless of your OS, you MUST set the YZMA_LIB environment variable to point to the directory containing your downloaded library files (.so, .dylib, or .dll).

    Linux:

    export YZMA_LIB=/path/to/your/lib

    macOS:

    export YZMA_LIB=/path/to/your/lib

    Windows:

    set YZMA_LIB=C:\path\to\your\lib

    Note for Windows CUDA users: You must also download the cudart files from the same location as the other llama.cpp libraries.

    export YZMA_LIB=/home/ron/Development/yzma/lib

  12. Install `llama.cpp` for NVIDIA Jetson Orin

    main

    For NVIDIA Jetson Orin, ensure you have the latest Jetpack software installed.

    For CUDA:

    yzma install --lib /path/to/lib --processor cuda

    For Vulkan: First, update the GLIBC shared libraries:

    sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get install --only-upgrade libstdc++6

    Then run:

    yzma install --lib /path/to/lib --processor vulkan