DOCSEARCHbyAlgolia
Submit Docs

LLM Rust Library

repository·main·Indexed 43 days ago

https://github.com/graniet/llm

A unified Rust library (v1.3.8) providing a builder-style API for interacting with multiple Large Language Model backends, including OpenAI, Anthropic, Ollama, Google, DeepSeek, xAI, Phind, and Elevenlabs. It features a CLI tool, support for chaining backends, agentic workflows, multimodal capabilities (Vision, Speech-to-Text, Text-to-Speech), and the ability to serve backends as an OpenAI-standard REST API.

Tokens
64.6K
Snippets
193
Records
283
Agent score
57%

What's inside LLM

  1. Overview of LLM features

    main

    LLM is a unified Rust library for interacting with multiple LLM providers through a single API.

    Core Capabilities:

    • Unified API: Use ChatProvider and CompletionProvider traits to interact with providers like OpenAI, Anthropic, Ollama, Google, and more.
    • Chaining: Create multi-step chains where different backends are used at different stages.
    • Advanced Prompting: Use templates for variable injection and structured output via JSON schemas.
    • Resilience: Built-in support for retries and exponential backoff.
    • Evaluation: Score LLM outputs and perform parallel evaluation across multiple providers to select the best response.
    • Agentic Workflows: Build reactive agents with shared memory, roles, and validation.
    • Multimodal & Audio: Support for Vision (images), Speech-to-Text (transcription), and Text-to-Speech.
    • Memory: Manage conversation history using sliding windows or shared memory.

  2. Install the LLM crate

    main

    To use LLM in your Rust project, add it to your Cargo.toml. You must enable the specific features for the backends you intend to use (e.g., openai, anthropic, ollama).

    [dependencies]
llm = { version = "1.3.8", features = ["openai", "anthropic", "ollama", "deepseek", "xai", "phind", "google", "groq", "mistral", "elevenlabs"] }

  3. Use the LLM builder pattern to configure a backend

    main

    The llm crate uses a builder pattern via LLMBuilder to initialize LLM clients. You specify the provider using LLMBackend, set your credentials, and configure model parameters like model, max_tokens, and temperature.

    Supported backends mentioned in the documentation include:

    • OpenAI
    • Anthropic
    • Ollama
    • DeepSeek
    • xAI
    • Google
    • Phind
    • Elevenlabs
    use llm::{
    builder::{LLMBackend, LLMBuilder},
};

let llm = LLMBuilder::new()
    .backend(LLMBackend::OpenAI)
    .api_key("your_api_key".into())
    .model("gpt-4.1-nano")
    .max_tokens(512)
    .temperature(0.7)
    .normalize_response(true)
    .build()
    .expect("Failed to build LLM");

  4. Serve LLM backends as a REST API

    main

    You can expose LLM backends (or multi-step chains of backends) as a REST API that follows the OpenAI standard format. To do this, you must enable the api feature in your Cargo.toml.

    [dependencies]
llm = { version = "1.3.8", features = ["openai", "anthropic", "ollama", "deepseek", "xai", "phind", "google", "groq", "api", "mistral", "elevenlabs"] }

  5. Add AWS Bedrock dependency to Cargo.toml

    main

    To use the AWS Bedrock backend, add fork-llm with the bedrock feature enabled and tokio for the async runtime to your Cargo.toml.

    [dependencies]
fork-llm = { version = "0.1.0", features = ["bedrock"] }
tokio = { version = "1.0", features = ["full"] }

  6. Configure AWS credentials for Bedrock

    main

    The Bedrock backend requires AWS credentials. You can configure them using one of the following methods:

    • Environment Variables: Set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_REGION.
    • AWS Profile: Set AWS_PROFILE to use a specific profile from your AWS config.
    • AWS Config File: Create or edit ~/.aws/credentials with your keys and region.
    • AWS CLI: Use the AWS CLI to authenticate; the SDK will pick up the credentials automatically.
    # Environment Variables
export AWS_ACCESS_KEY_ID=your_key
export AWS_SECRET_ACCESS_KEY=your_secret
export AWS_REGION=us-east-1

# AWS Profile
export AWS_PROFILE=your-profile

  7. Use the llm-cli to interact with LLM backends

    main

    The llm-cli provides a command-line interface to interact with various LLM providers. It supports several modes of operation:

    1. Command Mode: Execute specific commands via args.command_kind().
    2. Provider/Model Discovery: List available providers using --list-providers or available models using --list-models.
    3. Non-Interactive Mode: If the input is not a terminal (e.g., piped input) or if specific non-interactive flags are set, the CLI runs in a non-interactive mode suitable for scripting.
    4. TUI Mode: By default, in an interactive terminal, the CLI launches a Terminal User Interface (TUI) for a rich user experience.

    Configuration is loaded from a config file (specified via --config) and initializes logging and provider registries before execution.

  8. Available UI themes in the LLM CLI

    main

    The LLM CLI supports three distinct visual themes via the Theme struct:

    1. Warm Theme (warm_with_palette): Inspired by Anthropic/Claude. Uses orange, terracotta, and coral accents with dark warm brown backgrounds.
    2. Cool Theme (cool_with_palette): Uses blue, cyan, and ice blue tones with dark blue backgrounds.
    3. Mono Theme (mono_with_palette): A high-contrast monochrome theme designed for accessibility, using various shades of gray and white.

  9. Event types in the LLM CLI runtime

    main

    The LLM CLI runtime uses a hierarchical event system to manage user input, application lifecycle, and streaming LLM responses.

    • InputEvent: Captures low-level user interactions like Key presses, Mouse movements, Paste actions, or terminal Resize events.
    • AppEvent: The top-level event wrapper that includes Input events, Tick events (for periodic updates), Stream events (for LLM output), and Tool events (for tool-calling workflows).
    • StreamEvent: Represents the lifecycle of an LLM response stream, including Started, TextDelta (incremental text), ToolCallStart/Delta/Complete (for tool usage), Usage (token counts), Done, and Error states.
    • ToolEvent: Handles the tool-calling loop, specifically ApprovalRequested (when the LLM needs permission to run a tool) and Result (when a tool execution finishes).

  10. Define a UserTool for YAML persistence

    main

    A UserTool is a user-defined tool that can be serialized and persisted to YAML. It consists of a name, a description, a list of parameters, and the actual shell command to execute. This structure allows the LLM to understand what the tool does and how to invoke it via the command field.

    name: "get_weather"
description: "Fetches the current weather for a location"
command: "curl -s https://api.weather.com/v1/{{location}}"
params:
  - name: "location"
    param_type: "string"
    description: "The city name"
    required: true

  11. Use multi-LLM dialogues in a Conversation

    main

    A Conversation can transition from a standard single-LLM interaction into a multi-LLM dialogue mode. This is useful when multiple LLM participants are involved in a single session.

    • Starting a dialogue: Call start_dialogue(mode, config, participants) to set the dialogue_mode, dialogue_config, and the list of participants.
    • Checking dialogue status: Use is_dialogue() to determine if the conversation is currently in a multi-LLM mode (returns true if dialogue_mode is set and participants is not empty).
    • Ending a dialogue: Call end_dialogue() to clear participants and configuration, returning the conversation to standard single-LLM mode.
    // Start a multi-LLM dialogue
conversation.start_dialogue(mode, config, participants);

// Check if we are in dialogue mode
if conversation.is_dialogue() {
    // ... handle multi-LLM logic
}

// Return to single-LLM mode
conversation.end_dialogue();

  12. Specify LLM providers and models via provider strings

    main

    The llm-cli uses a specific string format to identify both the provider and the specific model in a single argument. The format is provider:model.

    • Provider: The name of the LLM backend (e.g., openai).
    • Model (Optional): The specific model identifier (e.g., gpt-4o).

    If the model part is omitted (e.g., just openai), the CLI will attempt to use the default model specified in your configuration or via the --model flag.

    openai:gpt-4o