liter-llm

The liter-llm Ruby package provides a native client for interacting with Large Language Models. Key capabilities include:

• Ruby-native client: Access chat, streaming, tools, structured output, and provider routing directly through Ruby objects.
• Provider/model routing: Use provider/model naming conventions to call models without needing provider-specific client logic.
• Production controls: Built-in support for retries, fallbacks, rate limits, cache layers, budgets, health checks, OpenTelemetry spans, and secret redaction.
• Cross-language consistency: Uses the same Rust core as all other language bindings (Python, Node.js, Go, Java, etc.), ensuring identical behavior across platforms.

Liter-llm is part of the Xberg family, a collection of open-source tools built on a fast Rust core. The ecosystem includes tools for document intelligence, web crawling, HTML conversion, and polyglot binding generation. Use these related projects to build complete data pipelines involving extraction, scraping, and LLM orchestration.

Related projects in the Xberg ecosystem:

- Xberg: Document intelligence (text, tables, metadata extraction from 101 formats).
- Xberg Enterprise: Managed extraction API with SDKs and observability.
- crawlberg: Web crawling and scraping (HTML→Markdown).
- html-to-markdown: Fast, lossless HTML to Markdown engine.
- liter-llm: Universal LLM API client (165 providers, 14 language bindings).
- tree-sitter-language-pack: Tree-sitter grammars and code-intelligence primitives.
- alef: Polyglot binding generator used to create language bindings.

The liter-llm Python package provides a high-level, async-first interface for interacting with LLMs. Key capabilities include:

• Async/Await & Streaming: Native support for asynchronous operations and streaming responses.
• Provider/Model Routing: Call models using a provider/model naming convention, abstracting away provider-specific client logic.
• Structured Output: Tools for obtaining typed request/response objects and structured data.
• Production Controls: Built-in mechanisms for retries, fallbacks, rate limiting, caching layers, usage budgets, health checks, and OpenTelemetry spans.
• Security: Automatic redaction of secrets.
• Cross-Language Consistency: Uses the same Rust core as all other language bindings (Node.js, Go, Java, .NET, etc.), ensuring identical behavior across platforms.

liter-llm supports multimodal capabilities, allowing you to:

• Send vision inputs (images and documents) alongside text prompts.
• Request structured JSON responses from models.
• Receive multimodal outputs, including images and audio.

This enables complex workflows where the model can 'see' visual context or provide machine-readable data structures.

Every language binding for liter-llm provides a single provider surface for the following capabilities across the provider registry:

• Chat & Streaming: Conversational interfaces with async streaming support.
• Embeddings: Vector generation.
• Images & Audio: Multimodal processing.
• Search & OCR: Information retrieval and optical character recognition.
• Tools & Structured Output: Function calling and schema-constrained responses.

All bindings share the same core Rust implementation, ensuring consistent production controls like retries, fallbacks, rate limits, cache layers, budgets, health checks, and OpenTelemetry spans.

The liter-llm-ffi package provides a stable C ABI for direct interaction with the Rust-powered liter-llm client. This is useful for C, C++, or any language that needs to link against a shared or static library when a first-party binding is not available.

Key Features

• Stable C ABI: Enables direct calls into the Rust client.
• Unified Routing: Uses the same provider and model registry as all other language packages (supporting chat, streaming, embeddings, tools, search, OCR, and structured output).
• Explicit Ownership: Memory management follows the contract defined in the generated liter_llm.h header, where C callers are responsible for handles and buffers.

#include "liter_llm.h"

int main(void) {
    // See https://github.com/xberg-io/liter-llm for usage examples.
    return 0;
}

liter-llm is part of the Xberg.io suite of tools designed for document intelligence and LLM orchestration:

• liter-llm: Universal LLM API client with native bindings for 14 languages and 165 providers.
• Xberg: Document intelligence for extracting text, tables, and metadata from 101 formats (with optional OCR).
• Xberg Enterprise: Managed extraction API with SDKs, dashboards, and observability.
• crawlberg: Web crawling and scraping with HTML→Markdown conversion.
• html-to-markdown: Fast, lossless HTML to Markdown engine.
• tree-sitter-language-pack: Tree-sitter grammars and code-intelligence primitives.
• alef: Polyglot binding generator used to produce per-language bindings.

In addition to tools, the liter-llm MCP server provides:

• Prompts: Reusable templates including summarize, translate, and extract.
• Resources: Read-only catalog endpoints:
  • liter-llm://models: Configured models.
  • liter-llm://providers: Built-in registry.
  • liter-llm://pricing/{model}: Pricing template.
  • liter-llm://provider/{name}: Provider template.
• Completion: Argument autocompletion for model (from configured models) and provider name (from the registry).

The CircuitState represents the observable state of a circuit breaker:

• Closed: Normal operation; requests flow through.
• Open: All requests are rejected while the circuit waits for the backoff period to elapse.
• HalfOpen: A single probe request is allowed through to test the health of the upstream service.

| Value | Description |
|-------|-------------|
| `Closed` | Requests flow through normally. |
| `Open` | All requests are rejected; the circuit is waiting for the backoff to elapse. |
| `HalfOpen` | One probe request is allowed through to test service health. |

The DefaultClient is a reqwest-backed implementation that supports automatic provider detection and routing for 165 LLM providers.

Key Behaviors:

• Construction-time Routing: The provider is resolved during construction based on a model_hint (defaulting to OpenAI).
• Per-request Overrides: You can override the provider for individual requests by using a model name prefix (e.g., "anthropic/claude-3-5-sonnet" will route to Anthropic even if the client was constructed for OpenAI).
• Fallback Mechanism: If a model prefix does not match a known provider, the client falls back to the provider specified at construction.
• Efficiency: The provider is stored in an Arc for cheap sharing in async tasks, and auth headers are pre-computed and cached.