LiteLLM

LiteLLM is an open source AI Gateway that provides a single, unified interface to call over 100 LLM providers (including OpenAI, Anthropic, Gemini, Bedrock, and Azure) using the standard OpenAI format.

It can be used in two primary ways:

1. Python SDK: For direct library integration within your Python applications.
2. AI Gateway (Proxy Server): As a centralized, self-hosted service for teams or organizations to manage LLM access.

The LiteLLM Rust AI Gateway is a minimal Axum-based service designed to front OpenAI's realtime API. It acts as a WebSocket proxy: clients connect via wss://<host>/v1/realtime?model=<model>, and the gateway authenticates the request, selects the appropriate deployment from a configuration, and splices the client and OpenAI sockets frame-by-frame.

Key Characteristics:

• High Performance: The realtime hot path is implemented in pure Rust. Python is only used at load time to read the configuration.
• Authentication: Requires a Bearer token via the Authorization header using the LITELLM_MASTER_KEY.
• Observability: Provides health endpoints (/health/readiness, /health/liveness, /health/gil) and sends session logs to a LiteLLM proxy via a non-blocking background worker.

LiteLLM provides an abstraction and routing layer for OpenAI's /v1/realtime endpoints. This allows you to interact with real-time audio/text capabilities across multiple providers using a unified interface.

Supported Endpoints

• WebSocket: /v1/realtime (Note: Use intent=transcription for sessions requiring transcription only).
• HTTP: /v1/realtime/client_secrets and /v1/realtime/transcription_sessions.

Supported Providers

You can route real-time requests to the following providers:

• OpenAI
• Azure OpenAI
• Bedrock
• Vertex AI
• xAI

The liteLLM Proxy Server provides a unified interface to interact with over 50+ LLM models from providers like Azure, OpenAI, Replicate, Anthropic, and Hugging Face.

Key capabilities include:

• Consistent Input/Output: All models are called using the OpenAI format (completion(model, messages)). Text responses are always accessible via ['choices'][0]['message']['content'].
• Error Handling: Supports model fallbacks (e.g., if GPT-4 fails, automatically try llama2).
• Logging: Integrates with providers like Supabase, Posthog, Mixpanel, Sentry, Lunary, Athina, and Helicone to log requests, responses, and errors.
• Observability: Tracks token usage, spend per model, and implements semantic caching.
• Streaming & Async: Supports streaming text responses via generators.

The Dotprompt manager automatically converts rendered text into structured chat messages based on the template content:

1. Simple Text: A template containing only text (e.g., Tell me about {{topic}}.) is converted into a single user role message.
2. Role-Based Format: If you use explicit role prefixes like System: and User:, the manager parses them into multiple messages.

Example Conversion: Template:

System: You are a {{role}}.
User: {{question}}

Becomes:

[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is AI?"}
]

The PassThroughEndpointHandler enables guardrails to run on requests sent through passthrough endpoints. It operates using three main mechanisms:

1. Field Targeting: It uses JSONPath expressions (configured via request_fields and response_fields) to extract specific data from the request or response for guardrail evaluation.
2. Full Payload Fallback: If no specific JSONPath fields are provided in the configuration, the handler processes the entire request/response payload.
3. Config Access: The handler retrieves guardrail configurations from request metadata using get_passthrough_guardrails_config() and set_passthrough_guardrails_config() helpers.

This module is designed for auto-discovery by the load_guardrail_translation_mappings() function in litellm/llms/__init__.py.

The LiteLLM UI uses a strict NextJS file-based routing system. Pages are defined by folders containing a page.tsx file. Routing is automatically determined by the directory path.

To create routes that do not appear in the URL path but still allow for shared layouts and organizational structure, use parentheses around the directory name (e.g., (dashboard)).

// Example: Visiting /settings/admin-settings renders this file
. 
├── settings
│   ├── admin-settings
│   │   └── page.tsx