liteLLM Proxy

The liteLLM Proxy acts as a middleware layer between your application and various LLM providers. Key features include:

• Consistent Interface: Call any model (Azure, OpenAI, Anthropic, etc.) using the standard OpenAI completion(model, messages) format.
• Error Handling & Fallbacks: Automatically handle failures by falling back to alternative models (e.g., if GPT-4 fails, try llama2).
• Observability: Log requests, responses, and errors to providers like Supabase, Posthog, Mixpanel, Sentry, Helicone, and more.
• Token Tracking: Monitor input/completion tokens and total spend per model.
• Semantic Caching: Implement caching to reduce latency and costs.
• Streaming & Async: Supports streaming text responses via generators and asynchronous calls.

To run the liteLLM Proxy server on your local machine, follow these steps:

1. Clone the repository:

   git clone https://github.com/BerriAI/liteLLM-proxy

2. Install dependencies:

   pip install -r requirements.txt

3. Configure environment variables:
   • Set your LLM provider keys (e.g., OPENAI_API_KEY) in a .env file.
   • (Optional) Set your proxy master key using LITELLM_PROXY_MASTER_KEY in your .env file.
4. Start the server:

   python main.py

Since the proxy provides a consistent OpenAI-compatible interface, you can use the standard openai Python library by pointing the api_base to your proxy URL and using your proxy's master key as the api_key.

import openai 

openai.api_key = "sk-litellm-master-key"
openai.api_base = "http://0.0.0.0:8080"

response = openai.ChatCompletion.create(
    model="gpt-3.5-turbo", 
    messages=[{"role": "user", "content": "Hey"}]
)

print(response)

The /chat/completions (POST) endpoint allows you to interact with 50+ LLM models (Azure, OpenAI, Anthropic, etc.) using a unified OpenAI-style format. You can use standard HTTP clients like requests to send JSON payloads to the proxy.

import requests
import json

url = "http://localhost:5000/chat/completions"

payload = json.dumps({
  "model": "gpt-3.5-turbo",
  "messages": [
    {
      "content": "Hello, whats the weather in San Francisco??",
      "role": "user"
    }
  ]
})
headers = {
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)
print(response.text)

The proxy server relies on several environment variables for configuration:

• LITELLM_PROXY_MASTER_KEY: The administrative key used to manage other keys and bypass certain restrictions. Defaults to sk-litellm-master-key if not set.
• PROJECT_NAME: Used by the BudgetManager to track costs and users.
• PORT: The port on which the server will run. Defaults to 8080.

The /chat/completions endpoint allows you to interact with various LLM models through the proxy. It supports both standard JSON responses and streaming responses.

Authentication: Requires a valid user API key or the master key passed via the Authorization: Bearer <key> header.

Streaming: To enable streaming, include `

The /chat/completions endpoint accepts the following JSON input parameters:

• model (string, required): The ID of the model to use (e.g., gpt-3.5-turbo, gpt-4, claude-2, command-nightly).
• messages (array, required): A list of message objects. Each object must contain:
  • role: The role of the message (system, user, assistant, or function).
  • content: The text content of the message.
  • name: (Optional) The name for the function role.
• temperature (optional): Sampling temperature.
• functions (optional): List of functions for tool calling.
• function_call (optional): Specific function to call.
• top_p (optional): Nucleus sampling parameter.
• n (optional): Number of chat completion choices to generate.
• stream (optional): Whether to stream the response.

The proxy returns responses in a consistent OpenAI-compatible format for all supported models. Text responses are always located at ['choices'][0]['message']['content'].

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "Response text here",
        "role": "assistant"
      }
    }
  ],
  "created": 1691790381,
  "id": "chatcmpl-7mUFZlOEgdohHRDx2UpYPRTejirzb",
  "model": "gpt-3.5-turbo-0613",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 41,
    "prompt_tokens": 16,
    "total_tokens": 57
  }
}

The proxy provides two endpoints to discover supported models:

1. /models/available: Returns a simple list of valid model names.
2. /models: Returns a structured list of models compatible with OpenAI-style client implementations, including metadata like id, object, created, and owned_by.

These endpoints do not require authentication.

GET /models/available
GET /models

The proxy includes built-in budget management and key generation capabilities.

Get cost for a specific key

Use GET /key/cost to retrieve the current model cost associated with a specific API key. This requires a valid user API key.

Generate a new API key

Administrators can create new user keys with specific budgets using POST /key/new. This endpoint requires the master_key (defined by LITELLM_PROXY_MASTER_KEY).

Request Body:

• total_budget: The amount of budget to assign to the new key.

Response: Returns the new api_key, the total_budget, and the duration (defaults to monthly).

POST /key/new
{
  "total_budget": 50
}

Response:
{
  "api_key": "sk-litellm-...",
  "total_budget": 50,
  "duration": "monthly"
}

The /health endpoint is a simple health check to verify that the proxy server is running and operational. It returns a 200 OK status with a JSON body.

GET /health

Response:
{
  "status": "ok"
}