DOCSEARCHbyAlgolia
Submit Docs

RelayFreeLLM

repository·main·Indexed 41 days ago

https://github.com/msmarkgu/relayfreellm

An open-source, OpenAI-compatible gateway that aggregates free-tier AI providers such as Gemini, Groq, and Mistral into a single endpoint. It features automatic failover, intent-based routing, image-aware routing, and multi-turn context management modes (Static, Dynamic, Reservoir, and Adaptive). Includes an admin dashboard for real-time limit management and a built-in chat interface.

Tokens
9.7K
Snippets
31
Records
46
Agent score
50%

What's inside RelayFreeLLM

  1. Use Session Affinity to pin users to a provider

    main

    To ensure a specific user is consistently routed to the same provider (which helps maintain state or consistency), pass the X-Session-ID header with a unique identifier. If the pinned provider fails, the gateway will automatically migrate the session to a different provider.

    X-Session-ID: user-123

  2. How Intent-Based Routing works

    main

    RelayFreeLLM supports intent-based routing, allowing you to request capabilities rather than specific APIs. When using meta-model, you can pass additional parameters to filter the available pool of models:

    • model_type: e.g., coding (filters for coding models).
    • model_scale: e.g., large, medium, small (filters by model size).
    • model_name: e.g., deepseek (prefers specific model families).

    Routing Logic Examples

    • {"model": "meta-model"}: Any available provider.
    • {"model": "meta-model", "model_type": "coding"}: Any coding model.
    • {"model": "meta-model", "model_scale": "large"}: Only large models.
    • {"model": "meta-model", "model_name": "deepseek"}: Prefers DeepSeek models.
    • {"model": "Gemini/gemini-2.5-flash"}: Direct routing to a specific provider/model.
    {
  "model": "meta-model",
  "model_type": "coding"
}

  3. How Image-Aware Routing works

    main

    RelayFreeLLM automatically detects if a user message contains image_url content parts (per OpenAI spec) and adjusts routing accordingly:

    1. Meta-model routing: If images are detected, the dispatcher only considers models with modality: "vision" in the registry.
    2. Specific routing: If a user explicitly requests a text-only model (e.g., nvidia/minimaxai/minimax-m2.7) but provides an image, the dispatcher will:
      • Search for a vision model within the same provider (e.g., nvidia/moonshotai/kimi-k2.6).
      • Fall back to a vision model from any other provider.
      • Return an error if no vision-capable models are available.

    This behavior is driven by the modality field in the provider_model_limits.json configuration.

  4. Understand Multi-Turn Context Management modes

    main

    RelayFreeLLM manages conversation history using different strategies to fit within token budgets:

    ModeBehavior
    StaticKeeps the last N messages verbatim.
    DynamicAdjusts context window based on real-time token usage.
    ReservoirRecent messages verbatim + extractive summary of older history. Uses a TF-scoring algorithm for selection (no LLM calls needed).
    AdaptiveDetects coding vs chat conversations and switches strategy automatically.

  5. Configure Agent Framework Support

    main

    Agent frameworks like LangChain, AutoGen, or CrewAI manage their own system prompts and history. To prevent RelayFreeLLM from injecting its own STANDARD_SYSTEM_PROMPT or reordering messages, you must disable server-side prompt injection.

    Per-request: Send the header X-Use-ServerSide-System-Prompt: false.

    Global configuration: Set the default in settings.json:

    {
  "routing": {
    "use_server_side_system_prompt": false
  }
}

    Note: For maximum fidelity in message order, pair this with CONTEXT_MANAGEMENT_MODE=static in settings.json. Modes like reservoir or adaptive may still summarize or reorder context to fit token budgets.

  6. Manage RelayFreeLLM via the Admin Dashboard

    main

    The Admin Dashboard at http://localhost:8000/admin allows for real-time management of the gateway without restarting the server or manually editing JSON files.

    Limits Tab

    • Provider Management: View providers as collapsible cards. You can add or remove entire providers or individual models.
    • Inline Editing: Edit model names, model_type (text/coding/image/etc.), model_scale (large/medium/small), max context length, and rate-limit values (requests/tokens per second/minute/hour/day).
    • Persistence: Changes are saved to provider_model_limits.json and hot-reloaded immediately.

    Usage Tab

    • Real-time Monitoring: View summary cards for total requests, prompt tokens, completion tokens, and total tokens.
    • Breakdown: View per-provider and per-model usage tables.
    • Reset: Use the Reset Stats button to zero out counters in usage_stats.json.

  7. Install and Setup RelayFreeLLM

    main

    RelayFreeLLM is an OpenAI-compatible gateway that aggregates multiple free-tier AI providers into a single endpoint with automatic failover.

    Local Installation

    1. Clone the repository and install dependencies:
    git clone https://github.com/msmarkgu/RelayFreeLLM.git && cd RelayFreeLLM
pip install -r requirements.txt
    1. Configure API keys by creating a .env file in the project root with the following keys:
    • GEMINI_APIKEY (ai.google.dev)
    • GROQ_APIKEY (console.groq.com)
    • MISTRAL_APIKEY (console.mistral.ai)
    • NVIDIA_APIKEY (build.nvidia.com)
    • SAMBANOVA_APIKEY (cloud.sambanova.ai)
    1. Verify connectivity:
    python -m tests.test_models_availability
    1. Start the server:
    python -m src.server

    Docker Installation

    1. Build the image:
    docker build -t relayfreellm .
    1. Run the container with your .env file:
    docker run --rm --env-file .env -p 8000:8000 relayfreellm
    git clone https://github.com/msmarkgu/RelayFreeLLM.git && cd RelayFreeLLM
pip install -r requirements.txt

  8. Use the built-in Chat Interface

    main

    A web-based chat UI is available at http://localhost:8000/chat.

    Key Features

    • Streaming: Real-time token-by-token output.
    • Provider Attribution: See which provider/model handled the response.
    • Image Support: Upload, paste, or drag-and-drop images for vision-capable models.
    • Intent-Based Switching: Use the dropdown to switch between meta-model or specific Provider/Model combinations.
    • Conversation History:
      • Browser Storage: Uses localStorage by default.
      • Server Storage: Switch to Server in the header dropdown to persist chats to conversations.json on the server, allowing history to survive browser resets.

  9. Use RelayFreeLLM with the OpenAI SDK

    main

    Since RelayFreeLLM is OpenAI-compatible, you can use existing OpenAI clients by simply changing the base_url and providing a dummy api_key.

    Basic Usage (Meta-model)

    Using meta-model allows the gateway to automatically pick the next available provider, providing automatic failover if one hits a rate limit.

    Specific Model Routing

    You can also route to a specific provider and model using the provider/model syntax.

    Note: To disable the server-side system prompt injection (which standardizes output style), include the header X-Use-ServerSide-System-Prompt: false in your request.

    from openai import OpenAI

# Use the meta-model for automatic failover
client = OpenAI(base_url="http://localhost:8000/v1", api_key="relay-free")

# Option 1: Meta-model routing
response = client.chat.completions.create(
    model="meta-model",
    messages=[{"role": "user", "content": "Hello!"}]
)

# Option 2: Specific provider/model routing
response = client.chat.completions.create(
    model="groq/llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "Hello!"}]
)

  10. How to add a new AI provider

    main

    To extend RelayFreeLLM with a new AI provider, you must implement a new client class that follows these requirements:

    1. Create a new file in the api_clients/ directory.
    2. The class must subclass ApiInterface.
    3. The class must define a PROVIDER_NAME attribute as a non-empty string.

    The ProviderRegistry will automatically discover and register your provider during its auto_discover() scan by inspecting the api_clients/ package for subclasses of ApiInterface that have a PROVIDER_NAME set.

  11. Understand the RelayFreeLLM startup lifecycle

    main

    When the server starts, it performs a multi-step initialization process via a lifespan manager to ensure the routing engine is valid and functional:

    1. Provider Auto-discovery: The ProviderRegistry scans for available provider clients and validates that API keys/credentials are present.
    2. Model Selector Initialization: The ModelSelector loads model limits from JSON configuration.
    3. Synchronization & Validation: The server synchronizes the ProviderRegistry (code/credentials) with the ModelSelector (JSON limits).
      • Providers with credentials but no JSON limits are pruned.
      • Providers with JSON limits but no credentials/code are pruned.
      • Critical: If no valid providers remain after synchronization, the server aborts startup.
    4. State Initialization: The ConversationStore (for persistence) and UsageTracker (for statistics) are initialized.
    5. Dispatcher Creation: The ModelDispatcher is created, composing the registry, selector, and usage_tracker to form the core routing engine.

    All these components are injected into app.state for access by the API and Admin routers.

  12. How context management modes work

    main

    RelayFreeLLM provides different strategies for managing conversation history to optimize token usage and context window utilization:

    • static: Keeps a fixed number of the most recent messages (CONTEXT_STATIC_RECENT_KEEP) verbatim. This is the simplest and safest mode.
    • dynamic: Adjusts the amount of context sent based on recent token usage patterns for a session. If usage is low, it boosts the context; if usage is high, it reduces it to stay within targets.
    • reservoir: Keeps a set number of recent messages verbatim (CONTEXT_RESERVOIR_RECENT_KEEP) and uses an extractive summarization algorithm to condense older messages into a single system message prepended to the history.
    • adaptive: Automatically switches between reservoir and static modes based on the content. If the conversation is detected to be "code-heavy" (containing keywords like def, class, import, etc.), it uses reservoir mode to preserve context via summarization. Otherwise, it uses static mode.
    • disabled: Returns an empty list, sending no history.