simple-one-api

The load_balancing configuration key controls how the backend selects between multiple available options. Setting load_balancing to random allows the system to automatically choose between multiple credential sets for the same model or between different services/models that are marked as enabled: true.

Load balancing multiple credentials

To rotate through different API keys for the same model, define multiple service entries with the same model name but different credentials:

{
  "server_port": ":9099",
  "load_balancing": "random",
  "services": {
    "openai": [
      {
        "models": ["@cf/meta/llama-2-7b-chat-int8"],
        "enabled": true,
        "credentials": { "api_key": "xxx" },
        "server_url": "https://api.cloudflare.com/..."
      },
      {
        "models": ["@cf/meta/llama-2-7b-chat-int8"],
        "enabled": true,
        "credentials": { "api_key": "yyy" },
        "server_url": "https://api.cloudflare.com/..."
      }
    ]
  }
}

Load balancing between different models/services

To let the backend randomly select a model from all enabled services, set load_balancing to random and ensure the desired models have enabled: true:

{
  "server_port": ":9099",
  "load_balancing": "random",
  "services": {
    "openai": [
      {
        "models": ["@cf/meta/llama-2-7b-chat-int8"],
        "enabled": true,
        "credentials": { "api_key": "xxx" },
        "server_url": "https://api.cloudflare.com/..."
      }
    ],
    "ollama": [
      {
        "models": ["llama2"],
        "enabled": true,
        "server_url": "http://127.0.0.1:11434/api/chat"
      }
    ]
  }
}

The service provides several advanced management features for LLM orchestration:

• Load Balancing: Supports configuring multiple models with random load balancing, and multiple api_keys per model with random load balancing.
• Unified API: Provides a single external api_key and supports both /v1 and /v1/chat/completions paths.
• Model Aliasing: Supports model name redirection (aliasing) and global redirection (using all mode to redirect all requests).
• Automatic Model Selection: Supports a random model type that automatically finds an available configured model.
• Prompt Compatibility: For models that do not support a system role, simple-one-api automatically prepends the system prompt to the first user prompt to ensure compatibility with clients like Immersive Translate.
• Rate Limiting: Supports setting QPS (Queries Per Second), QPM (Queries Per Minute), or concurrency limits per service.
• Endpoint Management: Allows setting specific service addresses for each model type.
• Standard Endpoints: Supports /v1/models and /v1/models/:model for model discovery.

You can control how the API selects between multiple enabled services or multiple API keys for the same model using the load_balancing field. Supported values are:

• first: Always picks the first enabled configuration.
• random: Randomly selects an enabled configuration.

To provide multiple API keys for a single model, add multiple objects to the service's array in services.

{
    "api_key":"123456",
    "load_balancing": "random",
    "services": {
       "xinghuo": [
         {
           "models": ["spark-lite"],
           "enabled": true,
           "credentials": {
             "appid": "xxx",
             "api_key": "xxx",
             "api_secret": "xxx"
           }
         },
         {
           "models": ["spark-lite"],
           "enabled": true,
           "credentials": {
             "appid": "yyy",
             "api_key": "yyy",
             "api_secret": "yyy"
           }
         }
       ]
   }
}

To avoid concurrency limits and automatically distribute requests across multiple models or multiple accounts for the same model, use the load_balancing configuration key.

Using random Load Balancing

Setting load_balancing to random allows the server to automatically select a model from the pool of available services where enabled is set to true.

Scenario 1: Mixing multiple different models

If you want to rotate between different providers (e.g., Spark, Qianfan, and Hunyuan), include them in your services configuration and set load_balancing: "random".

Scenario 2: Load balancing multiple accounts for a single model

If you have multiple sets of credentials (e.g., multiple appids for spark-lite), you can define multiple entries for the same model within a service array. When load_balancing is set to random, simple-one-api will randomly select one of the enabled credential sets for each request.

{
  "api_key":"sk-123456",
  "server_port":":9090",
  "load_balancing": "random",
  "services": {
    "qianfan": [
      {
        "models": ["yi_34b_chat", "ERNIE-Speed-8K"],
        "enabled": true,
        "credentials": {
          "api_key": "xxx",
          "secret_key": "xxx"
        }
      }
    ],
    "xinghuo": [
      {
        "models": ["spark-lite"],
        "enabled": true,
        "credentials": {
          "appid": "xxx",
          "api_key": "xxx",
          "api_secret": "xxx"
        }
      }
    ]
  }
}

simple-one-api acts as a unified proxy for various providers, normalizing them to the OpenAI API format.

Supported Providers

• OpenAI ChatGPT series (OpenAI, Azure OpenAI, Cloudflare Workers AI, Groq)
• ByteDance Coze (Coze.com)
• Ollama (Local LLM runner)

Key Capabilities

• Load Balancing: Supports random load balancing across multiple models or multiple api_keys for a single model.
• Global Settings: Ability to set a global api_key or a global proxy mode.
• Model Aliasing: Support for setting aliases for models.
• Automatic Model Selection: Using the random model name automatically finds an enabled model.
• Protocol Normalization: Supports both /v1 and /v1/chat/completions paths. For models that do not natively support a system role, simple-one-api automatically injects the system prompt into the first user prompt to ensure compatibility with clients like immersive translation.