modelrelay

Routing via auto-fastest, grouped-IDs, or tag:<name> uses a QoS score that balances model quality (intell percentile) and observed latency.

Latency Scoring: Latency is scored using the formula: target / (target + avg).

• An instant response scores 1.0.
• A model matching the qosLatencyTargetMs scores 0.5.
• The score decays continuously but never reaches zero, ensuring models remain as last-resort candidates.

Configuration: The qosLatencyTargetMs (default: 3000ms) can be adjusted in the Web UI under Settings → QoS Latency Target (ms). Lowering this value weights speed more heavily; raising it allows quality to dominate.

Model tags allow you to route by capability rather than specific model names. Tags come from two sources:

1. Curated tags: Fixed vocabulary including coding, reasoning, general, fast, and agentic.
2. Custom tags: Freeform labels assigned via the Web UI under Custom Routing Tags.

Filtering by Context Window

To ensure a model can handle your prompt size, use the +min_ctx:<size> modifier.

• tag:general+min_ctx:32000 (raw tokens)
• tag:coding+min_ctx:128k (k suffix)
• auto-fastest+min_ctx:1m (m suffix)

Modelrelay uses provider-reported context or curated values from sources.js. If a provider's reported context is higher than its observed live rate-limit quota, modelrelay uses the smaller of the two for filtering.

You can add custom OpenAI-compatible endpoints (like vLLM or llama.cpp) to modelrelay.

Via Web UI:

1. Click + Add Endpoint under the OpenAI-Compatible endpoints group.
2. Provide a name, base URL, model ID, and optional API key.
3. Discovery is enabled by default; modelrelay will probe /v1/models on the endpoint to expose its models.

Via API:

• POST /api/openai-compatible/endpoints (Body: {name, baseUrl, modelId, apiKey?})
• DELETE /api/openai-compatible/endpoints/<id>

Configuration Storage: Endpoints are stored in ~/.modelrelay.json under composite keys like openai-compatible:my-vllm.

{
  "apiKeys": {
    "openai-compatible:my-vllm": "sk-…"
  },
  "providers": {
    "openai-compatible:my-vllm": {
      "enabled": true,
      "name": "Local vLLM",
      "baseUrl": "http://localhost:8000/v1",
      "modelId": "qwen-coder"
    }
  }
}

To run modelrelay using Docker Compose, create a directory, download the necessary files, and start the container:

mkdir modelrelay

cd modelrelay

curl -fsSL -o Dockerfile https://raw.githubusercontent.com/ellipticmarketing/modelrelay/master/Dockerfile
curl -fsSL -o docker-compose.yml https://raw.githubusercontent.com/ellipticmarketing/modelrelay/master/docker-compose.yml

docker compose up -d --build

Once running, the router is accessible at http://localhost:7352/.

To move your configuration (including API keys, provider toggles, and bans) to another machine:

1. Using Web UI Tokens: In the Web UI, go to Settings -> Configuration Transfer to export/copy/import a token. Warning: Treat these tokens as secrets.
2. Manual File Copy: Copy the ~/.modelrelay.json file directly to the target machine at the same path.

To install modelrelay globally and start the router, use the following commands:

npm install -g modelrelay

# Start the router
modelrelay

Once running, the router is accessible at http://localhost:7352/.

npm install -g modelrelay

# Start it
modelrelay

You can automatically configure OpenClaw by running modelrelay onboard.

For manual configuration, merge the following into ~/.openclaw/openclaw.json:

{
  "models": {
    "providers": {
      "modelrelay": {
        "baseUrl": "http://127.0.0.1:7352/v1",
        "api": "openai-completions",
        "apiKey": "no-key",
        "models": [
          { "id": "auto-fastest", "name": "Auto Fastest" }
        ]
      }
    }
  },
  "agents": {
    "defaults": {
      "model": {
        "primary": "modelrelay/auto-fastest"
      },
      "models": {
        "modelrelay/auto-fastest": {}
      }
    }
  }
}

Model quality in modelrelay is determined through a tiered scoring system that prioritizes empirical benchmarks over heuristics:

1. Artificial Analysis Coding Index: If a model has a valid artificial_analysis.coding_index, it is used as the primary score (normalized to a 0-1 scale).
2. Design Arena Elo (Estimated): If no coding index is available, the system uses a linear regression model to estimate a score based on the model's design_arena Elo rating.
3. Metadata Heuristic: If neither benchmark is available, a score is calculated using metadataQuality, which considers:
   • Popularity: Rank within the catalog.
   • Recency: How recently the model was created.
   • Features: Support for reasoning, tools, and structured_outputs.
   • Context: The model's context_length (logarithmic scaling).

You can automatically configure OpenCode by running modelrelay onboard.

For manual configuration, add the following to ~/.config/opencode/opencode.json:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "router": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "modelrelay",
      "options": {
        "baseURL": "http://127.0.0.1:7352/v1",
        "apiKey": "dummy-key"
      },
      "models": {
        "auto-fastest": {
          "name": "Auto Fastest"
        }
      }
    }
  },
  "model": "router/auto-fastest"
}

You can move configurations between machines using encoded tokens. This is useful for syncing settings without manually copying JSON files.

Exporting

Use exportConfigToken(config) to generate a base64url encoded string prefixed with mrconf:v1:.

Importing

Use importConfigToken(token) to parse a token back into a normalized configuration object. The function supports:

1. Tokens prefixed with mrconf:v1: (preferred).
2. Raw JSON strings.
3. Legacy base64 encoded payloads.

You can deploy modelrelay using Docker Compose. The service exposes port 7352 and uses a named volume modelrelay_config to persist configuration files at /app/config inside the container.

Key environment variables:

• NODE_ENV: Set to production by default.
• HOME: Set to /app/config to ensure the application uses the persistent volume for its home directory/configuration.

services:
  modelrelay:
    build: .
    container_name: modelrelay
    restart: unless-stopped
    ports:
      - "7352:7352"
    volumes:
      - modelrelay_config:/app/config
    environment:
      - NODE_ENV=production
      - HOME=/app/config

volumes:
  modelrelay_config: