DOCSEARCHbyAlgolia
Submit Docs

GoModel Documentation

repository·main·Indexed 21 days ago

https://github.com/enterpilot/gomodel

GoModel is a high-performance, resource-efficient AI Gateway that sits between applications and LLM providers. It provides features including caching, cost tracking, rate limiting, and intelligent routing. The gateway is OpenAI-compatible and supports multiple providers such as OpenAI, Anthropic, Oracle, and vLLM. It can be deployed via Docker or Helm charts on Kubernetes 1.29+.

Tokens
115.1K
Snippets
295
Records
546
Agent score
77%

What's inside GoModel

  1. Overview of GoModel response caching

    main

    GoModel provides two layers of response caching for non-streaming requests on the following endpoints:

    • /v1/chat/completions
    • /v1/responses
    • /v1/messages
    • /v1/embeddings

    Cache Types

    1. Exact-match cache: Returns byte-identical responses. When a hit occurs, the response includes the header X-Cache: HIT (exact).
    2. Semantic cache: Uses embeddings and vector search to allow meaning-equivalent prompts to reuse stored responses. When a hit occurs, the response includes the header X-Cache: HIT (semantic).

    Both cache layers operate on the final request sent upstream, meaning they run after workflow and guardrail patching. To bypass caching for a specific request, use the Cache-Control: no-cache or Cache-Control: no-store headers.

  2. Overview of Budgets in GoModel

    main

    Budgets allow you to set spend limits for specific user_path subtrees or request labels. GoModel evaluates these limits against tracked usage cost records and blocks requests if a limit has been reached.

    Budgets use two types of scopes:

    • user_path: Limits a specific path (e.g., /team/alpha) and all its descendants.
    • label: Limits every request that carries a specific label (e.g., Mobile-App-iOS).

    Note: Budget enforcement only runs if budgets are globally enabled AND the active workflow has the budget feature enabled.

  3. Overview of the MCP Gateway

    main

    The GoModel MCP Gateway acts as an aggregator for Model Context Protocol (MCP) servers. Instead of configuring multiple MCP servers directly in every agent client (like Claude Desktop, Cursor, or VS Code), you connect your clients to a single authenticated GoModel endpoint.

    Key capabilities:

    • Aggregation: Exposes multiple upstream MCP servers through one endpoint. Tools are namespaced as {server}_{tool} to prevent collisions.
    • Credential Management: The gateway holds the upstream credentials (e.g., API keys); clients only need a GoModel bearer token.
    • Access Control: Supports per-server visibility via user_paths, tool allow/deny lists, and per-request scoping using the X-MCP-Servers header.
    • Observability: Provides audit logs, usage attribution, and rate limiting for every tool call.

  4. What is GoModel Pro and how does it differ from open-source?

    main

    GoModel Pro is a commercial distribution of GoModel that includes the same core gateway features (routing, failover, budgets, rate limits, virtual models, caching, audit, and the dashboard) but adds licensed features like Token Compression.

    Without a valid license, the binary operates as the plain open-source core. Importantly, GoModel Pro never blocks traffic; it only enables additional features if licensed.

  5. Overview of the Translated Inference Service

    main
    The translatedInferenceService is a core component responsible for the orchestration of inference requests. It manages the lifecycle of a request by evaluating fields, generating bounds, applying scopes, and enforcing processing limits. It is responsible for schema verification, identifying output paths, and mapping components to their respective providers and models.

  6. Understand the Rate Limiting Client Contract

    main

    When a client exceeds a rate limit, GoModel returns a 429 Too Many Requests status code.

    Error Response

    The error body follows the OpenAI error format to ensure compatibility with existing SDKs:

    {
  "error": {
    "type": "rate_limit_error",
    "message": "rate limit exceeded for /team/alpha: minute limit 100 requests used; retry after 12s",
    "code": "rate_limit_exceeded",
    "param": null
  }
}
    • code: Always returns rate_limit_exceeded regardless of whether requests, tokens, or concurrency was the cause.
    • Retry-After Header: An integer representing the seconds to wait until the sliding window estimate allows a retry. For concurrency breaches, this is a constant 1.

    Success Headers

    On successful requests, GoModel provides x-ratelimit-* headers (using OpenAI spelling) to help clients manage their own pacing:

    • x-ratelimit-limit-requests
    • x-ratelimit-remaining-requests
    • x-ratelimit-reset-requests (seconds until reset)
    • x-ratelimit-limit-tokens
    • x-ratelimit-remaining-tokens
    • x-ratelimit-reset-tokens

  7. Use PassthroughService for Proxying Provider Responses

    main

    The passthroughService and its associated components allow for proxying requests and responses to external providers. This is useful for creating a unified interface that can route queries to different LLM providers while maintaining semantic enrichment and provider resolution.

    Core components:

    • ProviderPassthrough(): Proxies responses and manages the lifecycle of a passthrough request.
    • resolvePassthroughProvider(): Resolves which provider should handle a specific passthrough request.
    • PassthroughSemanticEnrichment(): Enhances data streams and queries during the passthrough process.
    • passthroughAccessSelector(): Determines access permissions for passthrough operations.

  8. How GoModel estimates request costs

    main

    GoModel provides cost estimates for tracked requests by multiplying reported token counts by configured per-million-token (MTok) rates.

    It supports pricing for:

    • Input and output tokens.
    • Cached input, reasoning, and audio tokens (when reported by the provider).

    Cost Sources: GoModel uses different methods to determine cost, which are recorded in the cost_source field of each row:

    • model_pricing: Calculated using catalog or override rates.
    • openrouter_credits: Uses exact cost reported by OpenRouter.
    • xai_cost_in_usd_ticks: Uses exact cost reported by xAI.

    Important: These are estimates, not billing records. Discounts, promotions, or provider-side rounding may not be reflected. Always verify GoModel figures against your provider's official dashboard for invoicing.

  9. How GoModel's resilience layers work

    main

    GoModel provides two layers of fault tolerance for every upstream provider call. These layers are applied per provider and do not automatically switch to a different model or provider (for that, use Failover).

    1. Retry with exponential backoff: Repeats a failed request against the same provider with increasing delays between attempts.
    2. Circuit breaker: Short-circuits calls to a provider that has been failing repeatedly. Once the timeout elapses, it enters a 'probing' state to see if the provider has recovered.

    Important Note: These layers are in-memory and per gateway process. State is not shared between replicas and resets on restart.

    # No runnable code example provided for this concept

  10. Use SystemPromptGuardrail to enforce system instructions

    main

    The SystemPromptGuardrail allows you to manage the system prompt of a conversation. It supports different SystemPromptMode settings to:

    • Append: Add instructions to the existing system prompt.
    • Replace: Swap the existing system prompt with a new one.
    • Wrap/Decorate: Enclose the existing prompt within new instructions.

    This ensures that specific rules or personas are strictly enforced in the model's context.

  11. Understand potential conflicts in model selector resolution

    main

    Because provider-qualified selectors take precedence over raw model IDs containing slashes, you may encounter shadowing if a provider name matches the prefix of a raw model ID.

    Example of Shadowing: If you have a configured provider named gemini and a raw model ID named gemini/gemini-xyz belonging to a different provider, the selector gemini/gemini-xyz will always resolve to provider gemini with model gemini-xyz. The raw model ID gemini/gemini-xyz becomes unreachable via that selector.

    Ambiguity Note: Bare model names (those without slashes) remain ambiguous if multiple providers expose the same model name. In these cases, GoModel defaults to a 'first registered provider wins' strategy.

  12. Understand Guardrail persistence and configuration

    main

    Contrary to some older documentation, Guardrails in GoModel are not configured via config.yaml only.

    Guardrail definitions are persisted in a guardrail_definitions store across all supported backends (SQLite, Postgres, MongoDB). While YAML can be used to provide seed data at boot time, administrative CRUD operations can modify these definitions via the Admin API.