DOCSEARCHbyAlgolia
Submit Docs

Bifrost AI Gateway Documentation

repository·dev·Indexed 27 days ago

https://github.com/maximhq/bifrost

Bifrost is a high-performance AI gateway that unifies access to over 23 AI providers, including OpenAI, Anthropic, and AWS Bedrock, via a single OpenAI-compatible API. It features automatic failover, load balancing, semantic caching, and governance. The documentation covers deployment via Helm charts on Kubernetes, configuration of MCP servers (HTTP, SSE, and STDIO), audit log archival to S3/GCS, SCIM Auth Proxy for Zero Trust deployments, and community catalog contributions.

Tokens
310.2K
Snippets
646
Records
1.3K
Agent score
89%

What's inside Bifrost

  1. Overview of Datadog Plugin for Bifrost

    dev

    The Datadog plugin provides native integration with the Datadog observability platform for LLM operations. It provides three main pillars of observability:

    • APM Traces: Distributed tracing via dd-trace-go v2 with W3C Trace Context support.
    • LLM Observability: Native Datadog LLM Observability integration for AI/ML-specific monitoring.
    • Metrics: Operational metrics via DogStatsD or the Datadog Metrics API.

    Unlike the generic OpenTelemetry plugin, this plugin uses Datadog's native SDKs to enable specific features like LLM Observability dashboards and ML App grouping.

  2. Overview of Bifrost Open Source Features

    dev

    Bifrost provides several core features for managing AI workloads:

    Governance & Routing

    • Virtual Keys: The primary governance entity used to control access permissions, budgets, rate limits, and routing per consumer.
    • Routing: Direct requests to specific models, providers, and keys using weighted strategies.
    • Load Balancing: Intelligent API key distribution with weighted load balancing and model-specific filtering.
    • Budget & Rate Limits: Hierarchical cost control at the virtual key, team, and customer levels.
    • MCP Tool Filtering: Use strict allow-lists to control which MCP tools are available per virtual key.

    Reliability & Performance

    • Automatic Fallbacks: Seamless failover between providers and models when a primary provider fails.
    • Semantic Caching: Reduces costs and latency by caching responses based on semantic similarity.
    • Mocker Plugin: Mock AI provider responses for testing and development.

    Observability

    • Built-in Observability: Real-time monitoring of every AI request.
    • Prometheus Metrics: Native metrics via scraping or Push Gateway.
    • OpenTelemetry (OTLP): Integration for distributed tracing with Grafana, New Relic, Honeycomb, etc.

  3. Overview of Bifrost Enterprise Gateway

    dev

    Bifrost Enterprise is a production-grade AI gateway designed for organizational scale. It is a strict superset of the open-source Bifrost AI Gateway, meaning all OSS providers, integrations, plugins, and SDKs work identically in Enterprise using the same config.json schema.

    Key enterprise-only capabilities include:

    • High-availability clustering and predictive load balancing.
    • Identity federation (OIDC) and Role-Based Access Control (RBAC).
    • Audit-grade compliance logging and automated log exports.
    • Advanced content guardrails (PII, secrets, and custom regex).
    • Hierarchical governance and data access control.

  4. Overview of Code Mode for MCP

    dev

    Code Mode is a feature designed to optimize Model Context Protocol (MCP) usage at scale. Instead of exposing all available tools (which can number in the hundreds) directly to the LLM, Code Mode exposes only four generic meta-tools. The LLM uses these tools to write Python (Starlark) code that orchestrates tool calls within a sandbox.

    This approach significantly reduces input token usage and costs by preventing the LLM from having to read massive tool catalogs in every turn. It is particularly effective for large deployments with many MCP servers.

  5. Overview of Bifrost Telemetry

    dev
    Bifrost provides built-in Prometheus-based monitoring via a /metrics endpoint. The telemetry system tracks HTTP performance, upstream provider interactions, token usage, costs, and cache performance. Metrics collection is performed asynchronously to ensure zero impact on request latency.

  6. Overview of Bifrost Edge features

    dev

    Bifrost Edge is a cross-platform device agent (macOS, Linux, Windows) introduced in the v2.0.0-prerelease1 release. Key capabilities include:

    • Bifrost Edge Device Agent: A daemon with a system tray UI, independent health probes, and environment-specific builds.
    • MCP Shim: A stdio interception shim that wraps discovered MCP servers on enrolled devices. It routes tool calls through the Bifrost agent to enable policy enforcement, per-tool blocking, and tool catalog reporting.
    • Edge Control Management: Management of devices, MCP, and edge configuration via the enterprise dashboard, including a kill switch and CA flow UI.
    • App Attribution Classifier: Detects which AI application generated traffic and stamps runtime-app attribution on request headers.
    • Virtual Key Mode: Allows the Edge agent to authenticate using a virtual key as an alternative to IdP-managed authentication.
    • Server Config Sync: Enables agents to automatically sync their configuration from the server.

  7. Overview of the BigQuery plugin

    dev
    The BigQuery plugin is a Bifrost Enterprise feature that streams every LLM request through Bifrost into a structured Google BigQuery table. Each completed trace is stored as a single, denormalized row containing provider, model, token usage, cost, latency, governance attribution, and optional conversation content. This is optimized for SQL-based analytics, cost reporting, and long-term data retention, allowing you to join trace data directly with your existing data warehouse.

  9. Supported Operations for Mistral Provider

    dev

    Mistral is an OpenAI-compatible provider supported by Bifrost. It supports Chat Completions, Responses API, Transcription (STT), OCR, Embeddings, and Model Listing.

    Note on Unsupported Operations: Text Completions, Speech (TTS), Files, and Batch operations are not supported by the upstream Mistral API and will return UnsupportedOperationError. Image Generation is also currently unsupported by the Bifrost integration.

  10. Understand Google Workspace provisioning mechanisms in Bifrost

    dev

    Bifrost uses three distinct mechanisms to keep user and group data in sync with Google Workspace:

    MechanismFrequencyPurpose
    OIDC provisioningOn first loginProvisions users in Bifrost based on JWT claims during their initial successful login.
    Background reconciliationEvery 24 hoursReconciles all users, roles, and group memberships against the current Google Workspace state. Catches deactivations and new users who haven't logged in yet.
    Session refreshEvery 15 minutesAttempts to refresh active OIDC sessions. If a Google account is suspended or deleted, the Bifrost session is terminated at the next refresh.

  11. Understand MCP Connection Types in Bifrost

    dev

    Bifrost supports four Model Context Protocol (MCP) connection types to integrate external tools. Choose a connection type based on your requirements for latency, security, and scalability:

    • InProcess Connections: High-performance, in-memory tools running in the same process. Lowest latency (~0.1ms) and highest security, but limited to Go packages and cannot be configured via JSON.
    • STDIO Connections: Local command-line tools and scripts. Low latency (~1-10ms) with high security via local control. Best for filesystem operations or CLI utilities.
    • HTTP Connections: Remote web APIs and microservices. Scalable and supports multi-server deployment, but has higher network-dependent latency (~10-500ms).
    • SSE Connections: Real-time streaming tools for live data feeds and event streams. Provides persistent connections and event-driven updates.

  12. Understand Semantic Caching modes

    dev

    Bifrost provides two complementary lookup paths for caching LLM responses to reduce costs and latency:

    • Direct (hash) matching: A deterministic, exact-match replay. The request is normalized and hashed. If an identical request is found, it is served instantly without requiring an embedding provider.
    • Semantic (similarity) matching: An embedding-based lookup. It serves a cached answer when a new request is semantically similar to a previous one, even if the wording differs. This requires an embedding provider.

    Key Behaviors:

    • Execution Order: Direct matching runs first. Semantic search only runs if the direct hash lookup results in a miss.
    • Asynchronous Writes: On a cache miss, Bifrost returns the provider's response immediately and stores the result in the background. The first request never blocks on a cache write.
    • Persistence: Cache entries persist across Bifrost restarts as they are stored in your configured vector store.
    • Supported APIs: Chat completions, text completions, Responses API (including WebSocket), embeddings, transcriptions, speech, and image generation (including streaming variants).