DOCSEARCHbyAlgolia
Submit Docs

Manifest Documentation

repository·main·Indexed 27 days ago

https://github.com/mnfst/manifest

An open-source LLM gateway and observability tool for AI agents. Manifest provides a unified, OpenAI-compatible endpoint for custom routing across 300+ models and 31 providers, including API keys, subscriptions, and local models (Ollama, LM Studio, llama.cpp). Key features include cost and token tracking, automatic model fallbacks, and an Auto-fix system to detect and repair 4xx provider errors. Supports deployment via Docker, Easypanel, Railway, Render, DigitalOcean, AWS, GCP, and Apple Containers, with a dedicated community node for n8n.

Tokens
24.6K
Snippets
55
Records
147
Agent score
92%

What's inside Manifest

  1. What is Manifest?

    main

    Manifest is a smart model router designed for AI agents (e.g., OpenClaw, Hermes) that use OpenAI-compatible HTTP APIs. It acts as an intermediary between your agents and various AI providers, routing requests to the most appropriate model based on complexity to optimize for speed, cost, and reliability.

    Key features include:

    • Unified Endpoint: One endpoint to access every provider.
    • Automatic Fallbacks: If a model fails, the request is automatically picked up by the next available model.
    • Budget Limits: Set limits to prevent exceeding your budget.
    • Self-hosted: Complete control over your requests, providers, and data.

  2. Overview of Manifest

    main

    Manifest is an open-source LLM gateway designed for AI agents and applications. It provides a single OpenAI-compatible endpoint that routes queries to various backends, including API keys, subscriptions, local models, and custom providers. This prevents single-provider lock-in and enables advanced features like:

    • Custom Routing: Direct queries to specific API keys, subscriptions, local models, or custom providers.
    • Full Body Logs: Capture both success and error messages for all queries.
    • Cost Tracking: Monitor spending and set up notifications and limits.
    • Self-healing: Automatically fallback to different models if a query fails.

  3. Understand the role of the manifest package

    main

    The manifest package serves as the single source of truth for the canonical Manifest release version. It contains no functional code. Instead, it acts as a versioning anchor for the entire ecosystem:

    • Changesets: It is the only package targeted by changesets to represent a release of the whole system.
    • Docker Releases: The version defined in this package's package.json is used to tag the manifestdotbuild/manifest Docker image.
    • Changelog: The CHANGELOG.md within this package provides the unified release history for Manifest.

  4. Supported AI Providers and Model Routing

    main

    Manifest connects to over 300 models through 31 built-in provider connections. It supports three primary ways to access models:

    1. Bring your own API key: Connect directly to providers like OpenAI, Anthropic, Google, xAI, DeepSeek, and more.
    2. Subscription flows: Use existing subscriptions from services like GitHub Copilot, Kiro, Command Code, or ClinePass.
    3. Local models: Run models locally using Ollama, LM Studio (default port 1234), or llama.cpp (default port 8080).

    Model Routing: All requests are routed through a unified OpenAI-compatible endpoint. You can use the special model identifier "model": "auto" to let Manifest automatically select the appropriate model from your connected providers.

  5. Understand Auto-fix (self-healing requests) behavior

    main

    Auto-fix is a feature that intercepts requests that failed with a repairable error (specifically request-side 4xx errors like 400, 404, or 422). It sends the failed request and the provider's response to an external healing service, receives a patched request, and re-sends it once before the standard fallback chain runs.

    Key constraints and behaviors:

    • Single Attempt Only: The current implementation uses runHealOnce. There is no retry budget or autofix_max_attempts configuration.
    • Error Scope: Only request-side 4xx errors are healed (e.g., malformed parameters, unknown models). It does not heal 401/403 (auth), 429 (rate limit), 408 (timeout), 5xx (provider availability), or 200-but-bad responses.
    • Execution Order: Auto-fix runs first. If the healing attempt fails or is unfixable, the existing fallback chain acts as the safety net.
    • Streaming: Healing only occurs for requests that fail before the first byte is streamed. Mid-stream failures are out of scope.
    • Idempotency: Because it only targets 4xx errors (where no tokens were generated or side effects occurred), re-sending a patched version is considered idempotent-safe.

  6. Understand Manifest Auto-fix functionality

    main
    Manifest acts as an LLM proxy/gateway. The Auto-fix feature (powered by Phoenix) automatically detects repairable 4xx errors from providers, patches the request, and retries it. This allows developers to receive successful responses even when an upstream provider initially returns an error, without needing to handle the error manually in their application logic.

  7. Understand the Manifest Analytics Model (Requests vs. Attempts)

    main

    Manifest analytics are divided into two distinct domains. It is critical not to mix them when calculating totals or success rates:

    1. The Request world (agent side): Represents one logical request from an agent to Manifest. A Request belongs to an agent and has one caller-visible outcome. It may contain zero or more Provider Attempts. Recovery is a Request-level concept.
    2. The Attempt world (provider side): Represents every individual call Manifest makes to an AI provider. A Request has zero or more Attempts. Attempts belong to providers, Provider Connections, and models. An Attempt may be triggered by fallback or Auto-fix.

  8. Make your first routed request to Manifest

    main

    Once installed, follow these steps to route AI requests:

    1. Connect a Provider: In the dashboard, go to Providers → Usage-based (for API keys like OpenAI/Anthropic), Subscriptions, or Local (for Ollama/LM Studio).
    2. Get your Agent Key: Create an agent and copy its key (starts with mnfst_) from the agent's Settings.
    3. Use the OpenAI-compatible endpoint: Point your SDK or tool to http://localhost:2099/v1 using your mnfst_ key.

    Use "model": "auto" to let Manifest handle routing based on your configuration.

    curl -X POST http://localhost:2099/v1/chat/completions \
  -H "Authorization: Bearer mnfst_YOUR_KEY_HERE" \
  -H "Content-Type: application/json" \
  -d '{"model": "auto", "messages": [{"role": "user", "content": "Hello"}]}'

  11. Expose Manifest on a Local Area Network (LAN)

    main

    By default, the Compose setup binds to 127.0.0.1 (localhost only). To allow other devices on your network to access the dashboard:

    1. Edit docker-compose.yml: Change the ports mapping from "127.0.0.1:2099:2099" to "2099:2099".
    2. Update .env: Set BETTER_AUTH_URL to the IP address or domain used to reach the host (e.g., http://192.168.1.20:2099).
    3. Restart: Run docker compose up -d.

    Note: If you see a blank page on a LAN IP, ensure you are using a recent image version to avoid CSP issues with private-IP hosts.