DOCSEARCHbyAlgolia
Submit Docs

AxonHub Documentation

repository·unstable·Indexed 25 days ago

https://github.com/looplj/axonhub

AxonHub is an AI gateway platform that enables seamless switching between LLM providers such as OpenAI, Anthropic, and Gemini using existing SDKs without code changes. It provides unified API access, observability, and cost management. The documentation covers Kubernetes deployment via Helm, production configuration best practices, translation management for the frontend, and detailed integration testing for the Anthropic Go SDK, including streaming, tool calling, and multi-turn conversations.

Tokens
117.5K
Snippets
246
Records
683
Agent score
89%

What's inside AxonHub

  1. Overview of AxonHub Integrated API

    unstable

    AxonHub acts as an integrated API gateway that supports both OpenAI Chat Completions and Anthropic Messages API. This allows for cross-provider model access without changing your existing client code:

    • Call Anthropic models using the OpenAI SDK: Access Claude models while using OpenAI-compatible code.
    • Call OpenAI models using the Anthropic API: Access GPT models using Anthropic's native message format.
    • Call OpenAI models using the Gemini API: Access GPT models using Gemini's native format.
    • Automatic API Conversion: AxonHub handles the format transformations automatically.
    • Zero Code Changes: Existing OpenAI or Anthropic client implementations can be used directly by pointing them to AxonHub.

  2. Overview of AxonHub AI Development Platform

    unstable

    AxonHub is an all-in-one AI gateway designed to allow developers to switch between different model providers without changing a single line of code. It acts as a transparent proxy that converts requests from standard SDKs (like OpenAI or Anthropic) to work with any supported model provider.

    Key Benefits

    • Avoid Vendor Lock-in: Instantly switch between models like GPT-4, Claude, or Gemini by updating configuration instead of refactoring code.
    • Unified API Format: Interface with over 10 different providers using a single API format.
    • Observability: Built-in request tracing and timelines for faster debugging.
    • Cost Management: Real-time tracking of usage, including input, output, and cache tokens.
    • Reliability: Smart load balancing with automatic failover (under 100ms) to ensure continuous service.

  3. Understand Multiple Tool Call Scenarios

    unstable

    The OpenAI integration supports several multi-tool execution patterns:

    • Sequential Execution: Processing multiple tools one after another (e.g., getting weather data and then performing a calculation based on that data) in a single complex query.
    • Parallel Execution: Using the parallel_tool_calls parameter to allow the model to make multiple independent tool calls simultaneously when multiple pieces of information are needed.
    • Forced Tool Choice: Forcing the model to use a specific function for specialized queries (e.g., ensuring mathematical queries always use the calculate tool).

    Note: Parallel tool calls may not be supported by all models, and these scenarios typically require higher token limits and complex reasoning.

  4. Understand AxonHub Data Migrations

    unstable

    Data migrations in AxonHub are used to provide one-off backfill or repair logic specifically when upgrading an existing AxonHub instance to a newer version. They are intended to supplement or fix historical data to meet the requirements of a new version.

    Key Characteristics

    • Upgrade Only: Unlike schema migrations which run in every scenario, data migrations only run during upgrade scenarios.
    • Initialization Requirement: Data migrations only execute if the system has already been initialized (system_initialized=true).
    • Fresh Installs: For new installations where the system initialization flow has not yet been executed, data migrations are skipped.

    Execution Order

    1. Schema Migrations: First, schema migrations run to synchronize database structures.
    2. Application Startup: Once the application detects the system is initialized, it executes data migrations in version order (based on the order of migrator.Register(...)).
    3. Version Update: The system version is updated only after the data migrations have successfully finished.

  5. Understand AxonHub Load Balancing Strategies

    unstable

    AxonHub uses a multi-strategy load balancing system to distribute requests across channels. The final channel score is the sum of scores from several strategies. Trace sticky selection is applied first if a cached channel is eligible.

    Strategies and Scoring

    1. Trace Sticky Selection: Prioritizes the cached previous channel for a specific trace/thread to maintain affinity.
    2. ErrorAwareStrategy (Priority 1): Monitors health. Healthy channels get 200 points. Penalties include:
      • 1 consecutive failure: -50 points
      • 2 consecutive failures: -100 points
      • 3+ consecutive failures: -150 points
      • Recent errors (last 5 min): -20 points each.
    3. WeightRoundRobinStrategy (Priority 2): Distributes load based on channel weight. Score: 10-150 points. Uses the formula: normalizedCount = effectiveCount / (weight / 100.0).
    4. LatencyAwareStrategy (Priority 3): Prefers lower latency. Score: 0-80 points. Streaming requests prioritize lower EWMA first-token latency and higher EWMA output throughput; non-streaming requests prioritize lower EWMA end-to-end latency.
    5. RateLimitAwareStrategy (Priority 4): Respects RPM/TPM/concurrency limits and 429 cooldowns. Score: -10000 to 100 points.

  6. Understand the S3 Class A Operation Refactor Plan

    unstable

    The project is implementing a Pragmatic Hybrid architecture to reduce excessive S3 Class A operations (such as PutObject, ListObjectsV2, and multipart uploads) caused by using a POSIX file system abstraction (afero) over object storage.

    Key Goals:

    • Reduce logical writes from 3 PutObject calls to 1.
    • Eliminate ListObjectsV2 calls during GC deletions by using idempotent DeleteObject calls.
    • Prevent ListObjectsV2 fallbacks when reading non-existent keys (ensuring GetObject 404 maps to os.ErrNotExist).
    • Implement adaptive retries and controlled chunking for large objects to prevent rate-limiting avalanches.
    • Maintain support for all five backends (Database, Fs, S3, GCS, WebDAV) with zero data migration required.

  7. Understand the LLM Pipeline Architecture

    unstable

    The AxonHub LLM Pipeline is designed to provide a unified interface for interacting with various AI providers (OpenAI, Anthropic, Gemini, etc.). It follows a standardized request/response lifecycle through a series of transformation and middleware stages:

    1. Inbound Processing (Standardization): An InboundTransformer converts the incoming HTTP Request into a unified LLMRequest format.
    2. Middleware (Pre-processing): The request passes through middleware for Model Mapping, Channel Selection, MaxToken Enforcement, and Request Persistence.
    3. Outbound Processing (De-standardization): An OutboundTransformer converts the unified LLMRequest into the specific format required by the target AI provider (e.g., OpenAI Transformers, Anthropic Transformers).
    4. Execution: The LLM Pipeline executes the request and manages retry logic.
    5. Response Transformation: The provider's raw response is converted back into a unified LLMResponse via provider-specific Inbound Transformers.
    6. Middleware (Post-processing): The response undergoes Response Persistence, Usage Tracking, and potential Channel Switching (for retries).
    7. Client Delivery: The final response is delivered to the client via SSE or JSON.

  8. Use AI SDK Transformers for text and data streaming

    unstable

    The aisdk package provides transformers to implement the AI SDK protocol, supporting both legacy text streaming and the modern Data Stream Protocol (SSE).

    • TextTransformer: Implements the original text-based streaming format for backward compatibility.
    • DataStreamTransformer: Implements the AI SDK Data Stream Protocol using Server-Sent Events (SSE), supporting structured streaming of text, tool calls, and control signals.

  9. Understand Model Management and Routing Concepts

    unstable

    AxonHub uses a three-layer pipeline to process AI requests. Model Management sits in the middle of this pipeline to determine which provider (Channel) should handle a specific model request.

    Core Concepts

    • Model: An abstract name exposed to the client (e.g., gpt-4 or claude-sonnet).
    • Channel: An actual connection to an AI provider (e.g., OpenAI, Anthropic, or a specific courier-like service).
    • Model Association: The routing rule that maps a requested Model to a specific Channel and upstream model.
    • Developer Rule: A reusable rule configured at the developer group level. Models from the same developer inherit these rules by default.

    Request Flow

    1. API Key Profile renames: The incoming request model name is mapped.
    2. Model Association: The system looks up associations for the requested model to select a channel.
    3. Channel renames: The model name is mapped to the specific upstream provider's model ID.
    4. Send upstream: The request is dispatched to the selected channel.

  10. Verify multi-turn conversation and tool integration in a single trace

    unstable

    The TestSingleTraceMultipleCalls test ensures that AxonHub can manage complex, multi-step AI workflows within one trace. It specifically validates:

    • Multi-turn conversations: Maintaining context across sequential calls (e.g., a greeting followed by a task breakdown).
    • Function calling: Executing tools (like a calculator function) during the trace.
    • Context preservation: Ensuring information from the initial call is available for final calls (e.g., organizing a timeline based on previous discussion).
    • Response validation: Verifying mathematical accuracy and calculation results within the trace flow.

  11. OpenAI Conversation Test Scenarios and Features

    unstable

    The conversation integration tests cover the following capabilities:

    Features Tested

    • Multi-turn conversation management: Handling back-and-forth flows.
    • Context preservation: Maintaining state and context across multiple turns.
    • System prompt influence: Ensuring system instructions persist and guide the assistant.
    • Tool integration: Verifying tools are available and usable within the conversational flow.
    • Message history: Maintaining the correct order of message types.

    Message Types Supported

    • System Messages: Instructions guiding assistant behavior.
    • User Messages: Inputs and questions from the user.
    • Assistant Messages: AI responses and tool calls.
    • Tool Messages: Results returned from tool executions.

    Expected Behavior

    • Context from previous turns must be maintained.
    • Tools must remain available throughout the conversation.
    • System instructions must influence the entire session.
    • Message history must be maintained in the correct order for coherent flow.

  12. Understand the AxonHub Transformation Flow Architecture

    unstable

    AxonHub uses a bidirectional data transformation pipeline to facilitate communication between clients and various AI providers (like OpenAI, Anthropic, DeepSeek, and Zhipu AI). The architecture is divided into four main layers:

    1. Client Layer: Web apps, mobile apps, or API clients that send requests in an OpenAI-compatible format.
    2. AxonHub Gateway:
      • Inbound Processing: Parses, validates, and normalizes incoming requests into a unified internal format.
      • Core Processing: Handles channel selection (routing), load balancing, and failover logic.
      • Outbound Processing: Adapts the unified request to provider-specific formats, maps protocols (HTTP/WebSockets/SSE), and converts data formats.
      • Observability: Provides tracing (OpenTelemetry), metrics, and structured logging.
    3. Provider Layer: The target AI models and APIs.