AIGNE Hub

The AIGNE Hub (ai-kit) and Payment Kit are being developed through a three-phase implementation plan focused on billing accuracy, performance observability, and feature expansion.

Core Components

• AIGNE Hub (ai-kit): An AI Provider Hub platform supporting 11+ providers, Credit-based billing, and V2 API performance (RPS >= 100).
• Payment Kit: The underlying payment infrastructure supporting Stripe, ArcBlock, and EVM channels, dynamic pricing (Quote Final Freeze), and Credit metering pipelines.

Implementation Phases

1. Phase 1: Billing Accuracy & Performance Baseline: Focuses on increasing Credit consumption queue concurrency (1 $\rightarrow$ 5), reducing DB operations per consumption, establishing benchmarking frameworks, and completing Delegation security checks.
2. Phase 2: Rate Sync & Financial Visibility: Focuses on integrating OpenRouter Pricing API, automated rate change detection/alerting, model-pricing-analyzer Skills, and financial dashboards (revenue/cost/profit).
3. Phase 3: Capability Expansion & UX Optimization: Focuses on WebSocket proxy support (OpenAI Realtime API), differentiated pricing (batch/realtime), Meter Event query optimization, and Audio API billing.

The AIGNE Hub performance benchmarks are designed to measure both comparative latency and isolated overhead. The benchmarking suite uses two distinct testing parts to provide a complete picture of performance:

1. Comparison Testing (Part 1): Compares AIGNE Hub against direct provider access and OpenRouter. This measures TTFB (Time to First Byte) across different concurrency levels using streaming. The key metric is Hub Overhead, calculated as total - providerTtfb.
2. Isolation Testing (Part 2): Uses a Mock Provider (an OpenAI-compatible local HTTP service) to isolate the Hub's overhead by removing provider latency. This focuses on non-streaming requests with realistic payloads.

Key Statistical Metrics

To ensure accuracy and stability, the following metrics are reported:

• Percentiles: p50, p75, p90, and p99.
• Stability: stddev (Standard Deviation) and CV (Coefficient of Variation). CV is used specifically to measure performance stability.

Benchmarking Best Practices

• Warmup: All tests perform 3 warmup iterations before recording data to avoid cold-start bias (e.g., DB connection pools or cache warming).
• Concurrency: Comparison tests use stepped concurrency levels rather than fixed iterations to better reflect performance under varying loads.
• Client Implementation: Uses Node.js native fetch to match standard Hub calling patterns.
• Configuration: Managed via .env files (no CLI argument parsing).

To determine the specific performance cost added by the AIGNE Hub proxy layer, use the hubOverhead calculation. This metric represents the processing time of the Hub itself, excluding provider latency and network delay.

Calculation Formula: hubOverhead = serverTiming.total.pXX - serverTiming.providerTtfb.pXX

Where pXX is a specific percentile (e.g., p50).

Available Server-Timing Phases:

• session: Session / AccessKey validation.
• resolveProvider: Provider resolution logic.
• modelCallCreate: Memory initialization of ModelCall context.
• preChecks: Credit balance and ModelRate checks.
• getCredentials: Fetching AI credentials and initializing client.
• providerTtfb: Time spent waiting for the provider's first chunk.
• ttfb: Client-side time from request to receiving the first byte.
• streaming: The streaming phase duration.
• usage: Time taken to create usage records.
• modelStatus: Updating model status and credential recovery.
• total: Total request duration.

The Model Selector maintains performance and quality data through a dual-track benchmark mechanism using AFS ash and agent-run.

1. Data Sources

• External Metadata: Direct ingestion of pricing, context length, supported parameters, and architecture from providers like OpenRouter, LiteLLM, and Cloud Arena.
• Internal Benchmarks:
  • Posterior Statistics: Aggregated from Hub usage data (e.g., model_calls table) to determine real-world P50/P99 latency and success rates.
  • Prior Testing: Standardized test sets (coding, reasoning, writing) sent to models to establish baseline performance.

2. Implementation Details

• External Sync: The Ops Blocklet's sync-models task is responsible for discovering new models and writing metadata to the Hub. The Selector consumes this data from the Hub rather than fetching it directly.
• Prior Testing Execution:
  • Uses X-Ops-Bypass headers to use dedicated credential pools (avoiding user balance consumption).
  • Employs small, standardized test sets to control costs.
  • Runs a small number of iterations (e.g., 3) per task to calculate averages.
• Quality Assessment:
  • Objective: Automated checks for code execution, math correctness, or schema compliance (zero LLM cost).
  • Subjective: Uses LLM-as-judge (e.g., Claude Opus) to score responses based on clarity or creativity using a structured schema: {score: 0-100, reason}.

3. Writing Results to Hub

After benchmarking, the Selector updates the Hub's models table using the update_benchmark or bulk_update_benchmarks interfaces, including a quality_source tag for auditing.

AIGNE Hub uses two distinct data strategies to balance performance and accuracy for usage monitoring:

1. Real-time Queries: Used for data that must be exact and granular.

   • Call History: Fetched from the ModelCall table to show individual request details (status, duration, error reasons).
   • Quota/Credits: Fetched via the payment-kit API to ensure balance information is always current.

2. Pre-aggregated Queries: Used for high-volume summary data to ensure fast dashboard loading.

   • Overview Stats & Trends: Fetched from the ModelCallStat table. This table contains pre-calculated aggregates (total usage, credits, calls, success rates, etc.) computed via daily cron jobs.
   • Data Consistency: The system merges real-time data from the current 'bucket' with historical pre-aggregated data from ModelCallStat to provide a seamless view.

Key Rule: Only quota and call history are real-time; all other summary and trend data are pre-aggregated.

The Data Retention Policy automatically migrates old data from the main SQLite database to separate quarterly archive files to prevent database growth from impacting performance.

Core Concepts

• Quarterly Partitioning: Data is archived into independent SQLite files named archive_YYYY_QN.db (e.g., archive_2025_Q1.db).
• Atomic Migration: The system uses SQLite ATTACH DATABASE to perform cross-database transactions, ensuring data consistency during the move from the main database to the archive.
• Automatic Cleanup: The system automatically deletes archive files that exceed the configured ARCHIVE_RETENTION_QUARTERS limit.
• Concurrency Control: To prevent multiple processes from running the archive task simultaneously, the system uses a file lock at {DATA_DIR}/archive.lock via the proper-lockfile library.

Data Flow

1. The system identifies data older than the configured retention months.
2. It calculates the target quarterly archive file.
3. It uses ATTACH DATABASE to move rows from the main table to the archive table.
4. It logs the execution details in the ArchiveExecutionLogs table in the main database.