Traceloop Hub

Traceloop Hub acts as an intermediary between your Client Application and LLM Providers (like OpenAI). It supports two primary configuration modes: YAML Mode (for simple deployments) and Database Mode (which includes a Management API).

Data flow follows this pattern:

1. Client App sends requests to Traceloop Hub.
2. Traceloop Hub routes requests to the LLM Provider.
3. Traceloop Hub also handles/forwards data to Observability backends (such as OTel or Metrics).

YAML Mode is designed for simple deployments and development environments. It uses a static config.yaml file for configuration and has no external dependencies like a database.

Key Characteristics:

• Static configuration via config.yaml.
• No management API.
• Operates on a single port (3000).
• Ideal for simple provider and model setups.

providers:
  - key: openai
    type: openai
    api_key: sk-...

models:
  - key: gpt-4
    type: gpt-4
    provider: openai

pipelines:
  - name: chat
    type: Chat
    plugins:
      - ModelRouter:
          models: [gpt-4]

Database Mode is intended for production environments that require dynamic configuration and hot reloading without service restarts.

Key Characteristics:

• Uses a PostgreSQL database for configuration storage.
• Includes a REST Management API for dynamic updates.
• Supports hot reloading and configuration polling.
• Uses a SecretObject system for managing credentials.
• Operates on dual ports: 3000 for the Gateway and 8080 for the Management API.

### Setup

1. Set up a PostgreSQL database.
2. Run migrations: `sqlx migrate run`.
3. Set environment variables:

```bash
HUB_MODE=database
DATABASE_URL=postgresql://user:pass@host:5432/db

Daemon mode runs the Traceloop Hub containers in the background. This is useful for persistent services where you do not want to see the logs directly in your terminal session.

To start the service in the background: docker compose --env-file env.template up -d

To stop and remove the containers: docker compose --env-file env.template down

To run Traceloop Hub locally using Docker Compose, follow these steps to prepare your environment:

1. Copy the provided template file env.template to a new file named .env.
2. Open .env and customize the credentials and connection strings to match your environment requirements.
3. Use the --env-file env.template flag with Docker Compose commands to ensure the correct configuration is loaded.

cp env.template .env

You can quickly get started with Traceloop Hub using either Docker or Cargo. Choose YAML Mode for simple, static deployments or Database Mode for dynamic configuration with a management API.

### Using Docker

```bash
# YAML Mode (simple deployment)
docker run -p 3000:3000 -v $(pwd)/config.yaml:/app/config.yaml traceloop/hub

# Database Mode (with management API)
docker run -p 3000:3000 -p 8080:8080 \
  -e HUB_MODE=database \
  -e DATABASE_URL=postgresql://user:pass@host:5432/db \
  traceloop/hub

Using Cargo

# Clone and build
git clone https://github.com/traceloop/hub.git
cd hub
cargo build --release

# YAML Mode
./target/release/hub

# Database Mode  
HUB_MODE=database DATABASE_URL=postgresql://user:pass@host:5432/db ./target/release/hub

Simple mode runs the Traceloop Hub containers in the foreground. This is useful for development and debugging as it streams the container logs directly to your terminal.

docker compose --env-file env.template up

If the Hub is running without an active configuration, it uses a fallback router created by create_no_config_router. This router explicitly listens for common LLM endpoints:

• POST /chat/completions
• POST /completions
• POST /embeddings

When these endpoints are called without a configuration, the Hub returns a 404 Not Found error.

When managing providers or configuring plugins (like Tracing), you can use the SecretObject abstraction to define how sensitive values like API keys are retrieved. This prevents hardcoding secrets in your configuration.

Supported types:

• literal: A direct string value. Use {"type": "literal", "value": "your-secret"}.
• kubernetes: A reference to a Kubernetes secret. Use {"type": "kubernetes", "secret_name": "my-secret", "key": "api-key", "namespace": "my-namespace"}.
• environment: A reference to an environment variable. Use {"type": "environment", "variable_name": "MY_ENV_VAR"}.

{
  "type": "environment",
  "variable_name": "TRACING_API_KEY"
}

When running Traceloop Hub, you may interact with one or two distinct server interfaces depending on your HUB_MODE:

1. LLM Gateway: This server is always present. It handles the actual LLM requests, routing them through configured providers, models, and pipelines. It listens on the port defined by PORT (default 3000).

2. Management API: This server is only available in database mode. It provides an interface for managing the gateway's configuration via the database. It listens on the port defined by MANAGEMENT_PORT (default 8080).