OrcaRouter Lite

OrcaRouter Lite provides several core capabilities for managing LLM requests:

• OpenAI Compatibility: Supports standard Chat-Completions and Streaming (SSE).
• model="auto": Automatically routes requests to the most cost-effective suitable model.
• Hosted Fallback: Ability to use the hosted version as an upstream provider.
• Security: Encrypted BYOK (Bring Your Own Key) storage in memory.
• Analytics: Local dashboard for monitoring usage.
• Caching: Provider-agnostic prompt caching.
• Integrations: Compatible with tools like Continue.dev, Aider, LangChain, Cursor, and Vercel-AI-SDK.

Note: Embeddings and Image-Gen-Proxy are planned for future releases.

The signature feature of OrcaRouter is model="auto". When you pass this value, OrcaRouter automatically selects the cheapest model from your configured providers that meets the specific capability requirements of your request (such as tools, vision, or json_mode).

This eliminates the need for manual routing logic or hardcoded model names in your application code. The specific model selected is returned in the x-orca-resolved-model response header.

client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "What's in this image?"},
        {"type": "image_url", "image_url": {"url": "data:..."}},
    ]}],
)
# → Routes to the cheapest model supporting VISION

The model="auto" feature is the core capability of OrcaRouter. When you pass model="auto" in a chat completion request, OrcaRouter automatically selects the least expensive model among your configured providers that meets the specific requirements of your request (such as tools support, vision capabilities, or json_mode).

This eliminates the need for manual routing logic or cost-optimization code in your application. The specific model chosen by the router is returned in the x-orca-resolved-model response header.

client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "What's in this image?"},
        {"type": "image_url", "image_url": {"url": "data:..."}},
    ]}],
)
# → routes to the cheapest VISION-compatible model covered by your keys

You can combine self-hosting with the hosted service by setting the ORCAROUTER_API_KEY environment variable with your sk-orca-* key from www.orcarouter.ai. This makes the hosted service an additional provider in your routing chain, covering models that your local keys might not support. This is useful for testing without local keys, logging via Lite, or providing a safety net (failover) if local providers fail.

# .env
ORCAROUTER_API_KEY=sk-orca-hosted-abc...

OrcaRouter implements a deterministic prompt cache for requests with temperature=0 or a pinned seed. If a request with the same payload is repeated, it is served from the cache, bypassing the upstream provider and costing $0.

• If a cache hit occurs, the response includes the header x-orca-cache: HIT.
• If it is a miss, it returns x-orca-cache: MISS.
• The backend uses Redis if REDIS_URL is set; otherwise, it uses an in-process LRU cache.

# First request (Miss)
$ curl ... -d '{"model":"auto","messages":[...], "temperature": 0}' -i
HTTP/1.1 200 OK
x-orca-cache: MISS
x-orca-resolved-model: gpt-4o-mini

# Second request (Hit)
$ curl ... # same payload
HTTP/1.1 200 OK
x-orca-cache: HIT

OrcaRouter Lite is built as a FastAPI application with an OpenAI-compatible interface. Key components include:

• Routing & Intelligence: auto_routing.py handles the model="auto" logic and cost scoring. router_cache.py manages single-workspace routing.
• Caching: prompt_cache.py provides cross-provider exact-match caching using either Redis or an in-memory LRU.
• API Surface: The /v1/chat/completions endpoint supports both blocking and streaming (SSE) responses. Other endpoints include /v1/models, /v1/providers (BYOK CRUD), /v1/routing (strategy config), and /v1/analytics.
• Security: middleware/auth.py validates sk-orca-* keys, and the auth package uses AES-256-GCM for encryption.
• Adapters: The litellm_adapter package provides a router wrapper and a catalog of over 100 models.

OrcaRouter provides deterministic request caching. If a request is deterministic (e.g., temperature=0 or a fixed seed), it is served from the cache on subsequent calls. This works across all providers.

• If using Redis (via REDIS_URL), the cache is distributed.
• If Redis is not configured, an in-process LRU cache is used.
• Cache hits are identified by the x-orca-cache: HIT header and cost $0.

# First request (MISS)
$ curl ... -d '{"model":"auto","messages":[...], "temperature": 0}' -i
HTTP/1.1 200 OK
x-orca-cache: MISS
x-orca-resolved-model: gpt-4o-mini

# Second request (HIT)
$ curl ... # same payload
HTTP/1.1 200 OK
x-orca-cache: HIT

OrcaRouter includes built-in prompt caching and cost analysis features.

Prompt Caching: For deterministic requests (temperature=0 or a fixed seed), OrcaRouter caches the response. If a cache hit occurs, the response is returned immediately with the header x-orca-cache: HIT at $0 cost.

Savings Analytics: You can compare your actual spend against a baseline (e.g., 'what if I always used GPT-4') using the analytics endpoint.

# Check savings against a GPT-4 baseline for the last 7 days
curl ... /v1/analytics/savings?baseline=gpt-4o&days=7

# Example Cache Hit Header
# HTTP/1.1 200 OK
# x-orca-cache: HIT

OrcaRouter Lite implements prompt caching for deterministic requests (where temperature=0 or a seed is provided). This works across all providers.

• If a match is found, the response is served instantly with the header x-orca-cache: HIT at $0 cost.
• If no match is found, the header x-orca-cache: MISS is returned.
• The backend uses Redis if REDIS_URL is set; otherwise, it uses an in-process LRU cache.

# First request (MISS)
$ curl ... -d '{"model":"auto","messages":[...], "temperature": 0}' -i
HTTP/1.1 200 OK
x-orca-cache: MISS
x-orca-resolved-model: gpt-4o-mini

# Second request (HIT)
$ curl ...  # same payload
HTTP/1.1 200 OK
x-orca-cache: HIT