PromptCache

Version 0.4.0 introduced several key capabilities for production workloads:

• API Authentication: All management endpoints are now protected by Bearer-token middleware using the API_AUTH_TOKEN environment variable.
• Streaming (SSE): Full support for stream: true across OpenAI, Mistral, and Claude, including streamed cache hits.
• Runtime Config API: Ability to update similarity thresholds and gray-zone settings live via GET/PATCH /v1/config.
• Cache Warming: Ability to bulk pre-populate the cache from historical prompt/response pairs using POST /v1/cache/warm.

The documentation is organized into the following files within the docs/ directory:

• index.md: Home page
• getting-started.md: Installation & quick start guides
• api-reference.md: REST API documentation
• configuration.md: Configuration guide
• providers.md: Provider setup & comparison
• _config.yml: Jekyll configuration

PromptCache persists several types of data which should be treated as potentially sensitive storage:

• Prompts
• Model responses
• Embeddings
• Cache metadata
• Historical prompt/response pairs (supplied via cache warming)

Retention Note: While PromptCache has a configurable TTL (defaulting to 24 hours), this is not a complete retention policy. You must manage backups, logs, and exports separately to meet your specific deletion and retention requirements.

The X-Cache-Namespace header is a partition key, not an authorization mechanism.

Important Security Rule: Do not pass an arbitrary namespace directly from an untrusted end user if namespaces are intended to serve as security boundaries. Instead, use a trusted gateway or application layer to derive or validate the namespace from an authenticated context (e.g., tenant ID or organization ID) before forwarding the request to PromptCache.

Isolation Behavior:

• The default partition uses an Approximate Nearest Neighbor (ANN) index.
• Explicit namespaces use a filtered exact scan. This ensures that semantic lookup is strictly restricted to embeddings within that specific namespace, preventing ANN candidates from crossing namespace boundaries. This design prioritizes isolation correctness over the performance of ANN for namespaced traffic.

PromptCache uses a two-stage verification strategy to ensure cached responses are semantically correct rather than just textually similar. It categorizes requests based on similarity scores:

1. High similarity (>0.95): Results in a direct cache hit.
2. Low similarity (<0.70): The cache is skipped, and the request goes directly to the LLM provider.
3. Gray zone (0.70 - 0.95): An intent check is performed using a small, cost-effective verification model to decide if the cache should be used.

PromptCache uses a two-threshold strategy to balance speed and accuracy:

1. High similarity (score $\ge$ CACHE_HIGH_THRESHOLD): Direct cache hit; the cached response is returned immediately.
2. Low similarity (score $<$ CACHE_LOW_THRESHOLD): Cache miss; the request is forwarded to the upstream provider.
3. Gray zone (CACHE_LOW_THRESHOLD $\le$ score $<$ CACHE_HIGH_THRESHOLD): If ENABLE_GRAY_ZONE_VERIFIER is enabled, an optional intent verification step is performed using a smaller model to decide if the prompt is a match.

Warning: Semantic similarity is probabilistic and is not an authorization mechanism. Do not use it to separate users or tenants.

PromptCache uses a semantic understanding approach to determine if a prompt should be served from the cache or sent to the LLM. It employs a three-stage verification strategy based on similarity scores to ensure accuracy:

1. High similarity (≥70%): The prompt is considered a match; PromptCache returns a direct cache hit immediately.
2. Low similarity (<30%): The prompt is considered too different; PromptCache skips the cache and sends the request directly to the LLM.
3. Gray zone (30-70%): The prompt is in an ambiguous range; PromptCache uses LLM verification to ensure the cached response is semantically correct before serving it.

This mechanism balances speed and cost-savings with the need for high semantic accuracy.

To deploy PromptCache using Docker Compose, create a docker-compose.yml file with the following configuration. This setup includes volume persistence for the Badger database and a health check for Kubernetes readiness. Ensure you have an OPENAI_API_KEY available in your environment.

version: '3.8'

services:
  prompt-cache:
    image: messkan/prompt-cache:0.3.0
    ports:
      - "8080:8080"
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - EMBEDDING_PROVIDER=openai
      - LOG_LEVEL=info
    volumes:
      - ./badger_data:/app/badger_data
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health/ready"]
      interval: 30s
      timeout: 10s
      retries: 3

Then, execute:

export OPENAI_API_KEY=your_key_here
docker-compose up -d

To use Claude, set the EMBEDDING_PROVIDER environment variable to claude. Note that Claude requires two API keys because it uses Voyage AI for embeddings. You must provide both ANTHROPIC_API_KEY and VOYAGE_API_KEY. This setup is best for high-quality reasoning and safety-critical applications.

export EMBEDDING_PROVIDER=claude
export ANTHROPIC_API_KEY=your-anthropic-api-key
export VOYAGE_API_KEY=your-voyage-api-key