Portkey AI Gateway

The Portkey AI Gateway provides several categories of features to enhance LLM application development:

Reliable Routing

• Fallbacks: Automatically switch to a different provider or model if a request fails. You can specify which error codes trigger a fallback.
• Automatic Retries: Retry failed requests up to 5 times using an exponential backoff strategy.
• Load Balancing: Distribute requests across multiple API keys or providers using weights for high availability.
• Request Timeouts: Set granular timeouts to terminate requests that exceed a specific duration.
• Multi-modal LLM Gateway: Use a unified OpenAI-compatible signature to call vision, audio (TTS/STT), and image generation models.
• Realtime APIs: Support for OpenAI realtime APIs via an integrated websockets server.

Security & Accuracy

• Guardrails: Verify inputs and outputs against 40+ pre-built checks or bring your own custom guardrails.
• Secure Key Management: Use personal keys or generate virtual keys on the fly.
• Role-based access control: Granular control over users, workspaces, and API keys.
• Compliance: SOC2, HIPAA, GDPR, and CCPA compliant.

Cost Management

• Smart caching: Supports both simple and semantic caching to reduce latency and costs.
• Usage analytics: Monitor request volume, latency, costs, and error rates.
• Provider optimization: Automatically switch to the most cost-effective provider based on pricing and usage.

Collaboration & Workflows

• Agents Support: Integrates with frameworks like Autogen, CrewAI, LangChain, LlamaIndex, Phidata, and Control Flow.
• Prompt Template Management: Collaborative versioning and management of prompt templates.

Portkey allows you to manage production reliability features like Semantic Caching, Fallbacks, and Load Balancing via Configs. You define these configurations in the Portkey dashboard and apply them by passing the config ID during client instantiation.

Example Config JSON

To enable semantic caching and a fallback strategy from mistral-medium to mistral-tiny:

{
	"cache": {"mode": "semantic"},
	"strategy": {"mode": "fallback"},
	"targets": [
		{
			"provider": "mistral-ai", "api_key": "...",
			"override_params": {"model": "mistral-medium"}
		},
		{
			"provider": "mistral-ai", "api_key": "...",
			"override_params": {"model": "mistral-tiny"}
		}
	]
}

Applying a Config

Pass the Config ID (e.g., pp-mistral-cache-xx) to the config parameter when initializing the Portkey client.

# Python
portkey = Portkey(
    api_key="PORTKEY_API_KEY",
    config="pp-mistral-cache-xx"
)

// JavaScript
const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY",
    config: "pp-mistral-cache-xx"
});

You can define complex routing behaviors using a config object.

• Loadbalancing: Use strategy: { mode: 'loadbalance' } to distribute traffic across multiple targets. Use the weight property (e.g., 0.5) to specify the proportion of traffic sent to each target.
• Fallbacks: Use strategy: { mode: 'fallback' } to define a list of targets that will be tried sequentially if the preceding target fails.
• Nested Strategies: You can nest a fallback strategy inside a loadbalance target to ensure that if one provider in the loadbalancer fails, it immediately tries a backup provider.
• Overrides: Use override_params within a target to change specific parameters (like model or max_tokens) for that specific provider.

const config = {
  strategy: {
    mode: 'loadbalance'
  },
  targets: [
    {
      virtual_key: process.env['ANTHROPIC_VIRTUAL_KEY'],
      weight: 0.5,
      override_params: {
        max_tokens: 200,
        model: 'claude-3-opus-20240229'
      }
    },
    {
      strategy: {
        mode: 'fallback'
      },
      targets: [
        {
          virtual_key: process.env['OPENAI_VIRTUAL_KEY']
        },
        {
          virtual_key: process.env['AZURE_OPENAI_VIRTUAL_KEY']
        }
      ],
      weight: 0.5
    }
  ]
};

The AI Gateway uses Configs to define operational logic such as retries, fallbacks, load balancing, and guardrails. You can attach these configurations to your client using the .with_options(config=...) method.

Common configuration keys include:

• retry: Defines the number of attempts for failed requests.
• output_guardrails: A list of rules to inspect model responses. For example, you can use default.contains with an operator (like none) and a list of words to deny responses containing specific terms.

config = {
  "retry": {"attempts": 5},

  "output_guardrails": [{
    "default.contains": {"operator": "none", "words": ["Apple"]},
    "deny": True
  }]
}

# Attach the config to the client
client = client.with_options(config=config)

client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Reply randomly with Apple or Bat"}]
)

# Note: In this example, the response will always be "Bat" because the 
# guardrail denies any reply containing "Apple".

The cookbook provides several guides for implementing core Gateway capabilities:

• Caching: Use Portkey cache to reduce LLM costs and latency (enable-cache.md).
• Automatic Retries: Configure the gateway to automatically retry requests upon LLM failures (automatic-retries-on-failures.md).
• Gateway Configuration: Learn how to write your first Gateway Config (writing-your-first-gateway-config.md).
• Fallback Mechanisms: Implement automatic fallbacks (e.g., from OpenAI to Azure) to ensure high availability (fallback-from-openai-to-azure.ipynb).
• Image Generation: Use the Gateway for image generation tasks (image-generation.ipynb).