gollm

When using memory, gollm uses structured messages by default. This is highly compatible with providers like Anthropic that support prompt caching. You can explicitly control cache behavior using AddStructuredMessage.

Use the cache_control parameter (e.g., "ephemeral") to mark specific messages for caching, which reduces token costs and improves latency for repeated context.

// Add a message with cache control
// "ephemeral" caching is recommended for most use cases
llm.AddStructuredMessage("user", "Here is a long document to analyze...", "ephemeral")

// Generate - subsequent requests with the same context will use cached tokens
response, err := llm.Generate(ctx, gollm.NewPrompt("Summarize the key points"))

Gollm uses a provider system to connect to different LLM APIs through a consistent interface. Providers are categorized by their API format:

• OpenAI-compatible: Uses the OpenAI chat completion API format (e.g., OpenAI, Azure OpenAI, Groq, DeepSeek).
• Anthropic-compatible: Uses the Anthropic Claude API format (e.g., Anthropic, Claude).
• Custom: Requires a unique implementation for non-standard API formats.

The system uses a registry to manage providers. When you request a provider, the system either uses a built-in implementation or creates a GenericProvider based on a ProviderConfig.

OpenRouter supports model fallback and automatic model selection:

• Model Fallbacks: Specify a list of models to use if the primary model fails using llm.SetOption("fallback_models", []string{...}).
• Auto-Routing: Use the special model name openrouter/auto to let OpenRouter automatically select the best model for your prompt.

// Fallbacks
llm.SetOption("fallback_models", []string{"openai/gpt-4o", "gryphe/mythomax-l2-13b"})

// Auto-routing
llm, err := gollm.NewLLM(
    gollm.SetProvider("openrouter"),
    gollm.SetAPIKey(apiKey),
    gollm.SetModel("openrouter/auto"),
)

When using structured messages, you can optimize message processing by applying cache control instructions to individual messages. This is particularly useful for reducing latency and costs with APIs that support prompt caching.

Use the AddStructuredMessage method to specify a cache control option:

• "ephemeral": The message can be cached but may be evicted if space is needed.
• "persistent": The message should be kept in cache indefinitely.
• "" (empty string): No special caching instruction.

// Add message with cache control
memLLM.AddStructuredMessage("user", "Hello, who are you?", "ephemeral")

gollm supports two ways of handling conversation history in memory:

1. Structured Messages (Default): Each message is preserved as a separate object with its own metadata. This is more efficient for caching (especially with providers like Anthropic) and preserves conversation structure better.
2. Flattened Messages (Legacy): Conversations are flattened into a single text string.

By default, all newly created LLMWithMemory instances use structured messages. You can revert to the flattened approach by calling SetUseStructuredMessages(false).

To use memory, first create a base LLM instance, then wrap it using NewLLMWithMemory. The resulting instance uses structured messages by default. You may need to type-assert the result to *llm.LLMWithMemory to access memory-specific methods.

// Create base LLM instance
baseLLM, err := llm.NewLLM(cfg, logger, registry)
if err != nil {
    log.Fatalf("Failed to create LLM: %v", err)
}

// Create LLM with memory (uses structured messages by default)
memoryLLM, err := llm.NewLLMWithMemory(baseLLM, 4000, cfg.Model)
if err != nil {
    log.Fatalf("Failed to create LLM with memory: %v", err)
}

// Cast to access specific methods
memLLM := memoryLLM.(*llm.LLMWithMemory)

To use OpenRouter, initialize an LLM instance using gollm.NewLLM with the gollm.SetProvider("openrouter") option. You must provide an API key via gollm.SetAPIKey(apiKey) and specify a model using gollm.SetModel("model-name") (e.g., anthropic/claude-3-5-sonnet).

Prerequisites:

• Go 1.18 or later
• An OpenRouter API key from openrouter.ai

llm, err := gollm.NewLLM(
    gollm.SetProvider("openrouter"),
    gollm.SetAPIKey(apiKey),
    gollm.SetModel("anthropic/claude-3-5-sonnet"),
    gollm.SetTemperature(0.7),
    gollm.SetMaxTokens(1000),
)

prompt := gollm.NewPrompt("What are the main features of OpenRouter?")
response, err := llm.Generate(ctx, prompt)

To use the Azure OpenAI provider with Gollm, you must provide your credentials and resource details via environment variables. Ensure you have an Azure OpenAI Service account and a deployed model (e.g., GPT-4) before proceeding.

export AZURE_OPENAI_API_KEY="your-api-key"
export AZURE_OPENAI_RESOURCE_NAME="your-resource-name"
export AZURE_OPENAI_DEPLOYMENT_NAME="your-deployment-name"
export AZURE_OPENAI_API_VERSION="2023-05-15"

To verify the OpenRouter provider, you can run integration tests that make actual API calls.

Note: These tests consume credits from your OpenRouter account.

Option 1: Via main.go

export OPENROUTER_API_KEY="your_api_key_here"
go run main.go -test
# OR
go run main.go -test -key="your_api_key_here"

Option 2: Via go test

export OPENROUTER_API_KEY="your_api_key_here"
go test -v ./providers -run TestOpenRouterIntegration