How prompt caching works in AnthropicClient
mainThe AnthropicClient supports prompt caching for system messages, user messages (including images), assistant messages, tool_results, documents, and tools. There are two primary mechanisms for caching:
AutomaticToolsAndSystem: Automatically caches the System prompt and Tools when present. You can still manually add additional set-points in Messages if desired.FineGrained: Provides complete manual control. You declare cache control settings at the message and tool level usingCacheControl. Note that Anthropic allows a maximum of 4 cache points in total.
Important Considerations:
- Caches have a 5-minute expiry enforced by Anthropic.
- You can monitor cache performance by checking
Usage.CacheCreationInputTokensandUsage.CacheReadInputTokensin the response. - Cache misses can occur due to various model limitations.
// Example of Automatic caching
var parameters = new MessageParameters()
{
// ... other params
PromptCaching = PromptCacheType.AutomaticToolsAndSystem
};
// Example of Fine-Grained manual caching
var systemMessages = new List<SystemMessage>()
{
new SystemMessage("You are an expert."),
new SystemMessage(content, new CacheControl() { Type = CacheControlType.ephemeral })
};
var parameters = new MessageParameters()
{
// ... other params
PromptCaching = PromptCacheType.FineGrained
};