GPTCache Documentation

GPTCache provides several key advantages for LLM-based applications:

• Decreased expenses: Reduces costs by minimizing the number of requests and tokens sent to LLM services through query caching.
• Enhanced performance: Improves response times and query throughput by fetching similar queries directly from the cache instead of waiting for real-time LLM generation.
• Adaptable development and testing: Provides an interface that mirrors LLM APIs and supports both LLM-generated and mocked data, allowing for testing without active LLM connections.
• Improved scalability and availability: Helps bypass LLM service rate limits by serving cached responses, ensuring consistent performance as user volume grows.

GPTCache is a semantic caching library designed to optimize LLM usage through several key benefits:

• Decreased expenses: Reduces costs by caching query results, minimizing the number of requests and tokens sent to LLM services.
• Enhanced performance: Improves response times and throughput by fetching similar queries directly from the cache instead of calling the LLM.
• Adaptable development and testing: Provides an interface that mirrors LLM APIs, allowing developers to use mocked data and avoid live LLM connections during testing.
• Improved scalability and availability: Helps mitigate LLM rate limits by serving cached responses, allowing applications to scale more effectively.

Unlike traditional caches that rely on exact string matches, GPTCache uses semantic caching. It converts queries into embeddings using an Embedding Generator and performs similarity searches using a Vector Store. This allows the system to identify and retrieve responses for queries that are semantically similar, even if they are not identical.

To evaluate the performance of your semantic cache, monitor these three metrics:

• Hit Ratio: The proportion of successful content requests fulfilled by the cache relative to total requests.
• Latency: The time taken to process a query and retrieve data from the cache.
• Recall: The proportion of queries served by the cache out of the total number of queries that should have been served by the cache.

GPTCache is modular and allows you to custom assemble the following components:

• Adapter: Adapts different LLM model requests to the GPTCache protocol.
• Pre-processor: Extracts and preprocesses key information from the request.
• Context Buffer: Maintains session context.
• Encoder: Embeds text into dense vectors for similarity search.
• Cache manager: Handles searching, saving, or evicting data.
• Ranker: Evaluates similarity by judging the quality of cached answers.
• Post-processor: Determines which cached answers to return to the user and generates the response.

The OpenAI adapter allows you to wrap OpenAI's ChatCompletion API with GPTCache. You can use standard calls or stream responses.

Note: You must call cache.set_openai_key() after initializing the cache to ensure the adapter can access your credentials.

from gptcache.adapter import openai

cache.init(data_manager=get_data_manager())
cache.set_openai_key()

response = openai.ChatCompletion.create(
    model='gpt-3.5-turbo',
    messages=[{'role': 'user', 'content': 'What is 1+1?'}],
    stream=True
)

# For streaming, iterate through the response
for chunk in response:
    print(chunk)

GPTCache provides specialized integration guides (Bootcamps) for various LLM frameworks and providers. You can find tutorials for:

• LangChain: QA Generation, Question Answering, SQL Chain, and BabyAGI.
• Llama_index: WebPage QA.
• OpenAI: Chat completion, Language Translation, SQL Translate, Twitter Classifier, and Multimodal (Image Generation, Speech to Text).
• Replicate: Visual Question Answering.
• Temperature Parameter: OpenAI Chat and OpenAI Image Creation.

To use other LLMs supported by Langchain, use the LangChainLLMs adapter. This requires passing the LLM instance and a cache_obj to the cached_llm call.

from gptcache.adapter import LangChainLLMs

llm = OpenAI()
llm_cache = Cache()
llm_cache.init(pre_embedding_func=get_prompt, post_process_messages_func=postnop)

cached_llm = LangChainLLMs(llm)
answer = cached_llm(question, cache_obj=llm_cache)

To add a new data manager:

1. Implement the DataManager interface.
2. Register the new store in the get_data_manager method in gptcache/manager/data_manager.py.
3. Provide a usage example in the examples/data_manager directory.

When working with Chinese queries, use a specialized embedding model (e.g., from HuggingFace) via the Huggingface class in gptcache.embedding to ensure proper semantic representation.

from gptcache.adapter import openai
from gptcache.adapter.api import init_similar_cache
from gptcache.embedding import Huggingface
from gptcache.processor.pre import last_content

huggingface = Huggingface(model="uer/albert-base-chinese-cluecorpussmall")
init_similar_cache(pre_func=last_content, embedding=huggingface)

# Subsequent openai.ChatCompletion.create calls will use this embedding

This demo uses GPTCache to reduce costs when using OpenAI's DALL-E API by caching generated images. When a prompt is reused, the app retrieves the image from the cache instead of making a new API call.

Prerequisites:

• Python 3.6 or later
• An OpenAI API key

Setup and Execution:

1. Clone the repository.
2. Install dependencies: pip install -r requirements.txt
3. Launch the Streamlit app: streamlit run imagen.py
4. Access the app at http://localhost:8501.
5. Enter your OpenAI key and a prompt. If the prompt has been processed before, a "cache" message will appear below the image, indicating a cache hit.

GPTCache can be initialized in three ways depending on your requirements for matching (exact vs. similar) and configuration source:

1. Exact Match (Cache.init): Uses a simple map cache for exact key matching.
2. Similar Match (init_similar_cache): Uses a combination of ONNX, SQLite, and FAISS for semantic/fuzzy matching. This is the recommended method for most semantic cache use cases.
3. Config-based Similar Match (init_similar_cache_from_config): Initializes a similar cache using a YAML configuration file.

Multi-level Caching: You can implement a multi-level cache (e.g., L1 and L2) by setting the next_cache parameter during initialization. If L1 misses, it checks L2. If L2 hits, the result is stored in L1. If both miss, the LLM is called, and the result is stored in both L2 and L1.