LangChain Community

Semantic caches (like OpenSearchSemanticCache and SingleStoreDBSemanticCache) differ from standard key-value caches by using vector similarity. Instead of looking for an exact string match, they use an embedding model to convert prompts into vectors and perform a similarity search.

Workflow:

1. Lookup: When a prompt is provided, the cache performs a similarity search (e.g., similarity_search or similarity_search_with_score) to find the most similar previously seen prompt. If the similarity score exceeds a defined score_threshold (or search_threshold), the cached generation is returned.
2. Update: If a cache miss occurs, the LLM generates a response. The cache then stores the prompt and the serialized Generation objects (using dumps) in the backend.
3. Clear: Caches can be cleared either globally (like MemcachedCache) or for a specific llm_string (like OpenSearchSemanticCache), which typically involves deleting the specific index or container associated with that LLM.

The OpenSearchSemanticCache uses an OpenSearch vector store backend to perform semantic lookups. It stores prompts as text and their corresponding LLM generations in metadata. When a new prompt is received, it performs a similarity search to find the closest matching prompt in the cache.

Key Parameters:

• opensearch_url (str): The URL to connect to your OpenSearch instance.
• embedding (Embeddings): An embedding provider used to encode prompts for semantic search.
• score_threshold (float, default: 0.2): The minimum similarity score required for a result to be returned from the cache.
• **kwargs: Additional connection arguments for the OpenSearch client.

import langchain
from langchain_community.cache import OpenSearchSemanticCache
from langchain_openai import OpenAIEmbeddings

langchain.llm_cache = OpenSearchSemanticCache(
    opensearch_url="http://localhost:9200",
    embedding=OpenAIEmbeddings()
)

The MemcachedCache uses a Memcached backend via the pymemcache library to cache LLM generations. It uses a hashed key derived from the prompt and the llm_string to store and retrieve values.

Requirements:

• You must have the pymemcache package installed: pip install -U pymemcache.

Initialization: Pass an instance of a pymemcache client (such as Client, PooledClient, HashClient, or RetryingClient) to the constructor.

Methods:

• clear(**kwargs): Clears the entire cache. Supports optional arguments:
  • delay (int): Seconds to wait before flushing (default: 0).
  • noreply (bool): If True, does not wait for the reply.

from langchain_community.cache import MemcachedCache
from pymemcache.client.base import Client
from langchain_openai import OpenAI
from langchain.globals import set_llm_cache

llm = OpenAI(model="gpt-3.5-turbo-instruct", n=2, best_of=2)
set_llm_cache(MemcachedCache(Client('localhost')))

# First call: Cache miss
llm.invoke("Which city is the most crowded city in the USA?")

# Second call: Cache hit
llm.invoke("Which city is the most crowded city in the USA?")

The SingleStoreDBSemanticCache uses SingleStore DB as a backend.

⚠️ Deprecation Warning: This class is pending deprecation. Users should migrate to SingleStoreSemanticCache in the langchain_singlestore package.

Key Configuration Options:

• embedding (Embeddings): A text embedding model.
• cache_table_prefix (str, default: "cache_"): Prefix for the cache table name.
• search_threshold (float, default: 0.2): Minimum similarity score for a match.
• distance_strategy: Determines how distance is calculated. Options are DOT_PRODUCT (default) or EUCLIDEAN_DISTANCE.
• use_vector_index (bool, default: False): Toggles vector index usage (requires SingleStoreDB 8.5+).
• vector_size (int): Required if use_vector_index is True.
• host, user, password, port, database: Standard database connection parameters.

import langchain
from langchain_community.cache import SingleStoreDBSemanticCache
from langchain_openai import OpenAIEmbeddings

# Basic Usage
langchain.llm_cache = SingleStoreDBSemanticCache(
    embedding=OpenAIEmbeddings(),
    host="https://user:password@127.0.0.1:3306/database"
)

# Advanced Usage
langchain.llm_cache = SingleStoreDBSemanticCache(
    embedding=OpenAIEmbeddings(),
    use_vector_index=True,
    host="127.0.0.1",
    port=3306,
    user="user",
    password="password",
    database="db",
    table_name="my_custom_table",
    pool_size=10,
    timeout=60,
)