repository·master·Indexed 43 days ago
https://github.com/upstash/semantic-cacheA semantic cache built on top of Upstash Vector DB that allows developers to reduce LLM costs and API latency by retrieving cached responses for queries that are semantically similar to previous inputs. It features a configurable minProximity parameter for similarity thresholds, namespace partitioning, and support for synonyms and multilingual queries. Version 1.0.3.
The minProximity parameter (range 0 to 1) defines the minimum relevance score required for a cache hit.
A higher value ensures that the user input must be very similar to the cached content to trigger a hit.
Semantic Cache requires an Upstash Vector database. When creating your database in the Upstash Console, ensure you select a pre-made embedding model. You will need the url and token to connect.
Add these to your .env file:
UPSTASH_VECTOR_REST_URL=https://example.upstash.io
UPSTASH_VECTOR_REST_TOKEN=your_secret_token_hereTo use Semantic Cache, you must install both the @upstash/semantic-cache and @upstash/vector packages via npm.
npm install @upstash/semantic-cache @upstash/vectorTo use the cache, instantiate SemanticCache by passing an instance of Index from @upstash/vector. You can then use .set(key, value) to store data and .get(query) to retrieve values based on semantic similarity rather than exact string matching.
import { SemanticCache } from "@upstash/semantic-cache";
import { Index } from "@upstash/vector";
// 👇 your vector database
const index = new Index();
// 👇 your semantic cache
const semanticCache = new SemanticCache({ index, minProximity: 0.95 });
async function runDemo() {
await semanticCache.set("Capital of Turkey", "Ankara");
await delay(1000);
// 👇 outputs: "Ankara"
const result = await semanticCache.get("What is Turkey's capital?");
console.log(result);
}
function delay(ms: number) {
return new Promise((resolve) => setTimeout(resolve, ms));
}
runDemo();Semantic Cache supports various retrieval scenarios including synonyms, multilingual queries, and complex phrasing.
Note: When running manual tests, a delay (e.g., 1 second) is often necessary after a .set() call to allow the vector index to update before performing a .get().
// Handling Synonyms
await semanticCache.set("largest city in USA by population", "New York");
await delay(1000);
const result = await semanticCache.get("which is the most populated city in the USA?"); // outputs "New York"
// Multilingual Queries (requires multilingual embedding model)
await semanticCache.set("German Chancellor", "Olaf Scholz");
await delay(1000);
const resultLang = await semanticCache.get("Wer ist der Bundeskanzler von Deutschland?"); // outputs "Olaf Scholz"
// Complex Queries
await semanticCache.set("year in which the Berlin wall fell", "1989");
await delay(1000);
const resultComplex = await semanticCache.get("what's the year the Berlin wall destroyed?"); // outputs "1989"You can use the namespace option in the SemanticCache constructor to separate your data into logical partitions. This prevents queries in one namespace from retrieving results from another.
import { SemanticCache } from "@upstash/semantic-cache";
import { Index } from "@upstash/vector";
const index = new Index();
const semanticCache = new SemanticCache({ index, minProximity: 0.95, namespace: "user1" });
await semanticCache.set("Capital of Turkey", "Ankara");Use these methods to remove entries from the cache.
delete(key: string): Promise<number>: Deletes a specific entry by its ID. Returns the number of deleted items.bulkDelete(keys: string[]): Promise<number>: Deletes multiple entries by their IDs. Returns the number of deleted items.// Delete one key
const deletedCount = await cache.delete("my-key");
// Delete multiple keys
const bulkDeletedCount = await cache.bulkDelete(["key1", "key2"]);To use the SemanticCache, instantiate the SemanticCache class by passing a SemanticCacheConfig object. This configuration requires an Upstash serverless vector client (Index) and allows you to tune the similarity threshold and organize data using namespaces.
index (Required): An instance of an Upstash serverless vector client (Index).minProximity (Optional): A number between 0.0 and 1.0. Defaults to 0.9.1.0 behaves like a hash map (only exact lexical matches).0.0 behaves like a full text search (returns the closest match regardless of similarity).namespace (Optional): A string used to partition the cache. If provided, the cache will operate within that specific namespace of the provided index.import { SemanticCache } from '@upstash/semantic-cache';
// Assuming 'index' is an initialized Upstash Index
const cache = new SemanticCache({
index: index,
minProximity: 0.9,
namespace: 'my-cache-namespace'
});The get method retrieves cached values based on semantic similarity to the provided key(s). It returns the metadata value associated with the closest match, provided the similarity score exceeds the configured minProximity threshold.
get(key: string): Promise<string | undefined>get(keys: string[]): Promise<(string | undefined)[]>// Get a single value
const value = await cache.get("What is the capital of France?");
// Get multiple values
const values = await cache.get(["How are you?", "Tell me a joke"]);The set method stores a value in the cache associated with a key. The key is used as the vector data for semantic retrieval, and the value is stored in the metadata.
set(key: string, value: string): Promise<void>set(keys: string[], values: string[]): Promise<void>// Set a single value
await cache.set("What is the capital of France?", "Paris");
// Set multiple values
await cache.set(
["Who wrote Hamlet?", "What is 2+2?"],
["Shakespeare", "4"]
);The flush method clears the cache. If a namespace was provided during initialization, it will reset only that namespace. Otherwise, it resets the entire index.
await cache.flush();The @upstash/semantic-cache package exports the SemanticCache class, which serves as the primary entry point for interacting with the semantic cache service. You can import it directly from the package root.
import { SemanticCache } from "@upstash/semantic-cache";