HyperDB Documentation

If you intend to use Hugging Face models for local embedding within HyperDB, you must also install the sentence-transformers package.

pip install sentence-transformers

HyperDB allows you to instantiate a vector database from a list of documents, save the state to a file, reload it later, and perform similarity searches using text queries. When instantiating, you specify a key which points to the field in your document dictionary containing the text to be embedded.

import json
from hyperdb import HyperDB

# 1. Prepare documents (e.g., from a JSONL file)
documents = []
with open("demo/pokemon.jsonl", "r") as f:
    for line in f:
        documents.append(json.loads(line))

# 2. Instantiate HyperDB with documents and the text key
db = HyperDB(documents, key="info.description")

# 3. Save the instance to a file
db.save("demo/pokemon_hyperdb.pickle.gz")

# 4. Load the instance from the saved file
db.load("demo/pokemon_hyperdb.pickle.gz")

# 5. Query the database with a text input
results = db.query("Likes to sleep.", top_k=5)

Install the HyperDB package from PyPI using pip.

pip install hyperdb-python

Use the query method to perform a similarity search. Pass a string representing your search query and the top_k parameter to specify the number of results to return.

results = db.query("Likes to sleep.", top_k=5)

HyperDB provides methods to persist the database state to disk and reload it later, which is useful for avoiding re-indexing large datasets.

• save(path): Saves the current HyperDB instance to the specified file path.
• load(path): Loads a previously saved HyperDB instance from the specified file path.

db.save("demo/pokemon_hyperdb.pickle.gz")
db.load("demo/pokemon_hyperdb.pickle.gz")