bm25s Documentation

bm25s separates the text used for indexing from the values returned during retrieval. You pass strings to bm25s.tokenize(), and then pass the resulting tokens to retriever.index().

To control what is returned during a search, use the corpus argument in BM25(...), retrieve(...), or save(...). This argument can be a list of plain strings or a list of dictionaries (metadata).

Using Metadata (Dictionaries)

If your corpus contains dictionaries, retrieval is position-based: corpus[i] is returned for document ID i.

text_corpus = [
    "a cat is a feline and likes to purr",
    "a dog is the human's best friend and loves to play",
]

metadata_corpus = [
    {"id": "cat-doc", "title": "About Cat", "text": text_corpus[0]},
    {"id": "dog-doc", "title": "About Dog", "text": text_corpus[1]},
]

# Index the text field
corpus_tokens = bm25s.tokenize([doc["text"] for doc in metadata_corpus])

# Pass the metadata as the corpus so retrieval returns the full dictionary
retriever = bm25s.BM25(corpus=metadata_corpus)
retriever.index(corpus_tokens)

Note on Serialization: When saving, corpus entries must be strings, dictionaries, lists, or tuples that can be serialized to JSON. String entries are written to corpus.jsonl as {"id": i, "text": doc}.

text_corpus = [
    "a cat is a feline and likes to purr",
    "a dog is the human's best friend and loves to play",
]

metadata_corpus = [
    {"id": "cat-doc", "title": "About Cat", "text": text_corpus[0]},
    {"id": "dog-doc", "title": "About Dog", "text": text_corpus[1]},
]

# Pick the text field(s) you want to index.
corpus_tokens = bm25s.tokenize([doc["text"] for doc in metadata_corpus])

# Retrieved documents will be the original dictionary entries.
retriever = bm25s.BM25(corpus=metadata_corpus)
retriever.index(corpus_tokens)

This example demonstrates the standard workflow: creating a corpus, tokenizing it (optionally with a stemmer), indexing it with a BM25 model, and performing queries.

import bm25s
import Stemmer  # optional: for stemming

# 1. Create your corpus
corpus = [
    "a cat is a feline and likes to purr",
    "a dog is the human's best friend and loves to play",
    "a bird is a beautiful animal that can fly",
    "a fish is a creature that lives in water and swims",
]

# 2. Optional: create a stemmer
stemmer = Stemmer.Stemmer("english")

# 3. Tokenize the corpus (keeping only IDs is faster and saves memory)
corpus_tokens = bm25s.tokenize(corpus, stopwords="en", stemmer=stemmer)

# 4. Create the BM25 model and index the corpus
retriever = bm25s.BM25()
retriever.index(corpus_tokens)

# 5. Query the corpus
query = "does the fish purr like a cat?"
query_tokens = bm25s.tokenize(query, stemmer=stemmer)

# 6. Get top-k results as a tuple of (doc ids, scores).
# Both are arrays of shape (n_queries, k).
# To return docs instead of IDs, set the `corpus=corpus` parameter in retriever.retrieve.
results, scores = retriever.retrieve(query_tokens, k=2)

for i in range(results.shape[1]):
    doc, score = results[0, i], scores[0, i]
    print(f"Rank {i+1} (score: {score:.2f}): {doc}")

# 7. Save and Load
retriever.save("animal_index_bm25")
# To save with the corpus: retriever.save("animal_index_bm25", corpus=corpus)

# Reloading
reloaded_retriever = bm25s.BM25.load("animal_index_bm25", load_corpus=True)

import bm25s
import Stemmer  # optional: for stemming

# Create your corpus here
corpus = [
    "a cat is a feline and likes to purr",
    "a dog is the human's best friend and loves to play",
    "a bird is a beautiful animal that can fly",
    "a fish is a creature that lives in water and swims",
]

# optional: create a stemmer
stemmer = Stemmer.Stemmer("english")

# Tokenize the corpus and only keep the ids (faster and saves memory)
corpus_tokens = bm25s.tokenize(corpus, stopwords="en", stemmer=stemmer)

# Create the BM25 model and index the corpus
retriever = bm25s.BM25()
retriever.index(corpus_tokens)

# Query the corpus
query = "does the fish purr like a cat?"
query_tokens = bm25s.tokenize(query, stemmer=stemmer)

# Get top-k results as a tuple of (doc ids, scores). Both are arrays of shape (n_queries, k).
# To return docs instead of IDs, set the `corpus=corpus` parameter.
results, scores = retriever.retrieve(query_tokens, k=2)

for i in range(results.shape[1]):
    doc, score = results[0, i], scores[0, i]
    print(f"Rank {i+1} (score: {score:.2f}): {doc}")

# You can save the arrays to a directory...
retriever.save("animal_index_bm25")

# You can save the corpus along with the model
retriever.save("animal_index_bm25", corpus=corpus)

# ...and load them when you need them
import bm25s
reloaded_retriever = bm25s.BM25.load("animal_index_bm25", load_corpus=True)
# set load_corpus=False if you don't need the corpus"},{

Use bm25s.hf.BM25HF to save and load BM25 indices directly from the Hugging Face Hub. This requires a HUGGING_FACE_HUB_TOKEN environment variable. You can save the index along with the corpus to the hub and reload it later, including options for memory-mapping and loading the corpus.

import os
import bm25s
from bm25s.hf import BM25HF

# Create a BM25 index
retriever = BM25HF()
# Create your corpus here
corpus = [
    "a cat is a feline and likes to purr",
    "a dog is the human's best friend and loves to play",
    "a bird is a beautiful animal that can fly",
    "a fish is a creature that lives in water and swims",
]
corpus_tokens = bm25s.tokenize(corpus)
retriever.index(corpus_tokens)

# Set your username and token
user = "your-username"
token = os.environ["HF_TOKEN"]
retriever.save_to_hub(f"{user}/bm25s-animals", token=token, corpus=corpus)

# --- Loading from Hub ---

# Load a BM25 index from the Hugging Face model hub
retriever = BM25HF.load_from_hub(f"{user}/bm25s-animals")

# you can specify revision and load_corpus=True if needed
retriever = BM25HF.load_from_hub(
    f"{user}/bm25s-animals", revision="main", load_corpus=True
)

# if you want a low-memory usage, you can load as memory map with `mmap=True`
retriever = BM25HF.load_from_hub(
    f"{user}/bm25s-animals", load_corpus=True, mmap=True
)

You can build a search engine over local files (.csv, .json, .jsonl, .txt) or lists of text using the BM25 module.

1. Use BM25.load(path, document_column=...) to load documents. For CSV or JSONL, specify the column or key containing the text.
2. Use BM25.index(corpus) to build the search index (handles tokenization and stemming).
3. Use retriever.search(queries, k=...) to retrieve the top k results for a list of queries.

import BM25

# 1. Load your documents (supports .csv, .json, .jsonl, .txt)
# For csv/jsonl, you can specify which column/key holds the text
corpus = BM25.load("documents.csv", document_column="text")

# 2. Build the search index
retriever = BM25.index(corpus)

# 3. Search!
queries = ["how to learn python", "best search algorithms"]
results = retriever.search(queries, k=5) # Get top 5 results

# Print the top results for the first query
for result in results[0]:
    print(f"Score: {result['score']:.2f} | Document: {result['document']}")