rank_bm25 Documentation

To install the newest version directly from GitHub, use the following command:

pip install git+ssh://git@github.com/dorianbrown/rank_bm25.git

You can install the package using pip. To ensure you get the latest version, you can also install it directly from the GitHub repository.

pip install rank_bm25

To use the library, you must first create an instance of a BM25 class (such as BM25Okapi). The class requires a tokenized_corpus, which must be a list of lists of strings (each inner list containing the tokens for a single document).

Important: This package does not perform any automatic text preprocessing. You are responsible for tasks such as lowercasing, stopword removal, and stemming. You must apply the exact same preprocessing to both your corpus and your queries to ensure consistent results.

from rank_bm25 import BM25Okapi

corpus = [
    "Hello there good man!",
    "It is quite windy in London",
    "How is the weather today?"
]

tokenized_corpus = [doc.split(" ") for doc in corpus]

bm25 = BM25Okapi(tokenized_corpus)

Once initialized, you can use get_scores() to retrieve relevance scores for every document in your corpus relative to a tokenized query. The query must be tokenized using the same method used for the corpus.

query = "windy London"
tokenized_query = query.split(" ")

doc_scores = bm25.get_scores(tokenized_query)
# Returns an array of scores, e.g., array([0.        , 0.93729472, 0.        ])