repository·main·Indexed 21 days ago
https://github.com/prithivirajdamodaran/flashrankAn ultra-lite and super-fast Python library for re-ranking search results, designed to improve precision in retrieval pipelines before feeding results into LLMs. FlashRank runs efficiently on CPUs without requiring Torch or Transformers and supports various models including ms-marco-TinyBERT-L-2-v2, ms-marco-MiniLM-L-12-v2, and rank-T5-flan. It can be integrated into lexical, semantic, or hybrid search architectures and is compatible with serverless environments like AWS Lambda.
FlashRank is designed to be used as a reranking stage in various search architectures:
To maximize performance, set the max_length parameter in the Ranker constructor to accommodate your longest passage plus the query.
Formula: estimated_tokens = longest_passage_tokens + query_tokens.
If your estimate is 116 tokens, setting max_length=128 is efficient. Providing a value that is unnecessarily large (e.g., setting max_length=512 for small passages) will negatively affect response time.
Depending on your use case, install the appropriate version of FlashRank via pip:
[listwise] extra.FlashRank is designed to run on CPU and does not require Torch or Transformers.
# For lightweight pairwise rerankers [default]
pip install flashrank
# If you need LLM based listwise rerankers
pip install flashrank[listwise]If you encounter issues with model access, ensure you are using the latest version of the library, as models have been migrated to Hugging Face. Upgrade using pip:
pip install -U flashrankIn serverless environments like AWS Lambda where the filesystem is read-only, you should manage model loading and caching by specifying a custom directory. You can create a custom directory in your Dockerfile and use the cache_dir parameter during the Ranker initialization to load models and provide a cache between warm calls.
ranker = Ranker(model_name="ms-marco-MiniLM-L-12-v2", cache_dir="/opt")To rerank results, create a RerankRequest object containing your query and a list of passages. Each passage should be a dictionary containing at least an id and text. Metadata (meta) is optional and can be used to carry database IDs or other context through the pipeline.
Then, pass this request to the ranker.rerank() method.
from flashrank import Ranker, RerankRequest
ranker = Ranker(max_length=128)
query = "How to speedup LLMs?"
passages = [
{
"id": 1,
"text": "Introduce *lookahead decoding*: - a parallel decoding algo to accelerate LLM inference...",
"meta": {"additional": "info1"}
},
{
"id": 2,
"text": "LLM inference efficiency will be one of the most crucial topics...",
"meta": {"additional": "info2"}
}
]
rerankrequest = RerankRequest(query=query, passages=passages)
results = ranker.rerank(rerankrequest)
print(results)The Ranker class is the main entry point. You can choose from several models based on your requirements for speed, precision, or language support. You can also specify a cache_dir to manage where models are stored and a max_length to optimize performance.
Model Options:
ms-marco-TinyBERT-L-2-v2 (Default): ~4MB, blazing fast, competitive precision.ms-marco-MiniLM-L-12-v2: ~34MB, best performance for cross-encoders.rank-T5-flan: ~110MB, best non-cross-encoder reranker (zeroshot performance).ms-marco-MultiBERT-L-12: ~150MB, supports 100+ languages (not recommended for English).rank_zephyr_7b_v1_full: ~4GB, 4-bit-quantised GGUF, competitive performance with large context window.from flashrank import Ranker
# Nano (~4MB), blazing fast
ranker = Ranker(max_length=128)
# Small (~34MB), best cross-encoder performance
ranker = Ranker(model_name="ms-marco-MiniLM-L-12-v2", cache_dir="/opt")
# Medium (~110MB), best zeroshot performance
ranker = Ranker(model_name="rank-T5-flan", cache_dir="/opt")
# Medium (~150MB), multi-lingual (100+ languages)
ranker = Ranker(model_name="ms-marco-MultiBERT-L-12", cache_dir="/opt")
# Large LLM-based (4GB)
ranker = Ranker(model_name="rank_zephyr_7b_v1_full", max_length=1024)FlashRank supports several models with varying trade-offs between size and performance:
| Model Name | Description | Size | Notes |
|---|---|---|---|
ms-marco-TinyBERT-L-2-v2 | Default model | ~4MB | |
ms-marco-MiniLM-L-12-v2 | Best Cross-encoder reranker | ~34MB | |
rank-T5-flan | Best non cross-encoder reranker | ~110MB | |
ms-marco-MultiBERT-L-12 | Multi-lingual, supports 100+ languages | ~150MB | |
ce-esci-MiniLM-L12-v2 | Fine-tuned on Amazon ESCI dataset | - | |
rank_zephyr_7b_v1_full | 4-bit-quantised GGUF | ~4GB | |
miniReranker_arabic_v1 | Only dedicated Arabic Reranker | - |
ms-marco-TinyBERT-L-2-v2
ms-marco-MiniLM-L-12-v2
rank-T5-flan
ms-marco-MultiBERT-L-12
ce-esci-MiniLM-L12-v2
rank_zephyr_7b_v1_full
miniReranker_arabic_v1The default reranker produces a list of dictionaries representing the reranked documents. Each document object contains the following fields:
id: A unique identifier (can be database IDs or simple numeric indices).text: The content of the document.meta: An optional dictionary for storing additional metadata.score: The relevance score assigned by the reranker.Example output structure:
[
{
"id": 4,
"text": "Document content here...",
"meta": {
"additional": "info4"
},
"score": 0.016847236
}
][
{
"id":4,
"text":"Ever want to make your LLM inference go brrrrr but got stuck at implementing speculative decoding and finding the suitable draft model? No more pain! Thrilled to unveil Medusa, a simple framework that removes the annoying draft model while getting 2x speedup.",
"meta":{
"additional":"info4"
},
"score":0.016847236
},
{
"id":5,
"text":"vLLM is a fast and easy-to-use library for LLM inference and serving. vLLM is fast with: State-of-the-art serving throughput Efficient management of attention key and value memory with PagedAttention Continuous batching of incoming requests Optimized CUDA kernels",
"meta":{
"additional":"info5"
},
"score":0.011563735
}
]