rankify

repository·main·Indexed 20 days ago

https://github.com/datascienceuibk/rankify

A modular Python toolkit for retrieval, re-ranking, and Retrieval-Augmented Generation (RAG). Version 0.1.4 supports a wide array of state-of-the-art models and techniques, including a one-line pipeline API, REST API deployment, and integrations with LangChain and LlamaIndex. It features a comprehensive set of supported reranking methods, tools for indexing corpora via CLI, and a Gradio-based web playground for model comparison.

Tokens: 70.1K
Snippets: 207
Records: 264
Agent score: 70%

What's inside rankify

Overview of Rankify features
main
Rankify is a modular Python toolkit for unified retrieval, re-ranking, and retrieval-augmented generation (RAG) research.
Key Capabilities:
- Retrieval: Supports 7 techniques including BM25, DPR, ANCE, BPR, ColBERT, BGE, and Contriever.
- Re-ranking: Implements 24 primary models with 41 sub-methods (pointwise, pairwise, and listwise).
- RAG: Integrates with generative models like GPT, LLAMA, and T5 using strategies like zero-shot, Fusion-in-Decoder (FiD), and in-context learning.
- Datasets: Includes 40 benchmark datasets with pre-retrieved documents.
- Evaluation: Automated performance evaluation with retrieval, ranking, and RAG metrics (e.g., Top-K, EM, Recall).
https://github.com/datascienceuibk/rankify/blob/main/README-PyPI.md
Overview of Rankify API Modules
main
Rankify v0.1.4 provides a comprehensive toolkit for building retrieval and RAG pipelines. The API is organized into several functional modules:
- Dataset Module: For managing data used in retrieval and reranking.
- Metrics Module: For evaluating performance.
- Retrievers: A collection of unified interfaces for various retrieval strategies (Sparse, Dense, Web, etc.).
- Indexing Module: For building custom search indices.
- Rerankers: A wide array of reranking strategies categorized by their approach (Pointwise, Pairwise, Listwise, API-based, LLM-based, and Embedding-based).
- Generator Module: Provides unified interfaces for RAG methods and LLM endpoints (OpenAI, vLLM, etc.).
- Tools Module: Includes utilities like WebSearchTool for agentic workflows.
https://github.com/datascienceuibk/rankify/blob/main/docs/docs/api/index.md
Understand the pre-retrieved dataset format
main
Rankify provides pre-retrieved datasets (containing 1,000 documents per dataset) available on Hugging Face. Each dataset follows a specific JSON structure where each entry contains a question, a list of answers, and a list of contexts (ctxs). Each context includes a paragraph ID, a retriever score, and a boolean indicating if the paragraph contains the correct answer.
JSON Schema:
```
[
    {
        "question": "...",
        "answers": ["...", "...", ...],
        "ctxs": [
            {
                "id": "...",         // Paragraph ID in the database TSV
                "score": "...",      // Retriever score
                "has_answer": true|false  // Whether the paragraph contains the correct answer
            }
        ]
    }
]
```
https://github.com/datascienceuibk/rankify/blob/main/README_zh.md
Understand RAG methods in Rankify
main
Rankify supports 7 distinct RAG (Retrieval-Augmented Generation) methods, allowing you to choose a strategy based on your specific use case:
- Zero-Shot: Direct generation using the provided context.
- Basic RAG: Simple prompting using context and the question.
- Chain-of-Thought: Encourages step-by-step reasoning.
- Self-Consistency: Uses multiple reasoning paths to improve reliability.
- ReAct: Implements Reasoning + Action cycles.
- FiD (Fusion-in-Decoder): Uses the Fusion-in-Decoder architecture.
- In-Context RALM: Uses in-context retrieval-augmented language modeling.
Recommended Methods by Use Case
Use Case Recommended Method
Simple QA Basic RAG, Zero-Shot
Complex reasoning Chain-of-Thought
High accuracy Self-Consistency
Multi-step tasks ReAct
Encoder-decoder FiD
https://github.com/datascienceuibk/rankify/blob/main/docs/docs/tutorials/rag/introduction.md
Supported retrieval methods for indexing
main
Rankify's indexing module supports building indices for the following retrieval architectures:
- BM25: Sparse retrieval based on Lucene.
- DPR: Dense Passage Retrieval.
- ANCE: Approximate Nearest Neighbor Negative Contrastive Estimation.
- BGE: BAAI General Embedding.
- ColBERT: Contextualized Late Interaction over BERT.
- Contriever: Contrastive Retriever.
https://github.com/datascienceuibk/rankify/blob/main/docs/docs/api/indexing.md

Use Case	Recommended Method
Simple QA	Basic RAG, Zero-Shot
Complex reasoning	Chain-of-Thought
High accuracy	Self-Consistency
Multi-step tasks	ReAct
Encoder-decoder	FiD

Understand the different Pipeline Types

main

Rankify supports three main pipeline architectures depending on your goal:

Type	Description	Components
`"search"`	Document retrieval only	Retriever
`"rerank"`	Retrieve + rerank	Retriever + Reranker
`"rag"`	Full RAG pipeline	Retriever + Reranker + Generator

Search Pipeline

Used for finding relevant documents.

from rankify import pipeline

search = pipeline("search", retriever="bm25", n_docs=100)
results = search("Find papers about transformers")

for ctx in results.documents[0].contexts[:5]:
    print(f"- {ctx.text[:100]}...")

Rerank Pipeline

Used to improve retrieval quality by re-ordering documents.

from rankify import pipeline

rerank = pipeline("rerank", retriever="bge", reranker="flashrank")
results = rerank("Best Python frameworks for ML")

for ctx in results.documents[0].reorder_contexts[:5]:
    print(f"Score: {ctx.score:.3f} - {ctx.text[:50]}...")

RAG Pipeline

Full Retrieval-Augmented Generation workflow.

from rankify import pipeline

rag = pipeline(
    "rag",
    retriever="bge",
    reranker="monot5",
    generator="chain-of-thought-rag",
)

result = rag("Explain how transformers work")
print(result.answers[0])

How model caching works in Rankify
main
Rankify implements automatic model caching. When you initialize a Reranking object for the first time with a specific model_name, the model is downloaded and stored in the cache directory. Any subsequent initializations using the same method and model_name will automatically use the cached version instead of re-downloading.
```
# First call downloads and caches the model
reranker = Reranking(method="monot5", model_name="monot5-base-msmarco")

# Subsequent calls use cached version
reranker2 = Reranking(method="monot5", model_name="monot5-base-msmarco")
```
https://github.com/datascienceuibk/rankify/blob/main/docs/docs/tutorials/advanced/saving_models.md
Configure Generator backends (HuggingFace, OpenAI, vLLM, LiteLLM)
main
The Generator class supports multiple backends for LLM inference. Choose the backend based on your requirements for speed, cost, and hardware:
- huggingface: Uses local HuggingFace models. Requires a GPU. Speed is medium.
- openai: Uses OpenAI's API. Requires OPENAI_API_KEY environment variable. Fast and paid.
- vllm: Uses vLLM for high-throughput inference. Requires pip install "rankify[reranking]". Best for batch generation. Requires a GPU.
- litellm: Uses LiteLLM to support 100+ providers (e.g., Claude). Fast and cost varies.
To use OpenAI, ensure you set the environment variable:
```
import os
os.environ["OPENAI_API_KEY"] = "your-api-key"
```
```
from rankify.generator.generator import Generator

# Example: OpenAI backend
generator = Generator(method="zero-shot", model_name="gpt-4o-mini", backend="openai")

# Example: vLLM backend (requires pip install "rankify[reranking]")
generator = Generator(method="zero-shot", model_name="meta-llama/Llama-3.1-8B-Instruct", backend="vllm")

# Example: LiteLLM backend
generator = Generator(method="zero-shot", model_name="claude-3-5-sonnet-20241022", backend="litellm")
```
https://github.com/datascienceuibk/rankify/blob/main/docs/docs/tutorials/rag/zero_shot.md
Re-ranking methods in Rankify
main
Rankify supports 23 re-ranking methods categorized into four paradigms:
Pointwise Rerankers
Scores each query-document pair independently.
- MonoBERT: BERT cross-encoder
- MonoT5: T5 sequence-to-sequence
- UPR: Unsupervised passage reranker
Pairwise Rerankers
Compares document pairs.
- RankGPT: LLM-based pairwise ranking
- InRanker: Instruction-based reranking
- EchoRank: Echo-based pairwise comparison
Listwise Rerankers
Considers the entire document list at once.
- RankT5: T5-based listwise ranking
- ListT5: Listwise T5 model
- LiT5: Lightweight T5 reranker
API-Based Rerankers
Uses external API services.
- Cohere: Cohere Rerank API
- Jina: Jina Reranker API
- Voyage: Voyage Rerank API
- MixedBread: MixedBread.ai API
https://github.com/datascienceuibk/rankify/blob/main/docs/docs/tutorials/reranking/introduction.md
Use the unified Retriever interface
main
Rankify provides a Retriever base interface that allows you to swap between different retrieval methods seamlessly. Supported retrievers include:
- Sparse Retrieval: BM25 Retriever.
- Dense Retrieval: Dense Retriever (DPR), ANCE Retriever, BGE Retriever, Contriever Retriever.
- Late Interaction: ColBERT Retriever.
- Web/Real-time: Online Retriever.
- Advanced Strategies: HyDE Retriever (Hypothetical Document Embeddings).
https://github.com/datascienceuibk/rankify/blob/main/docs/docs/api/index.md
Compare FiD vs Standard RAG
main
FiD Architecture
- Architecture: T5-based encoder-decoder.
- Passage handling: Encodes passages independently, then performs joint decoding.
- Context limit: Flexible due to independent encoding.
- Training: Task-specific fine-tuning.
Standard RAG
- Architecture: Decoder-only LLM.
- Passage handling: Concatenated in the prompt.
- Context limit: Limited by the model's context window.
- Training: Pre-trained, prompt-based.
https://github.com/datascienceuibk/rankify/blob/main/docs/docs/tutorials/rag/fid.md
Use the unified Reranking interface
main
The reranking module provides a Reranking interface and a Base reranker class. Rerankers are categorized by their scoring logic:
- Pointwise: Scores a single passage (e.g., MonoBERT, MonoT5, UPR).
- Pairwise: Compares pairs of passages (e.g., RankGPT, InRanker, EchoRank).
- Listwise: Scores an entire list of passages (e.g., RankT5, ListT5, LiT5, Transformer Reranker).
- API-Based: Uses external services like Cohere, Jina, Voyage, or MixedBread.ai via API Reranker.
- LLM-Based: Uses large language models (e.g., First Reranker, Incontext Reranker, Vicuna Reranker, Zephyr Reranker).
- Embedding-Based: Uses vector representations (e.g., ColBERT Reranker, Sentence Transformer, SPLADE Reranker, LLM2Vec Reranker, LLM Layerwise).
- Specialized: FlashRank (ONNX-based), Blender Reranker (PairRM), TwoLAR (Two-stage listwise), and Rank FiD (Fusion-in-Decoder).
https://github.com/datascienceuibk/rankify/blob/main/docs/docs/api/index.md

rankify

What's inside rankify

Recommended Methods by Use Case

Search Pipeline

Rerank Pipeline

RAG Pipeline

Pointwise Rerankers

Pairwise Rerankers

Listwise Rerankers

API-Based Rerankers

FiD Architecture

Standard RAG