DOCSEARCHbyAlgolia
Submit Docs

Pyserini Documentation

repository·master·Indexed 24 days ago

https://github.com/castorini/pyserini

A Python toolkit for reproducible information retrieval research supporting sparse (Lucene-based) and dense (Faiss-based) representations. It provides capabilities for lexical models like BM25, learned sparse models such as SPLADE and uniCOIL, and dense models including DPR and BGE. The toolkit includes prebuilt indexes, evaluation scripts for IR test collections, a REST API, and an MCP server for RAG applications.

Tokens
137K
Snippets
326
Records
498
Agent score
82%

What's inside pyserini

  1. Overview of Pyserini capabilities

    master

    Pyserini is a Python toolkit designed for reproducible information retrieval (IR) research. It provides first-stage retrieval capabilities using both sparse and dense representations.

    Core Features:

    • Sparse Retrieval: Integrated with Anserini (built on Lucene) for traditional lexical models (e.g., BM25) and learned sparse models (e.g., SPLADE, uniCOIL).
    • Dense Retrieval: Integrated with Facebook's Faiss library for learned dense models (e.g., DPR, Contriever, BGE).
    • Hybrid Retrieval: Supports dense-sparse fusion.
    • Research Ready: Includes prebuilt indexes, queries, relevance judgments, and evaluation scripts for standard IR test collections.
    • Interfaces: Provides a REST API and an MCP server.

  7. Details of the msmarco-v2-doc-segmented-unicoil-0shot Lucene index

    master
    This index is a Lucene impact index of the MS MARCO V2 segmented document corpus, specifically prepared for uniCOIL with titles prepended. It was generated using Anserini and is intended for use in retrieval tasks requiring this specific corpus configuration.

  10. Details of the wikipedia-dpr-dkrr-tqa index

    master
    The wikipedia-dpr-dkrr-tqa index is a Faiss FlatIP index of Wikipedia. It uses DPR (Dense Passage Retrieval) embeddings generated by the retriever model from the paper "Distilling Knowledge from Reader to Retriever for Question Answering", which was trained on the TriviaQA dataset. This specific index was generated on 2022/02/17 using Pyserini commit cc91b2 and FiD commit 25ed1f.

  12. Details for the wiki-all-6-3-dpr2-multi Faiss FlatIP index

    master

    This index is a Faiss FlatIP index of the wiki-all-6-3 corpus (available at https://huggingface.co/datasets/castorini/odqa-wiki-corpora). It uses embeddings generated by a 2nd iteration DPR model trained on multiple QA datasets, specifically the castorini/wiki-all-6-3-multi-dpr2-passage-encoder model.

    Index Metadata:

    • Generation Date: 2023/01/03
    • Pyserini Commit: 186fa7
    • Tevatron Commit: 7a5afe