DOCSEARCHbyAlgolia
Submit Docs

BEIR

repository·main·Indexed 25 days ago

https://github.com/beir-cellar/beir

A heterogeneous benchmark for the zero-shot evaluation of Information Retrieval (IR) models. BEIR provides a common framework to evaluate diverse retrieval architectures, including lexical, dense, sparse, and reranking models, across various datasets. It supports 17 preprocessed benchmark datasets and allows users to integrate their own IR datasets and models using standard evaluation metrics like NDCG, MAP, Recall, and Precision.

Tokens
7.8K
Snippets
9
Records
30
Agent score
80%

What's inside beir

  1. Overview of BEIR features

    main

    BEIR (Benchmarking IR) is a heterogeneous benchmark for evaluating NLP-based retrieval models across diverse Information Retrieval (IR) tasks. Key features include:

    • Dataset Support: Preprocess your own IR datasets or use one of the 17 already-preprocessed benchmark datasets.
    • Diverse Benchmarks: Includes wide settings suitable for both academic research and industrial applications.
    • Architecture Evaluation: Supports evaluation of lexical, dense, sparse, and reranking-based retrieval architectures.
    • Extensible Framework: Easily add and evaluate your own models using various state-of-the-art evaluation metrics.

  2. Reproduce retrieval on BioASQ dataset

    main

    To reproduce retrieval experiments using the BioASQ dataset, follow these steps:

    Corpus Preparation

    1. Register at BioASQ: http://www.bioasq.org/
    2. Download documents from BioASQ task 9a (Training v.2020 ~ 14,913,939 docs).
    3. Extract the title and abstractText for each document.
    4. Manually add missing documents found in the test qrels using this spreadsheet: https://docs.google.com/spreadsheets/d/1GZghfN5RT8h01XzIlejuwhBIGe8f-VaGf-yGaq11U-k/edit#gid=2015463710

    Queries and Qrels

    1. Download the Training and Test datasets from the BioASQ 8B datasets (published in 2020).
    2. Treat all documents containing answers as relevant (binary label) for a given question.

  4. Reproduce retrieval on Signal-1M dataset

    main

    To reproduce retrieval experiments using the Signal-1M dataset, follow these steps:

    Corpus Preparation

    1. Manually scrape tweets for the IDs listed here: https://github.com/igorbrigadir/newsir16-data/tree/master/twitter/curated
    2. Use the tweepy Python package for scraping.
    3. Preprocess the text by removing emojis and links.
    4. Remove any tweets that are empty or contain no text.

    Queries and Qrels

    1. Sign up at the Signal1M website to download qrels: https://research.signal-ai.com/datasets/signal1m-tweetir.html
    2. Sign up at the Signal1M website to download queries: https://research.signal-ai.com/datasets/signal1m.html
    3. Use the query title for experiments.

  5. Access BEIR Wiki for advanced guides

    main

    The BEIR Wiki contains detailed documentation for several key tasks:

    Quick Start

    • Installing BEIR: Setup instructions.
    • Examples and Tutorials: Practical usage guides.

    Datasets

    • Datasets Available: Full list of datasets.
    • Multilingual Datasets: Information on multilingual support.
    • Load your Custom Dataset: How to integrate your own data.

    Models

    • Models Available: List of supported models.
    • Evaluate your Custom Model: How to run evaluations on your own models.

    Metrics

    • Metrics Available: Details on evaluation metrics.

    Miscellaneous

    • BEIR Leaderboard: Current rankings.
    • Course Material on IR: Educational resources.

  6. Reproduce retrieval on TREC-NEWS dataset

    main

    To reproduce retrieval experiments using the TREC-NEWS dataset, follow these steps for corpus and query preparation:

    Corpus Preparation

    1. Request access to the Washington Post (WaPo) Corpus via the NIST application: https://trec.nist.gov/data/wapost/
    2. Iterate through contents and extract all paragraph subtypes.
    3. If the mime type is text/html, extract text from HTML; if text/plain, include the text directly.
    4. Use the html2text Python package to handle HTML-to-text conversion.

    Queries and Qrels

    1. Download background linking topics and qrels from the 2019 News Track: https://trec.nist.gov/data/news2019.html
    2. Use the document title as the query for experiments.

  7. Evaluate Reranking Architectures using Cross-Encoders

    main

    This guide provides a leaderboard for evaluating different reranking architectures, specifically comparing performance (accuracy across various datasets) and speed (Docs / Sec). The evaluation focuses on re-ranking the top-100 results retrieved by BM25 using Cross-Encoder models.

    Model Architectures Evaluated

    • MiniLM Models: A family of models (e.g., cross-encoder/ms-marco-MiniLM-L-2-v2 through L-12-v2) that balance performance and throughput.
    • TinyBERT Models: Highly efficient models (e.g., cross-encoder/ms-marco-TinyBERT-L-2-v2) designed for high throughput.
    • Electra Models: e.g., cross-encoder/ms-marco-electra-base.

    When choosing a model, consider the trade-off between Docs / Sec (speed) and the retrieval metrics (like MSMARCO, TREC-COVID, etc.) provided in the benchmark tables.

  8. Evaluate ColBERT on a BEIR dataset using evaluate_beir.sh

    main

    The evaluation process follows a five-step pipeline: Preprocessing, Indexing, FAISS IVFPQ Indexing, Query Retrieval, and BEIR Evaluation. You can automate this using the evaluate_beir.sh script.

    1. BEIR Preprocessing

    Convert BEIR jsonl data into ColBERT-friendly tsv format using colbert.data_prep.

    2. ColBERT Indexing

    Precompute ColBERT representations of passages. Note: You must provide a path to a trained ColBERT model checkpoint.

    3. FAISS IVFPQ Index

    Train an IVFPQ faiss index for end-to-end retrieval. Note: You must choose a different number of partitions (--partitions) for each dataset (e.g., 96 for NFCorpus).

    4. Query Retrieval

    Retrieve the top-k documents. The resulting ranking.tsv file contains integer document IDs corresponding to the position in the original collection TSV.

    5. BEIR Evaluation

    Run the final evaluation using the ranking.tsv produced in the previous step.

  9. Reproduce retrieval on Robust04 dataset

    main

    To reproduce retrieval experiments using the Robust04 dataset, follow these steps:

    Corpus Preparation

    1. Request access to TREC disks 4 and 5: https://trec.nist.gov/data/cd45/index.html
    2. Download and format the data according to ir_datasets to obtain the preprocessed corpus from: https://ir-datasets.com/trec-robust04.html#trec-robust04

    Queries and Qrels

    1. Download queries and qrels using the ir_datasets key trec-robust04 from: https://ir-datasets.com/trec-robust04.html#trec-robust04
    2. Use the query description for retrieval experiments.

  10. Install the beir-ColBERT environment

    main

    To evaluate ColBERT models on the BEIR benchmark, you must use a modified version of the ColBERT repository. Follow these steps to set up the environment using Conda:

    1. Clone the modified repository: git clone https://github.com/NThakur20/beir-ColBERT.git
    2. Create and activate the Conda environment using the provided conda_env.yml file, which includes the beir repository via pip:
      conda env create -f conda_env.yml
conda activate colbert-v0.2

    Troubleshooting Note: If you encounter issues with _swigfaiss while using faiss-cpu or faiss-gpu on Ubuntu, refer to the Faiss issue discussion.

    git clone https://github.com/NThakur20/beir-ColBERT.git
conda env create -f conda_env.yml
conda activate colbert-v0.2

  11. Verify dataset integrity with md5sum

    main

    To ensure that a downloaded dataset file is not corrupted, use the md5sum command in your terminal and compare the output with the md5 value provided in the BEIR dataset documentation.

    md5sum filename.zip