DOCSEARCHbyAlgolia
Submit Docs

HotpotQA Documentation

repository·master·Indexed 20 days ago

https://github.com/hotpotqa/hotpot

A dataset and pipeline for diverse, explainable multi-hop question answering. It includes tools for data downloading, preprocessing, training baseline models, and evaluating performance in 'distractor' and 'fullwiki' settings. The pipeline is managed via main.py, supporting modes for training, preprocessing, and testing.

Tokens
2.5K
Snippets
7
Records
11
Agent score
70%

What's inside HotpotQA

  1. Understand the HotpotQA JSON data format

    master

    The dataset consists of a list of question-answer data points. Each data point is a dictionary containing:

    • _id: Unique identifier for the QA pair.
    • question: The question string.
    • answer: The ground truth answer string (not present in the test set).
    • supporting_facts: A list of [title, sent_id] pairs, where title is the paragraph title and sent_id is the 0-based sentence index.
    • context: A list of paragraphs, where each paragraph is [title, sentences] and sentences is a list of strings.

    Additional metadata keys (not used by the core code and not present in test sets):

    • type: comparison or bridge.
    • level: easy, medium, or hard.

  2. Train a HotpotQA model

    master

    Run the training pipeline using main.py. You can specify a single GPU using CUDA_VISIBLE_DEVICES or run on all available GPUs by omitting that variable.

    Example training command:

    CUDA_VISIBLE_DEVICES=0 python main.py --mode train --para_limit 2250 --batch_size 24 --init_lr 0.1 --keep_prob 1.0 --sp_lambda 1.0

    Note the generated file name (e.g., HOTPOT-20180924-160521) as it is required for evaluation.

  4. Install HotpotQA requirements

    master

    To run the HotpotQA pipeline, you need Python 3, pytorch 0.3.0, and spacy.

    Note: Installing pytorch 0.3.0 may require specific instructions based on your CUDA version. For example, using conda with CUDA 8.0:

    conda install pytorch=0.3.0 cuda80 -c pytorch
conda install spacy

  5. Submit models to Codalab

    master

    Evaluation for the test set is performed via Codalab.

    • Distractor Setting: You must submit your code and provide a Docker environment. Your code will be executed against the test set.
    • Fullwiki Setting: You only need to submit your prediction file.

    Detailed instructions can be found on the Codalab worksheet.

  6. Perform local evaluation

    master

    Local evaluation involves two steps: generating predictions and then running the evaluation script.

    1. Generate Predictions

    Use --mode test and provide a --save identifier and a --prediction_file name.

    Distractor Setting:

    CUDA_VISIBLE_DEVICES=0 python main.py --mode test --data_split dev --para_limit 2250 --batch_size 24 --init_lr 0.1 --keep_prob 1.0 --sp_lambda 1.0 --save HOTPOT-20180924-160521 --prediction_file dev_distractor_pred.json

    Fullwiki Setting:

    CUDA_VISIBLE_DEVICES=0 python main.py --mode test --data_split dev --para_limit 2250 --batch_size 24 --init_lr 0.1 --keep_prob 1.0 --sp_lambda 1.0 --save HOTPOT-20180924-160521 --prediction_file dev_fullwiki_pred.json --fullwiki

    2. Run Evaluation Script

    Pass the generated prediction file and the original ground truth JSON to hotpot_evaluate_v1.py.

    Distractor:

    python hotpot_evaluate_v1.py dev_distractor_pred.json hotpot_dev_distractor_v1.json

    Fullwiki:

    python hotpot_evaluate_v1.py dev_fullwiki_pred.json hotpot_dev_fullwiki_v1.json
    # Example for Distractor setting
CUDA_VISIBLE_DEVICES=0 python main.py --mode test --data_split dev --para_limit 2250 --batch_size 24 --init_lr 0.1 --keep_prob 1.0 --sp_lambda 1.0 --save HOTPOT-20180924-160521 --prediction_file dev_distractor_pred.json
python hotpot_evaluate_v1.py dev_distractor_pred.json hotpot_dev_distractor_v1.json

  7. Preprocess HotpotQA datasets

    master

    Preprocessing converts raw JSON files into a format suitable for training. Important: You must preprocess the training set before the dev sets to generate necessary vocabulary and embedding files.

    Distractor Setting

    Preprocess the training and dev sets using the --mode prepro flag:

    python main.py --mode prepro --data_file hotpot_train_v1.1.json --para_limit 2250 --data_split train
python main.py --mode prepro --data_file hotpot_dev_distractor_v1.json --para_limit 2250 --data_split dev

    Fullwiki Setting

    Preprocess the dev set using the --fullwiki flag:

    python main.py --mode prepro --data_file hotpot_dev_fullwiki_v1.json --data_split dev --fullwiki --para_limit 2250
    python main.py --mode prepro --data_file hotpot_train_v1.1.json --para_limit 2250 --data_split train
python main.py --mode prepro --data_file hotpot_dev_distractor_v1.json --para_limit 2250 --data_split dev
python main.py --mode prepro --data_file hotpot_dev_fullwiki_v1.json --data_split dev --fullwiki --para_limit 2250

  8. Format prediction files for submission

    master

    Prediction files (e.g., dev_distractor_pred.json) must be JSON objects with two primary keys:

    • answer: A dictionary where keys are the QA pair _id and values are the predicted answer strings.
    • sp: A dictionary where keys are the QA pair _id and values are lists of predicted supporting facts. Each supporting fact is a list: [title, sent_id] (where sent_id is 0-based).

  9. Configure fullwiki data paths

    master

    When the --fullwiki flag is provided, the script automatically prefixes several file paths with fullwiki. to load the full Wikipedia version of the datasets. This affects the following files:

    • dev_record_file
    • test_record_file
    • dev_eval_file
    • test_eval_file

  10. Reference: CLI arguments for main.py

    master

    The following command-line arguments are available to configure the execution mode, data paths, embedding settings, and model hyperparameters.

    --mode {train,prepro,test,count} (default: 'train')
--data_file (path to data file)
--glove_word_file (path to GloVe file, default: 'glove.840B.300d.txt')
--save (save directory, default: 'HOTPOT')
--word_emb_file (path to word embeddings, default: 'word_emb.json')
--char_emb_file (path to character embeddings, default: 'char_emb.json')
--train_eval_file (path to train/eval JSON, default: 'train_eval.json')
--dev_eval_file (path to dev eval JSON, default: 'dev_eval.json')
--test_eval_file (path to test eval JSON, default: 'test_eval.json')
--word2idx_file (path to word2idx mapping, default: 'word2idx.json')
--char2idx_file (path to char2idx mapping, default: 'char2idx.json')
--idx2word_file (path to idx2word mapping, default: 'idx2word.json')
--idx2char_file (path to idx2char mapping, default: 'idx2char.json')
--train_record_file (path to train pickle, default: 'train_record.pkl')
--dev_record_file (path to dev pickle, default: 'dev_record.pkl')
--test_record_file (path to test pickle, default: 'test_record.pkl')
--glove_char_size (int, default: 94)
--glove_word_size (int, default: 2200000)
--glove_dim (int, default: 300)
--char_dim (int, default: 8)
--para_limit (int, default: 1000)
--ques_limit (int, default: 80)
--sent_limit (int, default: 100)
--char_limit (int, default: 16)
--batch_size (int, default: 64)
--checkpoint (int, default: 1000)
--period (int, default: 100)
--init_lr (float, default: 0.5)
--keep_prob (float, default: 0.8)
--hidden (int, default: 80)
--char_hidden (int, default: 100)
--patience (int, default: 1)
--seed (int, default: 13)
--sp_lambda (float, default: 0.0)
--data_split (str, default: 'train')
--fullwiki (flag, enables fullwiki prefixing for files)
--prediction_file (path to prediction output)
--sp_threshold (float, default: 0.3)

  11. Run HotpotQA pipelines via CLI

    master

    The main.py script serves as the primary entrypoint for the HotpotQA project. You can execute different pipeline modes using the --mode argument. Supported modes include:

    • train: Executes the training pipeline.
    • prepro: Executes the data preprocessing pipeline.
    • test: Executes the testing/evaluation pipeline.
    • count: Executes a counting utility (calls cnt_len).

    By default, the mode is set to train.

    python main.py --mode train
python main.py --mode prepro
python main.py --mode test