SciFact: Scientific Claim Verification

Abstract-level evaluation determines if a predicted document is correct based on three criteria:

1. The abstract is a relevant abstract (exists in the gold set).
2. The predicted label matches the gold label.
3. At least one gold evidence set is fully contained within the predicted rationale sentences.

Important Constraint: The evaluation code only considers the first three predicted evidence sentences for a given abstract. Additional sentences are ignored to avoid penalizing models for over-predicting rationale sentences (similar to FEVER scoring).

Metrics:

• Precision: (# correctly predicted abstracts) / (# predicted abstracts)
• Recall: (# correctly predicted abstracts) / (# gold abstracts)

Sentence-level evaluation is more granular and uses all predicted evidence sentences (unlike abstract-level scoring).

An evidence sentence is considered correctly predicted if:

1. It belongs to a relevant abstract.
2. The abstract's predicted label matches the gold label.
3. It is part of a gold evidence set, AND all other sentences in that same gold evidence set are also included in the predicted sentences.

If a model predicts some but not all sentences of a gold evidence set, it receives no credit for any of those sentences at the sentence level.

Metrics:

• Precision: (# correctly predicted evidence sentences) / (# predicted evidence sentences)
• Recall: (# correctly predicted evidence sentences) / (# gold evidence sentences)

You can use provided shell scripts to reproduce the rationale selection and label prediction metrics reported in the paper. These scripts automatically handle dataset and model downloads.

Rationale Selection Metrics

Use ./script/rationale-selection.sh to recreate Table 3 rationale selection metrics.

Label Prediction Metrics

Use ./script/label-prediction.sh to recreate Table 3 label prediction metrics.

Arguments:

• [bert-variant]: roberta_large, roberta_base, scibert, or biomed_roberta_base.
• [training-dataset]: scifact, scifact_only_claim, scifact_only_rationale, fever_scifact, fever, or snopes.
• [dataset]: dev.

./script/rationale-selection.sh [bert-variant] [training-dataset] [dataset]
./script/label-prediction.sh [bert-variant] [training-dataset] [dataset]

To run the full prediction pipeline, use the ./script/pipeline.sh script. This is useful for generating predictions on the test set (note that test set labels are not public).

Arguments:

• [retrieval]: oracle (requires gold evidence, works on dev only) or open (works on dev and test).
• [model]: oracle-rationale, zero-shot, or verisci.
• [dataset]: dev or test.

Note: For the dev set, the script also computes performance metrics. For the test set, it only generates predictions.

./script/pipeline.sh [retrieval] [model] [dataset]

To get end-to-end results, you must merge the outputs from the rationale selection and label prediction steps, then evaluate the merged results.

1. Merge Predictions: Combine rationale and label files into a single prediction file.
2. Evaluate Pipeline: Run the full evaluation against the gold dataset and corpus.

# 1. Merge label and rationale predictions
python verisci/inference/merge_predictions.py \
    --rationale-file prediction/rationale_selection.jsonl \
    --label-file prediction/label_prediction.jsonl \
    --result-file prediction/merged_predictions.jsonl

# 2. Evaluate the merged predictions
python verisci/evaluate/pipeline.py \
    --gold data/claims_dev.jsonl \
    --corpus data/corpus.jsonl \
    --prediction prediction/merged_predictions.jsonl \
    --output predictions/metrics.json

Pre-trained BERT-to-BERT models are available via a download script. The script skips downloads if the model already exists.

Arguments:

• [model-component]: rationale or label.
• [bert-variant]: roberta_large, roberta_base, scibert, or biomed_roberta_base.
• [training-dataset]: scifact, scifact_only_claim, scifact_only_rationale, fever_scifact, fever, or snopes.

Recommended configurations for best performance:

• Rationale: roberta_large + scifact
• Label: roberta_large + fever_scifact.

./script/download-model.sh [model-component] [bert-variant] [training-dataset]

The dataset can be downloaded using the provided script, which stores the files in a data directory. Alternatively, you can download the tarball manually.

Files included:

• corpus.jsonl: The corpus of evidence documents.
• claims_train.jsonl, claims_dev.jsonl, claims_test.jsonl: Claim files (one claim per line). The train and dev sets contain labels; the test set is unlabeled.
• cross_validation/: A 5-fold cross-validation split for model development.

For detailed schema descriptions, refer to doc/data.md.

./script/download-data.sh

Label prediction determines the final veracity of the claim based on the selected rationales. You can use an Oracle or a Transformer-based model.

Transformer Label Prediction

Use verisci/inference/label_prediction/transformer.py. You can control the input format using the --mode flag.

--mode options:

• claims_and_rationale (default)
• only_claim
• only_rationale

# Transformer Label Prediction
python verisci/inference/label_prediction/transformer.py \
    --corpus data/corpus.jsonl \
    --dataset data/claims_dev.jsonl \
    --rationale-selection rationale_selection.jsonl \
    --model PATH_TO_MODEL \
    --output label_prediction.jsonl

You must acquire the Snopes dataset independently. Once acquired, use the following commands to train the modules.

Train rationale selection module:

python verisci/training/rationale_selection/transformer_snopes.py \
    --corpus "PATH TO snopes.pages.json" \
    --evidence-train "PATH TO snopes.evidence.train.jsonl" \
    --evidence-dev "PATH TO snopes.evidence.dev.jsonl" \
    --model "roberta-large" \
    --dest "PATH TO WEIGHT SAVING FOLDER"

Train label prediction module:

python verisci/training/label_prediction/transformer_snopes.py \
    --corpus "PATH TO snopes.pages.json" \
    --evidence-train "PATH TO snopes.stance.train.jsonl" \
    --evidence-dev "PATH TO snopes.stance.dev.jsonl" \
    --model "roberta-large" \
    --dest "PATH TO WEIGHT SAVING FOLDER"