DOCSEARCHbyAlgolia
Submit Docs

BioBERT Documentation

repository·master·Indexed 24 days ago

https://github.com/dmis-lab/biobert

A biomedical language representation model designed for fine-tuning on biomedical text mining tasks, including Named Entity Recognition (NER), Relation Extraction (RE), and Question Answering (QA). This implementation is based on the original Google BERT repository, utilizing Tensorflow 1 and Python <= 3.7. It provides various pre-trained weights (Base and Large) trained on PubMed and PMC corpora, along with scripts for training, inference, and evaluation using benchmark datasets.

Tokens
2.2K
Snippets
5
Records
8
Agent score
31%

What's inside BioBERT

  2. Download BioBERT pre-trained weights

    master

    Before fine-tuning, you must download the pre-trained weights. Several versions are available depending on your needs (Base vs Large, and different training corpora like PubMed or PMC).

    Available versions include:

    • BioBERT-Base v1.2 (+ PubMed 1M): Includes LM head, useful for probing (PyTorch compatible via HuggingFace).
    • BioBERT-Large v1.1 (+ PubMed 1M): Based on BERT-large-Cased with a custom 30k vocabulary.
    • BioBERT-Base v1.1 (+ PubMed 1M): Based on BERT-base-Cased.
    • BioBERT-Base v1.0 variants: Available with PubMed 200K, PMC 270K, or a combination of both.

    You can find the weights on HuggingFace or via the project's checkpoint links.

  3. Download benchmark datasets

    master

    BioBERT provides pre-processed benchmark datasets for Named Entity Recognition (NER), Relation Extraction (RE), and Question Answering (QA). You can download all datasets at once using the provided download.sh script, which will place them in a datasets folder.

    $ ./download.sh

  4. Fine-tune BioBERT for Relation Extraction (RE)

    master

    To fine-tune BioBERT for RE, set $RE_DIR (dataset folder), $TASK_NAME (either gad or euadr), and $OUTPUT_DIR (results folder).

    Training

    Run run_re.py with --do_train=true and --do_eval=true. Predictions are saved to test_results.tsv in $OUTPUT_DIR.

    Evaluation

    Use biocodes/re_eval.py to evaluate the results.

    Note for CHEMPROT: Since CHEMPROT is a multi-class classification dataset, you must include the --task=chemprot flag when running the evaluation script.

    # Setup environment
export RE_DIR=./datasets/RE/GAD/1
export TASK_NAME=gad
export OUTPUT_DIR=./re_outputs_1

# Run fine-tuning
python run_re.py --task_name=$TASK_NAME --do_train=true --do_eval=true --do_predict=true --vocab_file=$BIOBERT_DIR/vocab.txt --bert_config_file=$BIOBERT_DIR/bert_config.json --init_checkpoint=$BIOBERT_DIR/model.ckpt-1000000 --max_seq_length=128 --train_batch_size=32 --learning_rate=2e-5 --num_train_epochs=3.0 --do_lower_case=false --data_dir=$RE_DIR --output_dir=$OUTPUT_DIR

# Evaluate results
python ./biocodes/re_eval.py --output_path=$OUTPUT_DIR/test_results.tsv --answer_path=$RE_DIR/test.tsv

  5. Install BioBERT

    master

    BioBERT is based on Tensorflow 1 and requires Python version <= 3.7. To install the repository and its dependencies, clone the repository and install the requirements via pip. Note that this implementation is based on the original Google BERT repository. If you need to use the official BioASQ evaluation script, you should also install java on your system.

    $ git clone https://github.com/dmis-lab/biobert.git
$ cd biobert; pip install -r requirements.txt

  7. Fine-tune BioBERT for Named Entity Recognition (NER)

    master

    To fine-tune BioBERT for NER, prepare a dataset directory ($NER_DIR) containing train_dev.tsv, train.tsv, devel.tsv, and test.tsv. You must also set $BIOBERT_DIR to the path of your unpacked pre-trained weights and $OUTPUT_DIR for results.

    Training and Evaluation

    Run run_ner.py with --do_train=true and --do_eval=true.

    Inference

    To evaluate test.tsv using a trained model, use --do_train=false and --do_predict=true.

    Post-processing and Entity-level Evaluation

    1. run_ner.py produces token_test.txt and label_test.txt in $OUTPUT_DIR (token-level results).
    2. Use biocodes/ner_detokenize.py to convert these to a word-level prediction file (NER_result_conll.txt).
    3. Use biocodes/conlleval.pl to perform entity-level exact match evaluation.
    # Setup environment
export BIOBERT_DIR=./biobert_v1.1_pubmed
export NER_DIR=./datasets/NER/NCBI-disease
export OUTPUT_DIR=./ner_outputs

# Run fine-tuning
mkdir -p $OUTPUT_DIR
python run_ner.py --do_train=true --do_eval=true --vocab_file=$BIOBERT_DIR/vocab.txt --bert_config_file=$BIOBERT_DIR/bert_config.json --init_checkpoint=$BIOBERT_DIR/model.ckpt-1000000 --num_train_epochs=10.0 --data_dir=$NER_DIR --output_dir=$OUTPUT_DIR

# Detokenize for word-level results
python biocodes/ner_detokenize.py --token_test_path=$OUTPUT_DIR/token_test.txt --label_test_path=$OUTPUT_DIR/label_test.txt --answer_path=$NER_DIR/test.tsv --output_dir=$OUTPUT_DIR

# Entity-level evaluation
perl biocodes/conlleval.pl < $OUTPUT_DIR/NER_result_conll.txt

  8. Fine-tune BioBERT for Question Answering (QA)

    master

    To use the BioASQ dataset, you must register on the BioASQ website. Unpack the pre-processed BioASQ dataset into $QA_DIR and set $OUTPUT_DIR.

    Training

    Run run_qa.py with --do_train=True and --do_predict=True. Predictions are saved as predictions.json and nbest_predictions.json in $OUTPUT_DIR.

    Official BioASQ Evaluation

    1. Convert nbest_predictions.json to BioASQ format using biocodes/transform_nbset2bioasqform.py.
    2. Clone the BioASQ Evaluation-Measures repository.
    3. Run the Java evaluator. Note that the -e parameter should always be set to 5.
    # Setup environment
export QA_DIR=./datasets/QA/BioASQ
export OUTPUT_DIR=./qa_outputs

# Run fine-tuning
python run_qa.py --do_train=True --do_predict=True --vocab_file=$BIOBERT_DIR/vocab.txt --bert_config_file=$BIOBERT_DIR/bert_config.json --init_checkpoint=$BIOBERT_DIR/model.ckpt-1000000 --max_seq_length=384 --train_batch_size=12 --learning_rate=5e-6 --doc_stride=128 --num_train_epochs=5.0 --do_lower_case=False --train_file=$QA_DIR/BioASQ-train-factoid-4b.json --predict_file=$QA_DIR/BioASQ-test-factoid-4b-1.json --output_dir=$OUTPUT_DIR

# Convert to BioASQ format
python ./biocodes/transform_nbset2bioasqform.py --nbest_path=$OUTPUT_DIR/nbest_predictions.json --output_path=$OUTPUT_DIR

# Official Evaluation (requires Evaluation-Measures repo)
git clone https://github.com/BioASQ/Evaluation-Measures.git
cd Evaluation-Measures
java -Xmx10G -cp $CLASSPATH:./flat/BioASQEvaluation/dist/BioASQEvaluation.jar evaluation.EvaluatorTask1b -phaseB -e 5 ../$QA_DIR/4B1_golden.json ../$OUTPUT_DIR/BioASQform_BioASQ-answer.json