DOCSEARCHbyAlgolia
Submit Docs

BioGPT Documentation

repository·main·Indexed 26 days ago

https://github.com/microsoft/biogpt

BioGPT is a generative pre-trained transformer designed for biomedical text generation and mining tasks, including relation extraction, question answering, and document classification. It provides pre-trained and fine-tuned checkpoints (BioGPT, BioGPT-Large) and is compatible with fairseq and the Hugging Face transformers library via BioGptForCausalLM and BioGptTokenizer.

Tokens
4.5K
Snippets
9
Records
22
Agent score
88%

What's inside BioGPT

  3. Use BioGPT with Hugging Face Transformers

    main

    BioGPT is integrated into the Hugging Face transformers library. You can use BioGptForCausalLM and BioGptTokenizer for text generation, feature extraction, or beam-search decoding.

    # Text Generation Pipeline
from transformers import pipeline, set_seed
from transformers import BioGptTokenizer, BioGptForCausalLM

model = BioGptForCausalLM.from_pretrained("microsoft/biogpt")
tokenizer = BioGptTokenizer.from_pretrained("microsoft/biogpt")
generator = pipeline('text-generation', model=model, tokenizer=tokenizer)
set_seed(42)
print(generator("COVID-19 is", max_length=20, num_return_sequences=5, do_sample=True))

# Feature Extraction
encoded_input = tokenizer("Replace me by any text you'd like.", return_tensors='pt')
output = model(**encoded_input)

# Beam-search decoding
import torch
from transformers import BioGptTokenizer, BioGptForCausalLM, set_seed

tokenizer = BioGptTokenizer.from_pretrained("microsoft/biogpt")
model = BioGptForCausalLM.from_pretrained("microsoft/biogpt")
sentence = "COVID-19 is"
inputs = tokenizer(sentence, return_tensors="pt")
set_seed(42)

with torch.no_grad():
    beam_output = model.generate(**inputs,
                                 min_length=100,
                                 max_length=1024,
                                 num_beams=5,
                                 early_stopping=True
                                )
print(tokenizer.decode(beam_output[0], skip_special_tokens=True))

  4. Download BioGPT pre-trained and fine-tuned checkpoints

    main

    BioGPT provides pre-trained checkpoints (BioGPT, BioGPT-Large) and fine-tuned checkpoints for tasks like Question Answering (PubMedQA), Relation Extraction (BC5CDR, DDI, KD-DTI), and Document Classification (HoC).

    Download the checkpoints and extract them into a checkpoints folder within the project directory.

    mkdir checkpoints
cd checkpoints
# Example for Pre-trained BioGPT
wget https://msralaphilly2.blob.core.windows.net/release/BioGPT/checkpoints/Pre-trained-BioGPT.tgz?sp=r&st=2023-11-13T15:37:35Z&se=2099-12-30T23:37:35Z&spr=https&sv=2022-11-02&sr=b&sig=3CcG1TOhqJPBhkVutvVn3PtUq0vPyLBgwggUfojypfY%3D
tar -zxvf Pre-trained-BioGPT.tgz

  5. Install BioGPT and its dependencies

    main

    BioGPT requires specific versions of PyTorch, Python, and fairseq, along with Moses, fastBPE, sacremoses, and scikit-learn. You must also set the MOSES and FASTBPE environment variables to the paths of their respective directories.

    Requirements:

    • PyTorch == 1.12.0
    • Python == 3.10
    • fairseq == 0.12.0
    • sacremoses
    • scikit-learn
    # Install fairseq v0.12.0
git clone https://github.com/pytorch/fairseq
cd fairseq
git checkout v0.12.0
pip install .
python setup.py build_ext --inplace
cd ..

# Install Moses
git clone https://github.com/moses-smt/mosesdecoder.git
export MOSES=${PWD}/mosesdecoder

# Install fastBPE
git clone https://github.com/glample/fastBPE.git
export FASTBPE=${PWD}/fastBPE
cd fastBPE
g++ -std=c++11 -pthread -O3 fastBPE/main.cc -IfastBPE -o fast
cd ..

# Install Python dependencies
pip install sacremoses scikit-learn

  7. Use manual or learned prompts in language_modeling_prompt

    main

    When using the language_modeling_prompt task, you can define how the model is prompted:

    1. Manual Prompt: Provide a specific string via --manual-prompt. The task will encode this string using the dictionary.
    2. Learned Prompt: Provide an integer via --learned-prompt to specify the number of virtual tokens. The task generates these tokens using the --learned-prompt-pattern (e.g., learned1 learned2 ...).

    Constraint: You must choose exactly one method. Providing both will result in an error.

  9. Use pre-trained BioGPT with fairseq

    main

    You can use the pre-trained BioGPT model using the TransformerLanguageModel from fairseq.models.transformer_lm. This requires the checkpoint path, the specific checkpoint file, and the data directory containing BPE codes.

    import torch
from fairseq.models.transformer_lm import TransformerLanguageModel

m = TransformerLanguageModel.from_pretrained(
        "checkpoints/Pre-trained-BioGPT", 
        "checkpoint.pt", 
        "data",
        tokenizer='moses', 
        bpe='fastbpe', 
        bpe_codes="data/bpecodes",
        min_len=100,
        max_len_b=1024)
m.cuda()
src_tokens = m.encode("COVID-19 is")
generate = m.generate([src_tokens], beam=5)[0]
output = m.decode(generate[0]["tokens"])
print(output)

  10. Use fine-tuned BioGPT for Relation Extraction (KD-DTI)

    main

    For downstream tasks like drug-target-interaction on KD-DTI, use the TransformerLanguageModelPrompt class from src.transformer_lm_prompt.

    import torch
from src.transformer_lm_prompt import TransformerLanguageModelPrompt

m = TransformerLanguageModelPrompt.from_pretrained(
        "checkpoints/RE-DTI-BioGPT", 
        "checkpoint_avg.pt", 
        "data/KD-DTI/relis-bin",
        tokenizer='moses', 
        bpe='fastbpe', 
        bpe_codes="data/bpecodes",
        max_len_b=1024,
        beam=1)
m.cuda()
src_text="" # input text, e.g., a PubMed abstract
src_tokens = m.encode(src_text)
generate = m.generate([src_tokens], beam=1)[0]
output = m.decode(generate[0]["tokens"])
print(output)

  11. Configure the language_modeling_prompt task

    main

    The language_modeling_prompt task (registered via LanguageModelingPromptTask) extends standard language modeling to support prompt-based training. You can configure it using the following arguments:

    • --source-lang / -s: Source language.
    • --target-lang / -t: Target language.
    • --max-source-positions: Maximum number of tokens in the source sequence, excluding EOS (default: 384).
    • --manual-prompt: A string used as a manual prompt.
    • --learned-prompt: The number of virtual tokens to use for a learned prompt. Note: You cannot use both --manual-prompt and --learned-prompt simultaneously.
    • --learned-prompt-pattern: The pattern for virtual tokens (default: 'learned').
    • --prefix: Boolean flag; if set, the prompt is treated as a prefix.
    • --sep-token: Token used to separate the prompt source and target (default: <seqsep>).

  12. Debug output formats for BC5CDR evaluation

    main

    When debug mode is enabled, the evaluation produces tab-delimited INFO lines. The formats are:

    • Mention evaluation: INFO {TP|FP|FN} mention documentId startOffset endOffset mentionType
    • ID evaluation: INFO {TP|FP|FN} id documentId mentionType conceptId
    • Relation evaluation: INFO {TP|FP|FN} relation documentId relationType conceptId1 conceptId2