DOCSEARCHbyAlgolia
Submit Docs

SciBERT Documentation

repository·master·Indexed 23 days ago

https://github.com/allenai/scibert

A BERT-based language model trained on a corpus of 1.14M scientific papers from semantic scholar.org. Designed for scientific NLP tasks including NER, relation extraction, and text classification. Supports integration via Hugging Face Transformers, AllenNLP, and TensorFlow, with available models using both scivocab and basevocab in cased and uncased versions.

Tokens
1.3K
Snippets
2
Records
5
Agent score
33%

What's inside SciBERT

  1. Train new SciBERT models using AllenNLP

    master

    To train new models or reproduce results using AllenNLP, follow these steps:

    1. Setup Environment: Install dependencies using Python 3.6.

      pip install -r requirements.txt

    2. Prepare Model Files: Download a PyTorch model and decompress it.

      tar -xvf scibert_scivocab_uncased.tar

      This creates a directory containing vocab.txt and weights.tar.gz.

    3. Configure Task and Dataset: Modify the scibert/scripts/train_allennlp_local.sh script. For example, to run Named Entity Recognition (NER) on the bc5cdr dataset:

      DATASET='bc5cdr'
TASK='ner'

    4. Set Environment Variables: Point the script to your model files.

      export BERT_VOCAB=path-to/scibert_scivocab_uncased.vocab
export BERT_WEIGHTS=path-to/scibert_scivocab_uncased.tar.gz

    5. Run Training: Execute the script with a serialization directory.

      ./scibert/scripts/train_allennlp_local.sh [serialization-directory]
    pip install -r requirements.txt

tar -xvf scibert_scivocab_uncased.tar

export BERT_VOCAB=path-to/scibert_scivocab_uncased.vocab
export BERT_WEIGHTS=path-to/scibert_scivocab_uncased.tar.gz

./scibert/scripts/train_allennlp_local.sh [serialization-directory]

  2. Download SciBERT pretrained models

    master

    SciBERT models are available in three formats depending on your framework:

    1. Tensorflow: Compatible with Google Research's BERT implementation.
    2. PyTorch (AllenNLP): Designed for use within the AllenNLP framework.
    3. PyTorch (HuggingFace): Designed for use with the Hugging Face library.

    All combinations of scivocab (custom scientific vocabulary) and basevocab (original BERT vocabulary), as well as cased and uncased versions, are available. scivocab-uncased is generally recommended.

    Tensorflow Model Links:

    PyTorch AllenNLP Model Links:

    PyTorch HuggingFace Model Links:

  3. Install SciBERT via Hugging Face Transformers

    master

    The easiest way to use SciBERT is through the Hugging Face transformers library. You can install the pretrained models directly from the allenai organization.

    Recommended models:

    • allenai/scibert_scivocab_uncased (usually provides the best results)
    • allenai/scibert_scivocab_cased
    from transformers import *

tokenizer = AutoTokenizer.from_pretrained('allenai/scibert_scivocab_uncased')
model = AutoModel.from_pretrained('allenai/scibert_scivocab_uncased')

tokenizer = AutoTokenizer.from_pretrained('allenai/scibert_scivocab_cased')
model = AutoModel.from_pretrained('allenai/scibert_scivocab_cased')

  4. Cite SciBERT in research

    master

    If you use SciBERT in your research, please cite the following paper:

    @inproceedings{Beltagy2019SciBERT,
  title={SciBERT: Pretrained Language Model for Scientific Text},
  author={Iz Beltagy and Kyle Lo and Arman Cohan},
  year={2019},
  booktitle={EMNLP},
  Eprint={arXiv:1903.10676}
}

  5. Available scientific NLP datasets in SciBERT

    master

    The repository includes several scientific NLP datasets organized by task under the data/ directory:

    • ner (Named Entity Recognition): JNLPBA, NCBI-disease, bc5cdr, sciie
    • parsing: genia
    • pico: ebmnlp
    • text_classification: chemprot, citation_intent, mag, rct-20k, sci-cite, sciie-relation-extraction