DOCSEARCHbyAlgolia
Submit Docs

Swedish BERT Models

repository·master·Indexed 40 days ago

https://github.com/kungbib/swedish-bert-models

A collection of pretrained Swedish language models (BERT, ALBERT, Electra) provided by the National Library of Sweden (KBLab). Includes specialized models for Named Entity Recognition (NER), POS tagging, and Question Answering, such as bert-base-swedish-cased and bert-base-swedish-cased-ner.

Tokens
1.8K
Snippets
7
Records
8
Agent score
0%

What's inside Swedish BERT Models

  1. Use BERT Base Swedish models

    master

    This repository provides several model variants, including:

    • BERT Base Swedish: The foundational Swedish BERT model.
    • TF models: TensorFlow-compatible versions of the models.
    • BERT base fine-tuned for Swedish NER: A version of the model specifically fine-tuned for Named Entity Recognition (NER) tasks in Swedish.

  2. Load BERT Base Swedish models

    master

    The bert-base-swedish-cased model is a standard BERT base for Swedish trained on various sources with a vocabulary size of approximately 50k. You can load it using Huggingface Transformers in either PyTorch or TensorFlow.

    Important for Transformers 2.5.0+: To ensure accented characters like Å, Ä, and Ö are retained, add the parameter use_fast=False to AutoTokenizer.from_pretrained(...).

    from transformers import AutoModel, AutoTokenizer, TFAutoModel

# PyTorch
tok = AutoTokenizer.from_pretrained('KB/bert-base-swedish-cased')
model = AutoModel.from_pretrained('KB/bert-base-swedish-cased')

# TensorFlow
model = TFAutoModel.from_pretrained('KB/bert-base-swedish-cased')

  3. Install Swedish BERT Models environment

    master

    To run the provided examples, you need to set up a Python virtual environment and install the required dependencies. The examples require transformers version 2.4.1 and pytorch 1.3.1 or greater.

    Note for MacOS users: On some platforms (notably MacOSX < 10.15), you may need to install a Rust compiler to successfully install the transformers library.

    Follow these steps to set up your environment:

    # git clone https://github.com/Kungbib/swedish-bert-models
# cd swedish-bert-models
# python3 -m venv venv
# source venv/bin/activate
# pip install --upgrade pip
# pip install -r requirements.txt

  4. Install Swedish BERT models

    master

    To use the models provided in this repository, set up a Python virtual environment and install the necessary dependencies from requirements.txt.

    Follow these steps:

    1. Clone the repository.
    2. Navigate into the directory.
    3. Create and activate a virtual environment.
    4. Upgrade pip and install requirements.
    git clone https://github.com/Kungbib/swedish-bert-models
cd swedish-bert-models
python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

  5. Load ALBERT base Swedish models

    master

    The albert-base-swedish-cased-alpha model can be loaded using the standard Huggingface Transformers AutoModel and AutoTokenizer classes.

    from transformers import AutoModel, AutoTokenizer

tok = AutoTokenizer.from_pretrained('KB/albert-base-swedish-cased-alpha')
model = AutoModel.from_pretrained('KB/albert-base-swedish-cased-alpha')

  6. Use BERT base fine-tuned for Swedish NER

    master

    The bert-base-swedish-cased-ner model is fine-tuned on the SUC 3.0 dataset for Named Entity Recognition (NER).

    Entity Types:

    • TME: Time
    • PRS: Personal names
    • LOC: Locations
    • EVN: Events
    • ORG: Organisations

    Compatibility Note: For transformers versions earlier than 2.4.1, you must load the tokenizer separately to prevent the model from lower-casing input strings.

    from transformers import pipeline, TFBertForTokenClassification

# Using Huggingface pipeline (PyTorch default)
nlp = pipeline('ner', model='KB/bert-base-swedish-cased-ner', tokenizer='KB/bert-base-swedish-cased-ner')
nlp('Kalle och Pelle startar firman Kalle och Pelle.')

# Specifically using Tensorflow
tf = TFBertForTokenClassification.from_pretrained('KB/bert-base-swedish-cased-ner')
nlp = pipeline('ner', model=tf, tokenizer='KB/bert-base-swedish-cased-ner')

  7. Reconstruct words from BERT subword tokens in NER

    master

    The BERT tokenizer often splits words into multiple subword tokens starting with ##. When performing NER, you may want to glue these subparts back together to get the full word.

    This pattern iterates through the pipeline results, checks if a token starts with ##, and appends it to the previous token's text.

    text = 'Engelbert tar sin Rolls-Royce till Tele2 Arena för att titta på Djurgården IF ' +\\
       'som spelar fotboll i VM klockan två på kvällen.'

nlp = pipeline('ner', model='KB/bert-base-swedish-cased-ner', tokenizer='KB/bert-base-swedish-cased-ner', ignore_labels=[])
l = []
t = nlp(text)
in_word = False

for i, token in enumerate(t):
    if token['entity'] == 'O':
        in_word = False
        continue

    if token['word'].startswith('##'):
        # deal with (one level of) orphaned ##-tokens
        if not in_word:
            l += [ t[i-1] ]
            l[-1]['entity'] = token['entity']
        
        l[-1]['word'] += token['word'][2:]
    else:
        l += [ token ]

    in_word = True

print(l)

  8. Cite the Swedish BERT work

    master

    If you use these models in your research, please cite the work using the following BibTeX entry:

    @misc{swedish-bert,
    title={Playing with Words at the National Library of Sweden -- Making a Swedish BERT},
    author={Martin Malmsten and Love Börjeson and Chris Haffenden},
    year={2020},
    eprint={2007.01658},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}