DOCSEARCHbyAlgolia
Submit Docs

LoRA (Low-Rank Adaptation)

repository·main·Indexed 12 days ago

https://github.com/microsoft/lora

A method for efficient fine-tuning of large language models that reduces trainable parameters by learning low-rank decomposition matrices while keeping original model weights frozen. Includes implementations and examples for fine-tuning GPT-2, RoBERTa, BERT, DistilBERT, and XLNet across tasks such as Causal Language Modeling (CLM), Masked Language Modeling (MLM), Permutation Language Modeling (PLM), multiple-choice (SWAG), and question-answering (SQuAD).

Tokens
142.3K
Snippets
346
Records
610
Agent score
95%

What's inside LoRA

  1. Overview of FlauBERT models

    main
    FlauBERT is a transformer-based model pretrained using a masked language modeling (MLM) objective, specifically designed for the French language. It is based on the research presented in 'FlauBERT: Unsupervised Language Model Pre-training for French'. The models can be used for various NLP tasks including text classification, paraphrasing, natural language inference, parsing, and word sense disambiguation.

  2. Overview of the BORT model

    main
    BORT is an optimal subset of architectural parameters extracted from the BERT architecture using neural architecture search. It is significantly smaller than the original BERT-large, with an effective size of 5.5% and a net size of 16%. BORT is designed for high efficiency, offering faster pretraining and CPU inference compared to standard BERT variants while maintaining or improving performance on NLU benchmarks.

  3. Overview of Transformers for NLU and NLG

    main

    The Transformers library provides state-of-the-art Natural Language Processing (NLP) architectures for Natural Language Understanding (NLU) and Natural Language Generation (NLG). It supports a wide range of architectures including BERT, GPT-2, RoBERTa, XLM, DistilBert, and XLNet.

    Key capabilities include:

    • Support for over 32+ pretrained models in 100+ languages.
    • Deep interoperability between TensorFlow 2.0 and PyTorch, allowing users to move models between frameworks.
    • Integration with the Hugging Face Model Hub for seamless access to model checkpoints.

  4. Overview of I-BERT

    main
    I-BERT is an integer-only quantized version of RoBERTa designed for efficient inference. It replaces floating-point arithmetic with integer-only arithmetic for the entire inference process, including nonlinear operations like GELU, Softmax, and Layer Normalization. This allows for efficient utilization of integer-only logical units (such as Turing Tensor Cores or ARM processors). On a T4 GPU system, I-BERT can achieve a 2.4x to 4.0x speedup for INT8 inference compared to FP32 inference while maintaining accuracy similar to or slightly higher than the full-precision baseline.

  5. Overview of DeBERTa-v2

    main

    DeBERTa-v2 (Decoding-enhanced BERT with disentangled attention) is an evolution of the BERT and RoBERTa models. It introduces two primary improvements: a disentangled attention mechanism (representing words using separate content and position vectors) and an enhanced mask decoder.

    Key features of v2 include:

    • SentencePiece Tokenizer: Uses a 128K vocabulary built from training data.
    • nGiE (nGram Induced Input Encoding): Adds a convolution layer alongside the first transformer layer to capture local token dependencies.
    • Parameter Efficiency: Shares the position projection matrix with the content projection matrix in the attention layer.
    • Relative Position Encoding: Uses log buckets to encode relative positions (similar to T5).
    • Scalability: Available in various sizes, including 900M and 1.5B parameter versions.

  7. Overview of SqueezeBERT

    main

    SqueezeBERT is a bidirectional transformer architecture designed for efficiency, similar to BERT. It achieves speedups by using grouped convolutions instead of fully-connected layers for the Query (Q), Key (K), Value (V), and Feed-Forward Network (FFN) layers. This makes it significantly faster on mobile devices compared to BERT-base while maintaining competitive accuracy on the GLUE test set.

    Key Usage Tips:

    • Padding: Because SqueezeBERT uses absolute position embeddings, it is recommended to pad inputs on the right rather than the left.
    • Task Suitability: It is optimized for Masked Language Modeling (MLM) and Natural Language Understanding (NLU) tasks. It is not optimal for text generation (causal language modeling is preferred for that).
    • Fine-tuning: For sequence classification tasks, it is recommended to use the squeezebert/squeezebert-mnli-headless checkpoint for best results.

  8. Overview of XLM-ProphetNet

    main
    XLM-ProphetNet is an encoder-decoder model designed for multi-lingual sequence-to-sequence tasks. It is based on the ProphetNet architecture, which utilizes a 'future n-gram prediction' objective. Unlike traditional models that predict only the next token, XLM-ProphetNet is optimized to predict the next $n$ tokens simultaneously, encouraging the model to plan for future tokens and preventing overfitting on local correlations. The XLM variant is specifically trained on the multi-lingual 'wiki100' Wikipedia dataset.

  9. Overview of XLM models

    main

    XLM (Cross-lingual Language Model) is a transformer-based model designed for multilingual tasks. It can be pretrained using three different objectives:

    • Causal Language Modeling (CLM): Next token prediction (generative).
    • Masked Language Modeling (MLM): BERT-like objective.
    • Translation Language Modeling (TLM): An extension of MLM that uses multiple language inputs.

    Usage Tips

    • Select the correct checkpoint: Ensure the checkpoint's training objective matches your task. For example, MLM checkpoints are not suitable for text generation tasks.
    • Multilingual support: XLM includes multilingual checkpoints that utilize a specific lang parameter.

  10. Overview of XLM-RoBERTa

    main

    XLM-RoBERTa (XLM-R) is a large multilingual language model based on Facebook's RoBERTa. It is trained on 100 different languages using filtered CommonCrawl data.

    Key characteristics:

    • Multilingual Support: It supports 100 languages and does not require explicit lang tensors to identify the language; it determines the language from the input IDs.
    • Implementation: The implementation is identical to RoBERTa. For specific usage examples regarding inputs and outputs, refer to the RoBERTa documentation.

  11. Overview of DistilBERT

    main
    DistilBERT is a small, fast, and lightweight Transformer model trained via knowledge distillation from bert-base-uncased. It features 40% fewer parameters and runs approximately 60% faster while retaining over 95% of BERT's performance on GLUE benchmarks. It is designed for scenarios with constrained computational budgets or on-edge deployment.

  12. Use ConvBERT for Natural Language Understanding tasks

    main

    ConvBERT is an efficient variant of BERT that uses span-based dynamic convolution to model local dependencies, reducing memory footprint and computation cost compared to global self-attention. It is available for various NLU tasks via the transformers library.

    Available model architectures include:

    • Base Models: ConvBertModel (PyTorch) and TFConvBertModel (TensorFlow).
    • Masked Language Modeling: ConvBertForMaskedLM / TFConvBertForMaskedLM.
    • Sequence Classification: ConvBertForSequenceClassification / TFConvBertForSequenceClassification.
    • Multiple Choice: ConvBertForMultipleChoice / TFConvBertForMultipleChoice.
    • Token Classification: ConvBertForTokenClassification / TFConvBertForTokenClassification.
    • Question Answering: ConvBertForQuestionAnswering / TFConvBertForQuestionAnswering.

    Training tips for ConvBERT are generally similar to those used for BERT.