DOCSEARCHbyAlgolia
Submit Docs

AliceMind

repository·main·Indexed 24 days ago

https://github.com/alibaba/alicemind

A collection of encoder-decoder models and optimization techniques from Alibaba's MinD Lab. It includes ChatPLUG, a Chinese open-domain dialogue system for digital humans; Contrastive Pruning (CAP), a framework for compressing pre-trained language models using structured and unstructured pruning; LatticeBERT, which provides multi-granularity Chinese representations; and Child-Tuning.

Tokens
93.6K
Snippets
239
Records
374
Agent score
85%

What's inside AliceMind

  1. Overview of mPLUG-Owl

    main

    mPLUG-Owl is a modularized training paradigm for large multi-modal language models. It is designed to learn visual knowledge while supporting multi-turn conversations across different modalities. Key capabilities include:

    • Multi-image correlation
    • Scene text understanding
    • Vision-based document comprehension

    The project also includes OwlEval, a visually-related instruction evaluation set.

  2. Overview of the PLUG model

    main

    PLUG is a large-scale Chinese pre-training model designed for both language understanding and generation tasks. It features 27B parameters and has demonstrated strong performance in tasks like Question Generation (e.g., on the DuReader-Robust dataset).

    Key Specifications:

    • Model Size: 27B parameters
    • Fine-tuning Hardware Requirements: A single node with 8x 32G V100 GPUs.
    • Task Performance (DuReader-Robust): Achieves BLEU-1: 0.6310, BLEU-2: 0.5510, BLEU-4: 0.4223, and ROUGE-L: 0.6264.

    For detailed implementation details and model downloads, refer to the PLUG README.

  3. What is PLUG?

    main

    PLUG is a large-scale Chinese pre-training model designed for both language understanding and generation. It features 27 billion parameters and utilizes a two-stage architecture:

    1. A 24-layer StructBERT encoder.
    2. A 24-6-layer PALM encoder-decoder.

    It is capable of tasks such as novel generation, scientific literature generation, common sense Q&A, and zero-shot classification.

  4. Overview of ChatPLUG

    main
    ChatPLUG is a Chinese open-domain dialogue system designed for digital human applications. Unlike models that focus on scaling model size or pre-training corpora, ChatPLUG achieves diverse skills and multi-task generalization through internet-augmented instruction tuning. It is trained on a wide range of dialogue tasks using a unified internet-augmented format.

  5. Overview of AliceMind Pre-trained Models

    main

    AliceMind provides a collection of encoder-decoder models developed by Alibaba's MinD Lab. The models cover various modalities and tasks, including:

    • Multimodal (Vision-Language):

      • mPLUG-Owl2: Enhances LLMs/MLLMs through modal collaboration (CVPR 2024).
      • mPLUG-DocOwl: OCR-free multimodal model for universal document understanding (EMNLP 2023).
      • mPLUG-video: Chinese video large language model.
      • mPLUG-2: Modularized multi-modal foundation model across text, image, and video (ICML 2023).
      • mPLUG: Large-scale vision-language understanding and generation (EMNLP 2022).
      • StructVBERT: Vision-language understanding (CVPR 2020).

    • Language & Dialogue:

      • ChatPLUG: Chinese open-domain dialogue system for digital humans.
      • PLUG: Large-scale Chinese understanding and generation model (27B).
      • LatticeBERT: Chinese language understanding with multi-granularity inputs (NAACL 2021).
      • VECO: Cross-lingual (x-NLU and x-NLG) model (ACL 2021).
      • PALM: Generative language model (EMNLP 2020).
      • StructBERT: Natural language understanding (ICLR 2020).
      • StructuralLM: Document-image understanding (ACL 2021).

    • Specialized Tasks:

      • SDCUP: Pre-trained models for table understanding.

  6. Overview of mPLUG-2

    main

    mPLUG-2 is a modularized multi-modal foundation model designed for tasks involving text, image, and video. It uses a multi-module composition network that shares universal modules for modality collaboration while disentangling specific modality modules to prevent entanglement. This architecture allows for flexible selection of modules to perform various understanding and generation tasks across different modalities.

    Key capabilities include:

    • Multi-modal tasks: Image-text and video-text understanding and generation.
    • Uni-modal tasks: Text-only, image-only, and video-only understanding.
    • Performance: Achieves state-of-the-art results on tasks like MSRVTT video QA (48.0 top-1 accuracy) and video captioning (80.3 CIDEr) with efficient model size and data scale.

  7. Overview of Fairseq command-line tools

    main

    Fairseq provides a suite of command-line tools for the complete machine learning lifecycle, including data preparation, model training, inference, and evaluation:

    • fairseq-preprocess: Pre-processes data by building vocabularies and binarizing training data.
    • fairseq-train: Trains new models on single or multiple GPUs.
    • fairseq-generate: Performs translation/inference on pre-processed data using a trained model.
    • fairseq-interactive: Performs translation/inference on raw text using a trained model.
    • fairseq-score: Calculates BLEU scores for generated translations against reference translations.
    • fairseq-eval-lm: Evaluates language models.

  8. Overview of VECO: Variable Encoder-decoder Pre-training

    main

    VECO (Variable Encoder-decoder) is a pre-trained model designed for both cross-lingual understanding (NLU) and generation (NLG) tasks. It provides initialization for both encoder-only and encoder-decoder Transformer architectures using streamlined parameters.

    Key capabilities:

    • NLU: Achieves state-of-the-art results on the XTREME benchmark, including text classification, sequence labeling, question answering, and sentence retrieval.
    • NLG: Outperforms existing cross-lingual models on WMT14 English-to-German and English-to-French translation tasks.

  9. What is Child-Tuning and how to use it

    main

    Child-Tuning is a fine-tuning technique designed to improve the accuracy of large pre-trained models by updating only a subset of parameters (the "child network").

    How it works: During the backward pass, the technique strategically masks out the gradients of the non-child network. This allows users to treat the process similarly to dropout during back-propagation. It is particularly effective for small datasets and can improve accuracy by 1.5% to 8.6% compared to vanilla fine-tuning.

    Benefits:

    • Accuracy: Outperforms vanilla fine-tuning and prior techniques in GLUE benchmarks.
    • Generalization: Shows significant improvements in domain transfer and task transfer.
    • Ease of Use: Designed to be easily imported into existing codebases.

    Example Performance (cluewsc dataset):

    • child-tuning-D: 86.8% accuracy
    • child-tuning-F: 87.5% accuracy
    • Vanilla (No child-tuning): 86.18% accuracy

  10. What is Contrastive Pruning (CAP)?

    main

    Contrastive Pruning (CAP) is a general framework designed to maintain both task-specific and task-agnostic knowledge during the compression of pre-trained language models.

    Unlike traditional pruning that focuses only on downstream tasks, CAP uses contrastive learning to encourage the pruned model to learn from three distinct sources via three contrastive modules:

    • PrC: Encourages learning from the original pre-trained model.
    • SnC: Encourages learning from snapshots (intermediate models during the pruning process).
    • FiC: Encourages learning from the fine-tuned model.

  11. Overview of Parameter-Efficient Sparse Training (PST)

    main

    PST is a method designed to reduce the number of trainable parameters during sparse-aware training for large language models. It addresses the computational overhead and memory footprint challenges of sparse training by:

    1. Combining data-free and data-driven criteria to measure weight importance.
    2. Leveraging the low-rankness and structuredness of data-driven weight importance.
    3. Using two groups of small matrices to compute importance scores instead of large importance score matrices, making the process resource-efficient and parameter-efficient.

  12. How to use Tasks in fairseq

    main

    In fairseq.tasks, a Task is a central abstraction that manages dictionaries, handles dataset loading/iteration, and provides helpers for initializing models and calculating loss.

    To use a task, you typically follow this lifecycle:

    1. Setup: Initialize the task using setup_task(args).
    2. Initialization: Build the model and criterion using the task's build_model and build_criterion methods.
    3. Data Loading: Load datasets (e.g., 'train', 'valid') using load_dataset.
    4. Iteration: Create a batch iterator via get_batch_iterator.
    5. Training Loop: For each batch, compute the loss using get_loss(model, criterion, batch) and perform backpropagation.
    # setup the task (e.g., load dictionaries)
task = fairseq.tasks.setup_task(args)

# build model and criterion
model = task.build_model(args)
criterion = task.build_criterion(args)

# load datasets
task.load_dataset('train')
task.load_dataset('valid')

# iterate over mini-batches of data
batch_itr = task.get_batch_iterator(
    task.dataset('train'), max_tokens=4096,
)
for batch in batch_itr:
    # compute the loss
    loss, sample_size, logging_output = task.get_loss(
        model, criterion, batch,
    )
    loss.backward()