DOCSEARCHbyAlgolia
Submit Docs

FusionBench: A Comprehensive Benchmark of Deep Model Fusion

repository·main·Indexed 42 days ago

https://github.com/tanganke/fusion_bench

A benchmark suite and toolkit for evaluating deep model fusion techniques across diverse datasets and tasks. Version 0.2.33 features a modular configuration system powered by Hydra, supporting the definition of fusion algorithms, model pools (including CLIP-ViT and OpenCLIP), and task pools. It provides a CLI for running experiments and a Python API for implementing custom fusion strategies by extending the BaseAlgorithm class.

Tokens
153.6K
Snippets
403
Records
640
Agent score
51%

What's inside FusionBench

  1. Overview of FusionBench

    main
    FusionBench is a comprehensive benchmark suite and toolkit designed to evaluate the performance of various deep model fusion techniques. It provides a framework for comparing different fusion methods across a wide variety of datasets and tasks.

  2. Explore the Deep Model Fusion Reading List

    main

    The docs/readinglist/README.md file provides a curated collection of research papers and resources for studying deep model fusion. The list is categorized into several key research areas:

    • Survey Papers: High-level overviews of model merging, MoE (Mixture of Experts), and learning from models beyond fine-tuning.
    • Findings on Model Fusion: Research regarding scaling and multi-task learning optimization.
    • Model Ensemble: Techniques for ensembling and knowledge fusion (e.g., FuseChat).
    • Model Merging:
      • Mode Connectivity: Concepts regarding the connectivity of model modes.
      • Weight Interpolation: Methods like Model Soups, Task Arithmetic, and LoRA composition.
      • Alignment-based Methods: Merging models from different tasks or initializations (e.g., Git Re-Basin, ZipIt!).
      • Subspace-based Methods: Using subspace learning to eliminate interference.
      • Online Model Merging: Mitigating catastrophic forgetting during transfer.
    • Model Mixing/Upscaling/Expansion: Techniques for scaling LLMs via depth up-scaling or sparse mixture of experts (e.g., SMILE, SOLAR).
    • Benchmarks: Evaluation frameworks like FusionBench.

    Symbol Legend

    • :star: Highly recommended
    • :llama: LLaMA or Mistral model-related work
    • :simple-github: Code available on GitHub
    • :hugging: Models or datasets available on Hugging Face

  3. FusionBench Module Overview

    main

    FusionBench is organized into several functional modules:

    Core Components

    • fusion_bench.method: Fusion algorithms (ensemble, merging, mixing, compression).
    • fusion_bench.modelpool: Model management and loading.
    • fusion_bench.taskpool: Evaluation tasks and benchmarks.
    • fusion_bench.mixins: Reusable functionality.
    • fusion_bench.programs: Execution frameworks.

    Supporting Modules

    • fusion_bench.utils: Logging, caching, and data processing.
    • fusion_bench.models: Model wrapper definitions.
    • fusion_bench.dataset: Dataset definitions and loaders.
    • fusion_bench.metrics: Evaluation metrics.
    • fusion_bench.tasks: Task base classes.
    • fusion_bench.constants: Constant definitions.
    • fusion_bench.optim: Optimizers and learning rate schedulers.

  4. RegMean++ Files Structure

    main

    The regmean_plusplus example directory is organized as follows:

    • README.md: Documentation for the RegMean++ example.
    • clip_vit.sh: The execution script used to run RegMean++ on CLIP models.
    • results/: A directory containing pre-run experimental results for various CLIP architectures (clip-vit-base-patch16, clip-vit-base-patch32, and clip-vit-large-patch14).

  5. Understand the Fusion Module architecture

    main

    The Fusion Module is organized into several functional components that manage the lifecycle of deep model fusion, from data loading to evaluation:

    • method: Implements algorithms for model training, evaluation, ensemble methods, and model recombination.
    • modelpool: Manages collections of models (e.g., sequence-to-sequence, CLIP, GPT-2, or NYU Depth V2 specific models).
    • taskpool: Manages task definitions (e.g., CLIP image classification, FLAN-T5 text generation).
    • models: Provides model definitions, parameter management, and I/O handling.
    • tasks: Defines specific task implementations like classification and text generation.
    • dataset: Contains loaders and preprocessors for images, text, and specialized datasets like NYU Depth V2.
    • metrics: Provides evaluation metrics for various tasks (e.g., text-to-image generation).
    • optim: Contains optimization algorithms used for training.
    • mixins: Provides extension classes to add functionality to existing classes.
    • constants: Centralized configuration and constant values.
    • scripts: CLI and utility scripts for running training and management tasks.
    • utils: General utilities for device management, logging, timing, and data handling.

  6. Explore SMILE Upscaling experimental results

    main
    The examples/smile_upscaling/ directory contains pre-run experimental results and Jupyter notebooks demonstrating the SMILE (Sparse MIxture of Low-rank Experts) upscaling method across various model architectures and tasks. You can use these notebooks to study how upscaling performs on CLIP-ViT, Flan-T5, and Mistral-7B models, as well as ablation studies and projection merging experiments.

  7. Available Flan-T5 model variants

    main

    The benchmark provides several versions of Flan-T5 models fine-tuned on GLUE tasks, categorized by model size and fine-tuning method:

    Flan-T5-base

    • Full Fine-tuned: Models fully fine-tuned on specific GLUE tasks.
    • LoRA Fine-tuned (r=16): Models using Low-Rank Adaptation (LoRA) with rank $r=16$.

    Flan-T5-Large

    • LoRA Fine-tuned (r=16): Larger parameter models using LoRA with rank $r=16$.

    These models are available via Hugging Face collections linked in the documentation.

  9. Explore fusion_bench.utils utility modules

    main

    The fusion_bench.utils module is a collection of utility functions and classes designed to support model fusion workflows. It provides specialized tools for the following domains:

    • Logging: Managing system and process logs.
    • Profiling: Analyzing performance and resource usage.
    • Caching: Optimizing workflows through data/result caching.
    • Data: Handling data processing tasks.
    • Model: Managing model lifecycles and configurations.
    • Pytorch: Utilities specifically for PyTorch-based operations.
    • Filesystem: Interacting with the local file system.
    • ModelScope: Integration with ModelScope.
    • Package Management: Handling dependencies and environment packages.

  10. Intermediate FusionBench Use Cases

    main

    FusionBench supports advanced workflows beyond basic execution, including custom algorithm implementation, performance profiling, and experiment tracking via various logging backends.

    Key advanced capabilities include:

    • Customizing Algorithms: Implementing and integrating your own fusion logic.
    • Assessing Model Performance: Using profiling tools to monitor model performance during algorithm execution.
    • Selecting Loggers: Configuring experiment tracking with supported backends like TensorBoard, Wandb, MLFlow, CSV, or SwanLab.

  11. What is Model Stock merging?

    main

    Model Stock is a parameter-space merging method that uses the geometric relationship between fine-tuned models and a pretrained anchor to compute interpolation ratios.

    Instead of using fixed weights, it calculates an interpolation ratio $t$ for each parameter group based on the angle between fine-tuned models' displacement vectors relative to the pretrained model $\theta_0$.

    Core Logic:

    • If fine-tuned models move in similar directions (small angle), the merge relies heavily on the fine-tuned models.
    • If fine-tuned models diverge (large angle), the pretrained anchor $\theta_0$ acts as a stabilizer by increasing its contribution to the merge.

    The merged model is calculated as: $$\theta_{merged} = t \cdot \bar{\theta}{ft} + (1 - t) \cdot \theta_0$$ where $\bar{\theta}{ft}$ is the average of the fine-tuned models.

  12. What is DoP (Dual Projections)?

    main

    DoP (Dual Projections for Balancing Stability and Plasticity) is a continual model merging algorithm designed to merge fine-tuned models sequentially without requiring access to the original training data.

    It works by using Singular Value Decomposition (SVD) to define low-rank projection subspaces for each task vector (the difference between a fine-tuned model and the pretrained model). By projecting weight differences onto these subspaces, DoP balances:

    • Stability: Preserving knowledge from previously merged models.
    • Plasticity: Incorporating knowledge from new models.

    The algorithm processes models one at a time, making it suitable for scenarios where models arrive sequentially and cannot all be held in memory simultaneously.