DOCSEARCHbyAlgolia
Submit Docs

BAGEL Multimodal Foundation Model

repository·main·Indexed 27 days ago

https://github.com/bytedance-seed/bagel

BAGEL is an open-source multimodal foundation model with 7B active parameters (14B total) designed for unified multimodal understanding and generation. It supports text-to-image generation, image editing, and complex world-modeling tasks such as multiview synthesis and world navigation. The model includes a 'thinking' process for intermediate reasoning and provides a Gradio WebUI with various quantization modes (NF4, INT8) for different VRAM capacities.

Tokens
9.5K
Snippets
16
Records
31
Agent score
90%

What's inside BAGEL

  2. Evaluate GEdit-Bench

    main

    To evaluate GEdit-Bench, modify the model path, output path, API key, and API URL in scripts/eval/run_gedit.sh, then execute the script. The evaluation uses gpt-4.1-2025-04-14.

    bash script/eval/run_gedit.sh

  3. Prepare sample datasets for BAGEL

    main

    To set up the environment with sample data for T2I (text-to-image), Editing, and VLM (vision-language model) tasks, follow these steps:

    1. Download and unzip the sample dataset:

      wget -O bagel_example.zip https://lf3-static.bytednsdoc.com/obj/eden-cn/nuhojubrps/bagel_example.zip
unzip bagel_example.zip -d /data

    2. Verify the directory hierarchy: The expected structure is:

      • t2i/: text-to-image (parquet format)
      • editing/: image editing (parquet format)
      • vlm/: contains images/ (JPEG/PNG) and llava_ov_si.jsonl (vision-language SFT conversations)

    3. Configure paths: Edit every your_data_path placeholder in data/dataset_info.py to point to your actual data locations.

    4. Extend data (Optional): You can extend DATASET_INFO in data/dataset_info.py with your own parquet shards or JSONL files to mix additional data into training.

    wget -O bagel_example.zip \
  https://lf3-static.bytednsdoc.com/obj/eden-cn/nuhojubrps/bagel_example.zip
unzip bagel_example.zip -d /data

  4. Evaluate VLM benchmarks (MME, MMBench, MMMU, etc.)

    main

    To evaluate Vision-Language Model (VLM) performance on benchmarks like MME, MMBench, MMMU, MMVet, MathVista, and MMVP, use the scripts/eval/run_eval_vlm.sh script.

    Requirements:

    • Data Preparation: Follow InternVL2 instructions to prepare data. The data must be organized under a vlm directory with the following structure:
      data/vlm/
├── MathVista
├── mmbench
├── mme
├── MMMU
├── mm-vet
└── MMVP
    • MathVista Configuration: You must set $openai_api_key in scripts/eval/run_eval_vlm.sh and your_api_url in eval/vlm/eval/mathvista/utilities.py. The default model is gpt-4o-2024-11-20.
    • MMBench/MMVet: Use their respective official evaluation servers for submission.

    Execution: Set the following environment variables before running the script:

    • $model_path: Path to the model checkpoint.
    • $output_path: Path where logs and results will be saved.
    • $GPUS: Number of GPUs to use (increase for faster execution).
    • $openai_api_key: Required for MathVista.

  5. Run BAGEL via Gradio WebUI

    main

    Launch the BAGEL interface using app.py. Choose the command based on your GPU VRAM capacity:

    • 32GB+ VRAM or multi-GPU: Standard mode.
    • 12GB–32GB VRAM: Use NF4 quantization and Chinese interface (--mode 2 --zh).
    • 22GB–32GB VRAM: Use INT8 quantization (--mode 3). Note: INT8 is not recommended for the 22-32GB range.
    # For 32GB+ VRAM GPU or multi GPUs.
python app.py

# For 12~32GB VRAM GPU, recommend using NF4 quantization. And use Chinese interface.
python app.py --mode 2 --zh

# For 22~32GB VRAM GPU, not recommended to use INT8 quantization.
python app.py  --mode 3

  7. Install BAGEL environment

    main

    To set up the BAGEL environment, clone the repository, create a Conda environment with Python 3.10, install the required dependencies, and install flash_attn with no build isolation.

    git clone https://github.com/bytedance-seed/BAGEL.git
cd BAGEL
conda create -n bagel python=3.10 -y
conda activate bagel
pip install -r requirements.txt
pip install flash_attn==2.5.8 --no-build-isolation

  8. Evaluate ImgEdit-Bench

    main

    To evaluate ImgEdit-Bench, follow these steps for data preparation and execution.

    1. Data Preparation

    Download the benchmark data from ImgEdit-Bench and extract it into a Benchmark directory. The structure must be:

    Benchmark
├── hard
├── multiturn
└── singleturn
    ├── judge_prompt.json
    ├── singleturn.json
    ├── animal
    ├── architecture
    ├── clothes
    ├── compose
    ├── daily object
    ├── for_add
    ├── human
    ├── style
    └── transport

    2. Execution

    Run scripts/eval/run_imgedit.sh.

    Required Settings:

    • $model_path: Path to the model checkpoint.
    • $output_path: Path for logs.
    • $openai_api_key: Set in scripts/eval/run_imgedit.sh.
    • your_api_url: Set in eval/gen/imgedit/basic_bench.py.

    Default Configuration:

    • GPT Version: gpt-4o-2024-11-20
    • cfg_text_scale: 4
    • cfg_img_scale: 1.5
    • cfg_renorm_min: 0 (for CFG Renorm)

  9. Setup and evaluate RISEBench

    main

    To evaluate RISEBench, download the benchmark data from RISEBench and place it in the data directory.

    Required Directory Structure:

    data/
├── datav2_total_w_subtask.json
├── causal_reasoning_images
├── logical_reasoning_images
├── spatial_reasoning_images
└── temporal_reasoning_images

  10. Setup and evaluate GenEval

    main

    GenEval evaluation requires specific dependencies and a detector model.

    1. Install Dependencies:

    pip install open-clip-torch
pip install clip-benchmark
pip install --upgrade setuptools
sudo pip install -U openmim
sudo mim install mmengine mmcv-full==1.7.2

git clone https://github.com/open-mmlab/mmdetection.git
cd mmdetection; git checkout 2.x
pip install -v -e .

    2. Download Detector:

    cd ./eval/gen/geneval
mkdir model
bash ./evaluation/download_models.sh ./model

    3. Run Evaluation: Execute scripts/eval/run_geneval.sh.

    • Set $model_path and $output_path.
    • To use original GenEval prompts, set metadata_file to ./eval/gen/geneval/prompts/evaluation_metadata.jsonl.

  11. Evaluate WISE benchmark

    main

    To evaluate the WISE benchmark, run scripts/eval/run_wise.sh.

    Configuration:

    • Set $model_path and $output_path.
    • Set $openai_api_key in scripts/eval/run_wise.sh.
    • Set your_api_url in eval/gen/wise/gpt_eval_mp.py.
    • The default GPT version is gpt-4o-2024-05-13.
    • Use the think flag to enable thinking mode.

  12. Evaluate RISEBench

    main

    To evaluate RISEBench, run the scripts/eval/run_rise.sh script. You must configure the model checkpoint path, the output log path, and the OpenAI API credentials.

    Required Environment Variables/Settings:

    • $model_path: Path to the model checkpoint.
    • $output_path: Path where evaluation logs will be saved.
    • $openai_api_key: Set this within scripts/eval/run_rise.sh.
    • your_api_url: Set this in eval/gen/rise/gpt_eval.py.

    Default Configuration:

    • GPT Version: gpt-4.1-2025-04-14
    • cfg_text_scale: 4
    • cfg_img_scale: 2.0
    • cfg_renorm_min: 0 (for CFG Renorm)

    To use thinking mode, include the think flag/option as specified in the script.