ShareGPT4V

To evaluate MMBench:

1. For English (EN): Download mmbench_dev_20230712.tsv and place it in ./playground/data/eval/mmbench.
2. For Chinese (CN): Download mmbench_dev_cn_20231003.tsv and place it in ./playground/data/eval/mmbench.
3. Run multi-GPU inference using the respective script (mmbench_en.sh or mmbench_cn.sh).
4. Submit results from the answers_upload subfolder to the OpenCompass leaderboard.

# MMBench-EN Single node
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/sharegpt4v/eval/mmbench_en.sh

# MMBench-CN Single node
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/sharegpt4v/eval/mmbench_cn.sh

To evaluate Q-Bench:

1. Download llvisionqa_dev.json (for dev) or llvisionqa_test.json (for test) and place them in ./playground/data/eval/qbench.
2. Download and extract images into ./playground/data/eval/qbench/images_llviqionqa.
3. Run single-GPU inference using the provided shell script. Note: Local machine support is currently limited to dev evaluation only.

# Single node inference
CUDA_VISIBLE_DEVICES=0 bash scripts/sharegpt4v/eval/qbench.sh

# Slurm inference
srun -p Your_partition --gres gpu:1 bash scripts/sharegpt4v/eval/qbench.sh

To install ShareGPT4V, clone the InternLM-XComposer repository, create a Conda environment, and install the package in editable mode along with training dependencies and flash-attn.

git clone https://github.com/InternLM/InternLM-XComposer --depth=1
cd projects/ShareGPT4V
conda create -n share4v python=3.10 -y
conda activate share4v

pip install --upgrade pip
pip install -e .
pip install -e ".[train]"
pip install flash-attn --no-build-isolation

To evaluate VQAv2:

1. Download test2015 and place it in ./playground/data/eval/vqav2.
2. Run inference using the provided shell script (supports single-node or Slurm).
3. Submit results found in ./playground/data/eval/vqav2/answers_upload to the evaluation server.

# Single node inference
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/sharegpt4v/eval/vqav2.sh

# Slurm inference
srun -p Your_partition --gres gpu:8 bash scripts/sharegpt4v/eval/vqav2.sh

To evaluate TextVQA:

1. Download TextVQA_0.5.1_val.json and the images, then extract them to ./playground/data/eval/textvqa.
2. Run single-GPU inference and evaluation using the provided shell script.

# Single node inference
CUDA_VISIBLE_DEVICES=0 bash scripts/sharegpt4v/eval/textvqa.sh

# Slurm inference
srun -p Your_partition --gres gpu:1 bash scripts/sharegpt4v/eval/textvqa.sh

Training consists of two stages:

1. Feature Alignment Stage: Fine-tune the vision encoder, projector, and LLM using the ShareGPT4V-PT dataset to align textual and visual modalities.
2. Visual Instruction Tuning Stage: Fine-tune the projector and LLM to improve instruction following.

Pretraining

Requires a pretrained MLP projector (e.g., from LLaVA-1.5). Run the Slurm script: projects/ShareGPT4V/scripts/sharegpt4v/slurm_pretrain_7b.sh.

Finetuning

Fine-tune the projector and LLM using the mixed dataset. Run the Slurm script: projects/ShareGPT4V/scripts/sharegpt4v/slurm_finetune_7b.sh.

Scaling Training

To train on fewer GPUs, reduce per_device_train_batch_size and increase gradient_accumulation_steps. Ensure the global batch size remains constant: per_device_train_batch_size * gradient_accumulation_steps * num_gpus.

To evaluate SEED-Bench-Image:

1. Download images following official instructions and place them under ./playground/data/eval/seed_bench/SEED-Bench-image.
2. Run multi-GPU inference and evaluation using the provided shell script.
3. Optionally, submit results from ./playground/data/eval/seed_bench/answers_upload to the leaderboard using the official Jupyter notebook.

# Single node inference
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/sharegpt4v/eval/seed.sh

# Slurm inference
srun -p Your_partition --gres gpu:8 bash scripts/sharegpt4v/eval/seed.sh

To evaluate GQA:

1. Download GQA data and evaluation scripts following official instructions and place them in ./playground/data/eval/gqa/data. Note: You may need to modify eval.py as suggested in this gist due to missing assets in GQA v1.2.
2. Run inference using the provided shell script.

# Single node inference
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/sharegpt4v/eval/gqa.sh

# Slurm inference
srun -p Your_partition --gres gpu:8 bash scripts/sharegpt4v/eval/gqa.sh

To launch the local web application demo, run the app.py script located in the tools directory.

python tools/app.py

To evaluate MM-Vet:

1. Extract mm-vet.zip to ./playground/data/eval/mmvet.
2. Run single-GPU inference using the provided shell script.
3. Evaluate predictions in ./playground/data/eval/mmvet/results using the official Jupyter notebook.

# Single node inference
CUDA_VISIBLE_DEVICES=0 bash scripts/sharegpt4v/eval/mmvet.sh

# Slurm inference
srun -p Your_partition --gres gpu:1 bash scripts/sharegpt4v/eval/mmvet.sh