MiniGPT-4 and MiniGPT-v2

By default, LLMs are loaded in 8-bit with a beam search width of 1 to save memory. This requires approximately:

• 13B LLM: ~23G GPU memory
• 7B LLM: ~11.5G GPU memory

To run the model in 16-bit (requires more powerful GPUs), set low_resource to False in the following config files:

• MiniGPT-v2: eval_configs/minigptv2_eval.yaml
• MiniGPT-4 (Llama2): eval_configs/minigpt4_llama2_eval.yaml
• MiniGPT-4 (Vicuna): eval_configs/minigpt4_eval.yaml

After completing the preparation steps, your dataset directory should follow this structure:

. 
├── ${MINIGPT4_DATASET}
│   ├── cc_sbu
│   │   ├── convert_cc_sbu.py
│   │   ├── download_cc_sbu.sh
│   │   ├── ccs_synthetic_filtered_large.json
│   │   ├── ccs_synthetic_filtered_large.tsv
│   │   └── cc_sbu_dataset
│   │       ├── 00000.tar
│   │       ├── 00000.parquet
│   │       └── ...
│   ├── laion
│   │   ├── convert_laion.py
│   │   ├── download_laion.sh
│   │   ├── laion_synthetic_filtered_large.json
│   │   ├── laion_synthetic_filtered_large.tsv
│   │   └── laion_dataset
│   │       ├── 00000.tar
│   │       ├── 00000.parquet
│   │       └── ...

To use the VQA Python API and evaluation code, ensure you have the following dependencies installed:

• python 2.7
• scikit-image
• matplotlib

To prepare for pre-training, you must first download the filtered synthetic caption JSON files prepared by BLIP. Note that storing the full LAION and CC3M+CC12M+SBU datasets requires approximately 2.3TB of storage.

Annotation Files:

• CC3M+CC12M+SBU: ccs_synthetic_filtered_large.json
• LAION115M: laion_synthetic_filtered_large.json

The repository provides two main toolsets for interacting with the VQA dataset:

Data Access and Visualization

Located in ./PythonHelperTools, this directory contains the API to read and visualize the VQA dataset.

• vqaTools: The core API for data reading and visualization.
• vqaDemo.py: A demo script to show how to use the API.

Evaluation

Located in ./PythonEvaluationTools, this directory contains the code for evaluating model performance.

• vqaEvaluation: The core evaluation logic.
• vqaEvalDemo.py: A demo script for running evaluations.

To run a demo evaluation, you can use the example results file provided in ./Results/OpenEnded_mscoco_train2014_fake_results.json.

To set up the environment, clone the repository, create a Conda environment from the provided environment.yml, and activate it.

git clone https://github.com/Vision-CAIR/MiniGPT-4.git
cd MiniGPT-4
conda env create -f environment.yml
conda activate minigptv

The VQA tools expect a specific directory structure for questions, annotations, and images. Follow these steps to organize your data:

1. Questions and Annotations

• Place question files in ./Questions.
• Place annotation files in ./Annotations.
• Download files from the VQA download page (v2.0) or VQA v1 download page (v1.0).

2. Images

Depending on the dataset type, organize images as follows:

For Real (MS COCO) datasets: Create a directory structure: ./Images/mscoco/[train2014 | val2014 | test2015].

For Abstract datasets: Create a directory structure: ./Images/abstract_v002/[train2015 | val2015 | test2015].

Note: Images must be downloaded from the MS COCO website or the VQA download page as specified for each version.

Add the MiniGPT-4 root directory to your PYTHONPATH to ensure the evaluation scripts can resolve necessary modules.

export PYTHONPATH=$PYTHONPATH:/path/to/directory/of/MiniGPT-4

To finetune MiniGPT-v2, you need to download various datasets including COCO, Visual Genome, TextCaps, RefCOCO, and others. Most datasets require both an image source and an annotation file (JSON).

Common image sources used across multiple datasets include:

• COCO 2014 images (used for COCO captions, COCO VQA, RefCOCO series, OKVQA, AOK-VQA, Multi-task conversation, and LLaVA).
• Visual Genome images.
• GQA images.
• TextCaps images.
• Flickr-30k images.

To download AOK-VQA annotations, use the following commands:

export AOKVQA_DIR=YOUR_DATASET_PATH
mkdir -p ${AOKVQA_DIR}
curl -fsSL https://prior-datasets.s3.us-east-2.amazonaws.com/aokvqa/aokvqa_v1p0.tar.gz | tar xvz -C ${AOKVQA_DIR}

Expected structure:

${MINIGPTv2_DATASET}
└── aokvqa
    └── aokvqa_v1p0_train.json

In minigpt4/configs/datasets/aokvqa/defaults.yaml, set:

• image_path to the COCO 2014 image folder.
• ann_path to the location of the AOKVQA dataset.

To evaluate MiniGPT-v2, you must download images and annotations for the target datasets. The datasets supported include OKVQA, GQA, Hateful Meme, IconQA, VizWiz, RefCOCO, RefCOCO+, and RefCOCOg.

Ensure your directory structure follows the required pattern for the evaluation scripts to locate files correctly. For example:

• GQA: Requires test_balanced_questions.json, testdev_balanced_questions.json, and a gqa_images folder.
• OKVQA: Requires okvqa_test_split.json, mscoco_val2014_annotations_clean.json, and OpenEnded_mscoco_val2014_questions_clean.json.
• RefCOCO/+/g: Requires instances.json and specific .p annotation files (e.g., refs(google).p or refs(unc).p).