LLaVA-Med Documentation

LLaVA-Med is a large language and vision model designed for biomedical research.

Intended Use

• Research Only: The model, code, and data are intended solely for future research on visual-language processing and for reproducing experimental results from the reference paper.
• Not for Clinical Use: The model is not intended for clinical care, clinical decision-making, or any deployed use cases (commercial or otherwise).

Limitations

• Language: Developed using English corpora and may be considered English-only.
• Domain Scope: Evaluated on a narrow set of biomedical benchmark tasks.
• Accuracy: May make inaccurate predictions and inherits limitations from the base LLaVA model.
• Data Bias: Built using the PMC-15M dataset, which may contain biases reflecting academic publication practices (e.g., enrichment for positive findings or extreme cases).

LLaVA-Med-1.0 performance is evaluated across several dimensions:

1. Medical Visual Chat: Compared against a language-only GPT-4 baseline. In this baseline, GPT-4 is provided with golden captions and inline mentions as context, representing the performance upper bound because it does not need to process the raw image.
2. Medical VQA: Performance is measured by fine-tuning LLaVA-Med on established Medical Question Answering (QVA) datasets.

Note that standard LLaVA models tend to hallucinate or refuse to provide domain-specific knowledgeable responses in medical contexts, which LLaVA-Med is designed to address.

To use the LLaVA-Med Web UI, you must launch three components in order: a controller, a model worker, and finally the Gradio web server.

If your GPU VRAM is $\le$ 24GB (e.g., RTX 3090, RTX 4090), you can use the --num-gpus flag to distribute the model across multiple GPUs.

To install LLaVA-Med, clone the repository, navigate to the folder, and set up a Conda environment with Python 3.10. Ensure you upgrade pip to enable PEP 660 support before installing the package in editable mode.

https://github.com/microsoft/LLaVA-Med.git
cd LLaVA-Med

conda create -n llava-med python=3.10 -y
conda activate llava-med
pip install --upgrade pip  # enable PEP 660 support
pip install -e .

To download the language-image multimodal instruction-following dataset, run the provided shell script. You can also manually download specific files such as alignment data, instruction-tuning data, or evaluation files from the repository.

sh download_data.sh

If you use LLaVA-Med in your research, please cite the following BibTeX entry:

@article{li2023llavamed,
  title={Llava-med: Training a large language-and-vision assistant for biomedicine in one day},
  author={Li, Chunyuan and Wong, Cliff and Zhang, Sheng and Usuyama, Naoto and Liu, Haotian and Yang, Jianwei and Naumann, Tristan and Poon, Hoifung and Gao, Jianfeng},
  journal={arXiv preprint arXiv:2306.00890},
  year={2023}
}