MedRAX

You can start the production-ready Gradio interface by running main.py.

Important Configuration:

• Set the model_dir inside main.py to the directory where you want to store or access model weights.
• Comment out any tools you do not have access to.
• Ensure your OpenAI API key is configured in a .env file.

If you encounter permission issues, use the following command:

# Start the Gradio interface
python main.py

# If you run into permission issues
sudo -E env "PATH=$PATH" python main.py

ChestAgentBench is an evaluation framework for CXR interpretation. To use it, download the dataset from Hugging Face and unzip the figures into your MedMAX directory.

To evaluate using GPT-4o, set your OPENAI_API_KEY and run the quickstart.py script.

# Download the benchmark
huggingface-cli download wanglab/chestagentbench --repo-type dataset --local-dir chestagentbench

# Unzip figures
unzip chestagentbench/figures.zip

# Run evaluation with GPT-4o
export OPENAI_API_KEY="<your-openai-api-key>"
python quickstart.py \
    --model chatgpt-4o-latest \
    --temperature 0.2 \
    --max-cases 2 \
    --log-prefix chatgpt-4o-latest \
    --use-urls

CheXbench requires local dataset files. Use chexbench_gpt4.py as a reference for running models.

1. Download Datasets

Download the following and place them in the specified directories:

• SLAKE: Save to MedMAX/data/slake.
• Rad-ReStruct: Save images to MedMAX/data/rad-restruct/images.
• Open-I: Save images to MedMAX/data/openi/images. (Note: Rad-ReStruct and Open-I use the same images; you can download NLMCXR.zip once and copy it to both.)

2. Configure Paths

After downloading, use the provided script to update chexbench.json with your local file paths:

python MedMax/data/fix_chexbench.py

Use compare_runs.py to evaluate model performance based on accuracy and various axes.

Single File Analysis

Analyze a single result file:

python compare_runs.py results/medmax.json

Direct Model Comparison

Compare two models on the exact same questions:

python compare_runs.py results/medmax.json results/gpt4o.json

Multi-Model Comparison

Compare all models by providing multiple log files:

python compare_runs.py results/medmax.json results/gpt4o.json results/llama.json results/chexagent.json results/llavamed.json

ChestAgentBench is a benchmark containing over 2,500 complex medical queries across 8 categories. You can run specific models by executing their corresponding scripts within the experiments directory.

Model Scripts

• GPT-4o: python benchmark_gpt4o.py
• Llama 3.2 Vision 90B: python benchmark_llama.py
• CheXagent: python benchmark_chexagent.py
• LLaVA-Med: Requires cloning the LLaVA-Med repository and following their setup. Move the provided script into their directory and run via module:

  mv benchmark_llavamed.py ~/LLaVA-Med/llava/serve
  python -m llava.serve.benchmark_llavamed --model-name llava-med-v1.5-mistral-7b --controller http://localhost:10000

After running benchmarks, use the following tools to inspect logs and analyze performance axes.

Inspect Logs

To view logs (defaults to the most recent log file):

python inspect_logs.py [optional: log-file] -n [num-logs]

Analyze Results

To analyze results across different axes:

python analyze_axes.py results/[logfile].json ../benchmark/questions/ --model [gpt4|llama|chexagent|llava-med] --max-questions [optional:int]

To install MedRAX, ensure you have Python 3.8+ and a CUDA-enabled GPU for optimal performance. Follow these steps:

1. Clone the repository.
2. Install the package in editable mode using pip install -e ..

# Clone the repository
git clone https://github.com/bowang-lab/MedRAX.git
cd MedRAX

# Install package
pip install -e .

MedRAX can connect to local LLM providers (like Ollama or LM Studio) or other OpenAI-compatible APIs (like Alibaba Cloud DashScope) by configuring environment variables.

Using Local LLMs (e.g., Ollama)

Set OPENAI_BASE_URL to your local endpoint.

Using OpenAI-compatible Providers (e.g., DashScope)

Set the base URL, API key, and the specific model name.

# Example: Local LLM (Ollama)
export OPENAI_BASE_URL="http://localhost:11434/v1"
export OPENAI_API_KEY="ollama"

# Example: Alibaba Cloud DashScope (Qwen3-VL)
export OPENAI_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
export OPENAI_API_KEY="<your-dashscope-api-key>"
export OPENAI_MODEL="qwen3-vl-235b-a22b-instruct"

Before running experiments, configure the following directory paths in your script:

• ROOT: The base directory of the MedRAX repository.
• PROMPT_FILE: Path to medrax/docs/system_prompts.txt.
• BENCHMARK_FILE: Path to the benchmark questions directory.
• MODEL_DIR: Directory containing your tool model weights.
• FIGURES_DIR: Directory containing benchmark images.
• medrax_logs: Directory where JSON logs of agent interactions will be saved.

ROOT = "/home/MedRAX"
PROMPT_FILE = f"{ROOT}/medrax/docs/system_prompts.txt"
BENCHMARK_FILE = f"{ROOT}/benchmark/questions"
MODEL_DIR = "/home/models"
FIGURES_DIR = f"{ROOT}/benchmark/figures"

model_name = "medrax"
temperature = 0.2
medrax_logs = f"{ROOT}/experiments/medrax_logs"
# ... setup logging ...

After initializing the agent and tools, you can create and launch a web-based demonstration interface using create_demo() and the Gradio .launch() method.

1. Call create_demo(agent, tools_dict) to wrap the agent and tools into a Gradio interface.
2. Call .launch() on the resulting demo object to start the server.

Common parameters for .launch():

• server_name: The network address to bind to (e.g., "0.0.0.0").
• server_port: The port number (e.g., 8585).
• share: Boolean to create a public URL via Gradio.

from interface import create_demo

# Assuming agent and tools_dict are already initialized
demo = create_demo(agent, tools_dict)
demo.launch(server_name="0.0.0.0", server_port=8585, share=True)

MedRAX allows for selective tool initialization. You can specify a list of tools to use when calling initialize_agent. This is useful for managing resource constraints or using only the capabilities you need.

selected_tools = [
    "ImageVisualizerTool",
    "ChestXRayClassifierTool",
    "ChestXRaySegmentationTool",
    # Add or remove tools as needed
]

agent, tools_dict = initialize_agent(
    "medrax/docs/system_prompts.txt",
    tools_to_use=selected_tools,
    model_dir="/model-weights"
)

MedRAX tools handle model weights differently:

Automatically Downloaded Tools

Initializing these tools will automatically download weights to the specified cache_dir or model_dir:

• ChestXRayClassifierTool(device=device)
• ChestXRaySegmentationTool(device=device)
• XRayPhraseGroundingTool(cache_dir=model_dir, temp_dir=temp_dir, load_in_8bit=True, device=device)
• LlavaMedTool(cache_dir=model_dir, device=device, load_in_8bit=True)
• ChestXRayReportGeneratorTool(cache_dir=model_dir, device=device)
• XRayVQATool(cache_dir=model_dir, device=device)

Manual Setup Required

• ChestXRayGeneratorTool: Requires RoentGen weights. You must contact the authors of RoentGen, download the weights, and place them in {model_dir}/roentgen.