DOCSEARCHbyAlgolia
Submit Docs

HuatuoGPT-o1 Documentation

repository·main·Indexed 23 days ago

https://github.com/freedomintelligence/huatuogpt-o1

A specialized medical LLM designed for complex medical reasoning using a 'think-before-answer' approach. The project includes models based on LLaMA-3.1 and Qwen2.5 (ranging from 7B to 72B), and provides workflows for direct inference via transformers, constructing verifiable medical problems, Supervised Fine-Tuning (SFT), and Reinforcement Learning (PPO) guided by a medical verifier.

Tokens
1.8K
Snippets
4
Records
5
Agent score
30%

What's inside HuatuoGPT-o1

  1. Train HuatuoGPT-o1 via Supervised Fine-Tuning (SFT)

    main

    Stage 1 training uses accelerate launch with DeepSpeed Zero-3. This is designed for an 8-GPU setup. You need to provide a base model path and the SFT dataset path.

    accelerate launch --config_file ./configs/deepspeed_zero3.yaml \
    --num_processes 8  \
    --num_machines 1 \
    --machine_rank 0 \
    --deepspeed_multinode_launcher standard SFT_stage1.py \
    --model_path [meta-llama/Llama-3.1-8B-Instruct] \
    --data_path [FreedomIntelligence/medical-o1-reasoning-SFT]

  2. Construct verifiable medical problems and reasoning paths

    main

    Use the provided scripts to prepare data for fine-tuning.

    1. Construct Verifiable Problems: Converts multi-choice questions into open-ended verifiable problems.
    2. Search Reasoning Paths: Generates complex reasoning trajectories (CoT) for SFT data.

    Both scripts require a --data_path to a JSON file and a --api_key for the model used (e.g., gpt-4o).

  3. Train HuatuoGPT-o1 via Reinforcement Learning (PPO)

    main

    Stage 2 uses the trl library for PPO. This requires a reward model (the medical verifier) and a value model.

    Key parameters for the PPO script include:

    • --reward_model_path: Path to the medical verifier.
    • --value_model_path: Path to the value model (e.g., Llama-3.2-3B-Instruct).
    • --dataset_name: The verifiable problem dataset.
    • --kl_coef: KL divergence coefficient.
    accelerate launch \
	--num_processes 8 \
	--num_machines 1 \
	--machine_rank 0 \
    --config_file ./configs/deepspeed_zero3.yaml \
	--deepspeed_multinode_launcher standard RL_stage2.py \
    --model_name_or_path [FreedomIntelligence/HuatuoGPT-o1-8B] \
    --reward_model_path [FreedomIntelligence/medical_o1_verifier_3B] \
    --value_model_path [meta-llama/Llama-3.2-3B-Instruct] \
    --dataset_name  [FreedomIntelligence/medical-o1-verifiable-problem]\\
    --response_length 1300 \
    --temperature 0.5 \
    --local_rollout_forward_batch_size 8 \
    --num_ppo_epochs 3 \
    --num_mini_batches 1 \
    --total_episodes 20000 \
    --per_device_train_batch_size 1 \
    --gradient_accumulation_steps 16 \
    --bf16 True \
    --output_dir ./ckpts \
    --save_strategy steps \
    --save_step 20 \
    --save_total_limit 1 \
    --eval_strategy steps \
    --eval_steps 20 \
    --kl_coef 0.03 \
    --learning_rate 5e-7 \
    --warmup_ratio 0.05 \
    --gradient_checkpointing True \
    --dataloader_num_workers 4 \
    --run_name ppo_medical_o1_8B \
    --num_sample_generations -1 \
    --report_to wandb

  4. Evaluate HuatuoGPT-o1 models

    main

    Evaluation is a three-step process using Sglang:

    1. Deploy the model: Use sglang.launch_server to host the model on a specific port.
    2. Run evaluation: Execute evaluation/eval.py pointing to the deployed port and an evaluation JSON file. Use the --strict_prompt option for more precise answer extraction.
    3. Cleanup: Run evaluation/kill_sglang_server.sh to stop the server and release GPU memory.
    # 1. Deploy with Sglang
log_num=0
model_name="FreedomIntelligence/HuatuoGPT-o1-8B"
port=28${log_num}35
CUDA_VISIBLE_DEVICES=0  python -m sglang.launch_server --model-path $model_name --port $port --mem-fraction-static 0.8 --dp 1 --tp 1  > sglang${log_num}.log 2>&1 &

# 2. Run evaluation
python evaluation/eval.py --model_name $model_name  --eval_file evaluation/data/eval_data.json --port $port 

# 3. Stop Sglang service
bash evaluation/kill_sglang_server.sh

  5. Perform direct inference with HuatuoGPT-o1

    main

    You can perform direct inference using the transformers library. HuatuoGPT-o1 follows a thinks-before-it-answers approach, where the output is structured into two distinct sections: ## Thinking (the reasoning process) and ## Final Response (the actual answer).

    Available models include:

    • HuatuoGPT-o1-8B (LLaMA-3.1-8B, English)
    • HuatuoGPT-o1-70B (LLaMA-3.1-70B, English)
    • HuatuoGPT-o1-7B (Qwen2.5-7B, English & Chinese)
    • HuatuoGPT-o1-72B (Qwen2.5-72B, English & Chinese)
    from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("FreedomIntelligence/HuatuoGPT-o1-8B",torch_dtype="auto",device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("FreedomIntelligence/HuatuoGPT-o1-8B")

input_text = "How to stop a cough?"
messages = [{"role": "user", "content": input_text}]

inputs = tokenizer(tokenizer.apply_chat_template(messages, tokenize=False,add_generation_prompt=True
), return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=2048)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))