LLaMA Factory: Unified Efficient Fine-Tuning of 100+ LLMs

LlamaFactory is an open-source platform designed for easy fine-tuning of hundreds of large language models (LLMs) using either a zero-code Command Line Interface (CLI) or a Web UI (LLaMA Board) powered by Gradio.

Key capabilities include:

• Model Support: Wide range of models including LLaMA, LLaVA, Mistral, Mixtral-MoE, Qwen, DeepSeek, Gemma, GLM, Phi, etc.
• Training Methods: Supports (incremental) pre-training, instruction fine-tuning (including multimodal), reward model training, and various RLHF methods like PPO, DPO, KTO, and ORPO.
• Precision Options: Full parameter fine-tuning (16-bit), frozen fine-tuning, LoRA, and QLoRA (supporting 2/3/4/5/6/8-bit quantization via AQLM/AWQ/GPTQ/LLM.int8/HQQ/EETQ).
• Advanced Algorithms: Integration of GaLore, BAdam, APOLLO, Adam-mini, Muon, OFT, DoRA, LongLoRA, LLaMA Pro, Mixture-of-Depths, LoRA+, LoftQ, and PiSSA.
• Optimization & Speed: Supports FlashAttention-2, Unsloth, Liger Kernel, KTransformers, RoPE scaling, NEFTune, and rsLoRA. Inference can be accelerated using vLLM or SGLang.
• Task Versatility: Handles multi-turn dialogue, tool calling, image understanding, visual grounding, video recognition, and speech understanding.
• Experiment Monitoring: Integrates with LlamaBoard, TensorBoard, Wandb, MLflow, and SwanLab.

The Alpaca format is used for Supervised Fine-Tuning (SFT), Pre-training, and Preference learning.

Supervised Fine-Tuning (SFT)

Concatenates instruction and input to form the prompt (instruction\ninput). The output column is the target response.

• System Prompts: Use the system column.
• Multi-turn Conversation: Use the history column (a list of [user_prompt, assistant_response] tuples). Note: history responses are also learned.
• Reasoning/CoT: For models like Qwen3, place Chain-of-Thought in the response as <think>cot</think>output.

Pre-training

Requires a single text column containing the raw document string.

Preference Learning (DPO, ORPO, SimPO)

Requires a chosen column (better response) and a rejected column (worse response). Set ranking: true in dataset_info.json.

Specialized Formats

• KTO: Requires a kto_tag column.
• Multimodal: Requires images, videos, or audios columns.

// SFT Example
[
  {
    "instruction": "user instruction (required)",
    "input": "user input (optional)",
    "output": "model response (required)",
    "system": "system prompt (optional)",
    "history": [["user instruction 1", "model response 1"]]
  }
]

// Pre-training Example
[{"text": "document"}]

// Preference Example
[
  {
    "instruction": "user instruction",
    "chosen": "chosen answer",
    "rejected": "rejected answer"
  }
]

For SFT using the Alpaca format, the prompt is constructed by concatenating instruction and input (i.e., instruction\ninput). The output column contains the model's response.

• System Prompt: Use the system column if provided.
• History: The history column should be a list of string tuples [[instruction, response], ...] representing dialogue turns. Note that history responses are also used for model learning.
• Reasoning/CoT: For reasoning models, include the Chain of Thought (CoT) within the output column using the format <think>cot</think>output.

Note on enable_thinking: If the model supports reasoning (e.g., Qwen3) but the dataset lacks CoT, LLaMA-Factory adds an empty CoT.

• If enable_thinking=True (Slow Thinking/Default): Empty CoT is added to the response and loss is calculated.
• If enable_thinking=False (Fast Thinking): Empty CoT is added to the user instruction and loss is NOT calculated. Keep this parameter consistent between training and inference.

[
  {
    "instruction": "User instruction (required)",
    "input": "User input (optional)",
    "output": "Model response (required)",
    "system": "System prompt (optional)",
    "history": [
      ["First turn instruction (optional)", "First turn response (optional)"],
      ["Second turn instruction (optional)", "Second turn response (optional)"]
    ]
  }
]

LlamaFactory is open-source under the Apache-2.0 license.

Important Note on Model Weights: While the LlamaFactory code is Apache-2.0, the model weights you use are subject to their own specific licenses (e.g., Llama 3, DeepSeek, Qwen, Mistral, etc.). Always verify the license of the specific model weights you are fine-tuning.

If you use this project in research, you can cite it using the following BibTeX format:

@inproceedings{zheng2024llamafactory,
  title={LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models},
  author={Yaowei Zheng and Richong Zhang and Junhao Zhang and Yanhan Ye and Zheyan Luo and Zhangchi Feng and Yongqiang Ma},
  booktitle={Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  address={Bangkok, Thailand},
  publisher={Association for Computational Linguistics},
  year={2024},
  url={http://arxiv.org/abs/2403.13372}
}