LEAP-Finetune

GRPO uses vLLM for rollouts. You can choose between two main modes:

1. Colocate Mode (vllm_mode: "colocate"): Runs vLLM inside each training worker. This is the default.
2. Server Mode (vllm_mode: "server"): Starts trl vllm-serve on driver GPUs before Ray initializes. This is better for multi-node training if you point vllm_server_base_url to an external server.

GPU Management: Configure GPU counts using specific keys rather than device IDs. Use server_gpus to reserve GPUs for the vLLM server and training_gpus for the training process. If server_gpus is set, the remaining GPUs are allocated to training.

Note on Multi-node: Local server partitioning is single-node only. For multi-node, use colocate mode or an externally managed vLLM server.

grpo_rollout:
  server_gpus: 1
  judge_gpus: 1
  tensor_parallel_size: 1
  dtype: "bfloat16"
  gpu_memory_utilization: 0.9

training_config:
  extends: "DEFAULT_GRPO"
  vllm_mode: "server"
  vllm_server_host: "auto"
  vllm_server_port: 8000

Different training types require specific JSON/JSONL data structures.

DPO (Direct Preference Optimization)

Uses preference columns. For multi-turn, prompt is the shared history, while chosen and rejected are the completions.

{
  "prompt": [{"role": "user", "content": "..."}],
  "chosen": [{"role": "assistant", "content": "..."}],
  "rejected": [{"role": "assistant", "content": "..."}]
}

KTO (Kahneman-Tversky Optimization)

Learns from unpaired binary feedback. Requires prompt, completion, and a boolean label (true = desirable).

{
  "prompt": [{"role": "user", "content": "..."}],
  "completion": [{"role": "assistant", "content": "..."}],
  "label": false
}

Note: per_device_train_batch_size must be > 1 for KTO.

Retrieval and Embedding Models

Use training_type: "embedding" or training_type: "colbert". Requires query, positive, and optionally negative columns.

{
  "query": "search text",
  "positive": "relevant document",
  "negative": "irrelevant document"
}

Benchmark data must follow the HuggingFace messages schema. Supported formats include JSONL, JSON, Parquet, and CSV.

Generation Benchmarks

The last assistant turn is treated as the ground truth. Everything preceding it is the prompt.

{
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "image", "image": "/path/to/img.jpg" },
        { "type": "text", "text": "What is shown in this image?" }
      ]
    },
    {
      "role": "assistant",
      "content": [{ "type": "text", "text": "A dog sitting on a beach." }]
    }
  ]
}

Logprob MCQ Benchmarks

No assistant turn is required. The model scores each option provided in the options list by log-probability and selects the highest. The correct answer is specified by answer_id.

{
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "image", "image": "/path/to/img.jpg" },
        { "type": "text", "text": "What animal is in this image?" }
      ]
    }
  ],
  "options": ["cat", "dog", "bird"],
  "answer_id": 1
}

By default, training pauses during evaluation (sync mode). To run benchmarks without blocking training, use the async_eval block with one of the following modes:

Mode Details:

• sidecar: Launches a short sbatch job per eval_steps. Scales to multi-node training.
• reserved: Carves GPUs off the same SLURM allocation via CUDA_VISIBLE_DEVICES. Single-node only. If used in multi-node training, it will raise NotImplementedError.

Failures in async eval are isolated; training continues unless failure.max_consecutive is reached.

async_eval:
  mode: sidecar # sync (default) | sidecar | reserved
  vllm_gpus: 1
  tensor_parallel_size: 1
  gpu_memory_utilization: 0.9

  sbatch:
    time: "00:30:00"

  reserved:
    weight_reload: respawn
    server_port: 8100

Tool calls must use LFM bracket notation in the assistant content field. Tool definitions belong in the system prompt, and responses use role: "tool".

Required Format:

• Assistant content: <|tool_call_start|>[func(args)]<|tool_call_end|>
• Tool response: Do not include <|tool_response_start|> or <|tool_response_end|> markers; the LFM2 template adds these automatically.
• Tool definitions: For LFM2, wrap definitions in <|tool_list_start|> and <|tool_list_end|>. For LFM2.5, omit them (the pipeline will auto-strip them if present).

Example:

{
  "messages": [
    { "role": "system", "content": "List of tools: [\"...\"]" },
    { "role": "user", "content": "What's the weather in Boston?" },
    { "role": "assistant", "content": "<|tool_call_start|>[get_weather(location=\"Boston\")]<|tool_call_end|>" },
    { "role": "tool", "content": "{\"temperature\": 72}" },
    { "role": "assistant", "content": "It's 72 F and sunny in Boston." }
  ]
}