Absolute Zero Reasoner (AZR)

EvalPlus provides three primary CLI entrypoints for the code generation and evaluation lifecycle:

• evalplus.codegen: Performs code generation followed by automatic code post-processing.
• evalplus.evaluate: Performs code generation, post-processing, and full evaluation.
• evalplus.sanitize: Performs only code post-processing (cleaning up LLM-generated text to ensure compilability).

The evalplus.evalperf process follows four distinct steps:

1. Sampling: Samples 100 solutions (n_samples) from the target LLM for each task.
2. Efficiency Evaluation: For tasks where at least 10 samples pass (min_correct), the script performs efficiency profiling.
3. Data Generation: Produces a .jsonl output file containing detailed results for each task, including solutions, pass status, and profiling metrics (like _num_cpu_instructions and dps).
4. Score Computation: Computes the final Differential Performance Score (DPS).

AZR uses the Deepseek R1 <think> and <answer> tags as the prompt template. The assistant is expected to reason within <think> tags and provide the final answer within <answer> tags.

Template Structure:

A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think> <answer> answer here </answer>. User: {question}
Assistant: <think>

The Absolute Zero Reasoner (AZR) uses a reinforced self-play reasoning approach that requires zero external training data. The algorithm follows an iterative two-step loop:

1. PROPOSE: The model generates reasoning tasks using abduction, deduction, and induction. These tasks are validated via Python execution and assigned a learnability reward.
2. SOLVE: The model attempts to solve the self-generated tasks. Solutions are verified through Python execution and assigned an accuracy reward.

This cycle uses TRR++ to create a self-evolving loop, strengthening reasoning capabilities through self-play.

Use the evalplus.evaluate command to evaluate LLM code generation performance on datasets like humaneval or mbpp. You must specify a --model, a --dataset, and a --backend. For most evaluations, adding --greedy is recommended.

Results and generated code are saved to the evalplus_results/[humaneval|mbpp]/ directory.

evalplus.evaluate --model "ise-uiuc/Magicoder-S-DS-6.7B" \
                  --dataset humaneval \
                  --backend hf \
                  --greedy

You can use the vllm backend directly for local models, or use the openai backend to connect to a running vLLM OpenAI-compatible server.

Direct vLLM usage: Use the --tp flag to specify the Tensor Parallel size.

vLLM OpenAI-compatible server usage: Set the --base-url to point to your server (e.g., http://localhost:8000/v1).

evalplus.evaluate --model "ise-uiuc/Magicoder-S-DS-6.7B" \
                  --dataset [humaneval|mbpp]             \
                  --backend vllm                         \
                  --tp [TENSOR_PARALLEL_SIZE]            \
                  --greedy

Execute the self-play training scripts based on your model size.

Hardware Requirements:

• 3b models: 2 x 80GB GPUs
• 7/8b models: 4 x 80GB GPUs
• 14b models: 8 x 80GB GPUs

Standard usage:

bash scripts/selfplay/<7b|14b|coder3b|coder7b|coder14b|llama>.sh

Using custom seed datasets: If you have custom datasets, export their paths first:

export OUTPUT_SEED_PATH=data/<your_ded_abd_seed_data_name>.jsonl
export OUTPUT_CODE_F_SEED_PATH=data/<your_ind_seed_data_name>.jsonl
bash scripts/selfplay/<7b|14b|coder3b|coder7b|coder14b|llama>.sh

Using Sandbox-Fusion Executor: To use the sandbox-fusion executor, run via Docker and set the configuration flag azr.executor=sandboxfusion.

⚠️WARNING⚠️: The Python executor in this repository is very raw and intended for research purposes only. It is not secure for production environments.

If you want to create your own seed data instead of using the provided datasets in data/, use the seeding scripts. You must export the output paths for the ded/abd seed data and the individual seed data before running the script.

export OUTPUT_SEED_PATH=data/<new_ded_abd_seed_data_name>.jsonl
export OUTPUT_CODE_F_SEED_PATH=data/<new_ind_seed_data_name>.jsonl
bash scripts/seeding/<7b|14b|coder3b|coder7b|coder14b|llama>.sh

To resume a previous run, provide the original Weights & Biases (wandb) run ID in the script using the trainer.wandb_run_id parameter.

Example: trainer.wandb_run_id=<run_id>