DeepConf Documentation

You can install DeepConf via pip or by using uv to install dependencies from the requirements file.

pip install deepconf
uv pip install -r requirements.txt

To use the example scripts, ensure you have the following installed:

• Python 3.10+
• vllm
• deepconf (DeepThinkLLM)
• dynasor (required for math_equal)

Install additional data processing dependencies with:

pip install pandas numpy tqdm

Use the provided analysis utilities to aggregate accuracy, tokens, and timing across (qid, rid) runs.

For Online runs: Use examples/example_analyze_online.py.

For Baseline runs: Use examples/example_analyze_online_baseline.py.

Both scripts accept the following flags:

• --output_dir: The directory containing the run logs.
• --max_qid: Limit analysis to questions up to this ID.
• --rids: Specify which run IDs to include.

# Analyze online runs
python examples/example_analyze_online.py --output_dir ./online-dpsk/ --max_qid 29 --rids 1

# Analyze baseline runs
python examples/example_analyze_online_baseline.py --output_dir ./baseline-dpsk/ --max_qid 29 --rids 1

Use examples/example_online.py to run DeepThinkLLM in accelerated online mode with confidence-aware early-exit. In this mode, voting algorithms are applied only to the traces that pass the online filtering step.

Required CLI flags:

• --qid: Question ID
• --rid: Run ID
• --dataset: Path to the JSONL dataset file
• --total_budget: Token budget
• --output_dir: Directory to save results

python examples/example_online.py --qid $1 --rid $2 --dataset brumo_2025.jsonl --total_budget 256 --output_dir online-dpsk

To evaluate the accuracy of the reasoning, iterate through the voting_results dictionary and compare the extracted answer against your ground truth.

# Run inference
result = deep_llm.deepthink(prompt=prompt, mode="offline", budget=32)

# Evaluate results
ground_truth = "1024"
for method, method_result in result.voting_results.items():
    if method_result:
        # Note: conversion to int may be required depending on your task
        is_correct = int(method_result['answer']) == int(ground_truth)
        print(f"{method}: {is_correct}")

Use examples/example_online_baseline.py to run the baseline version of DeepThinkLLM (no early-exit). This is used for fair comparison against the online mode to measure accuracy, token usage, and generation time.

Required CLI flags:

• --qid: Question ID
• --rid: Run ID
• --dataset: Path to the JSONL dataset file
• --total_budget: Token budget
• --output_dir: Directory to save results

python examples/example_online_baseline.py --qid $1 --rid $2 --dataset brumo_2025.jsonl --total_budget 256 --output_dir baseline-dpsk

The scripts expect datasets in JSONL format where each line is a JSON object containing a question and an answer. You can convert datasets from the Hugging Face datasets library using the following pattern:

import json
from datasets import load_dataset

# Load dataset
dataset = load_dataset("MathArena/aime_2025", split="train")

# Convert to JSONL
with open("aime_2025.jsonl", "w", encoding="utf-8") as f:
    for example in dataset:
        entry = {
            "question": example["problem"],
            "answer": str(example["answer"])
        }
        f.write(json.dumps(entry, ensure_ascii=False) + "\n")

print(f"Converted {len(dataset)} examples to aime_2025.jsonl")

The DeepThinkLLM class is a wrapper around vLLM that adds reasoning capabilities. When initializing, you provide the model path or name and can pass any standard **vllm_kwargs (e.g., tensor_parallel_size, enable_prefix_caching) to configure the underlying vLLM engine.

from deepconf import DeepThinkLLM

# Initialize with a model name and standard vLLM arguments
deep_llm = DeepThinkLLM(
    model="deepseek-ai/DeepSeek-R1-0528-Qwen3-8B",
    tensor_parallel_size=1
)

from deepconf import DeepThinkLLM

deep_llm = DeepThinkLLM(model="deepseek-ai/DeepSeek-R1-0528-Qwen3-8B")

Online mode uses warmup traces to establish confidence thresholds and applies early stopping to improve efficiency.

Arguments:

• prompt: The formatted prompt string.
• mode: Set to "online".
• warmup_traces: Number of calibration runs to establish thresholds.
• total_budget: Maximum number of traces allowed.
• sampling_params: Optional sampling parameters.

result = deep_llm.deepthink(
    prompt=prompt,
    mode="online",
    warmup_traces=16,
    total_budget=256,
    sampling_params=sampling_params
)

Offline mode generates all traces in a single batch and applies multiple voting strategies to determine the best answer.

Arguments:

• prompt: The formatted prompt string.
• mode: Set to "offline".
• budget: Total number of traces to generate.
• compute_multiple_voting: Boolean to enable all available voting methods.
• sampling_params: Optional sampling parameters.

result = deep_llm.deepthink(
    prompt=prompt,
    mode="offline",
    budget=512,
    compute_multiple_voting=True,
    sampling_params=sampling_params
)