Qwen-AgentWorld

Evaluation is performed using the eval/eval.py script via a three-step pipeline: inference, judging, and scoring. The script uses OpenAI-compatible APIs for both the world model and the LLM judge.

Step 1: World Model Inference

Generate predictions from your world model.

• --data-dir: Path to the downloaded AgentWorldBench directory.
• --model-base-url: The OpenAI-compatible endpoint for your world model.
• --model-name: The name of the model being evaluated.
• --output-dir: Where to save the predictions.

Step 2: LLM Judge Scoring

Score the predictions using an LLM judge.

• Requires OPENAI_API_KEY to be set in the environment.
• --predictions: Path to the predictions.jsonl generated in Step 1.
• --judge-base-url: The endpoint for the judge LLM.
• --judge-model: The specific model to use as the judge.
• --output-dir: Where to save the judged results.

Step 3: Aggregate Scores

Calculate and display the final metrics.

• --predictions: Path to the judged.jsonl generated in Step 2.

cd eval

# Step 1: Run world model inference
python eval.py infer \
    --data-dir ../AgentWorldBench \
    --model-base-url http://localhost:8000/v1 \
    --model-name Qwen/Qwen-AgentWorld-35B-A3B \
    --output-dir ./results

# Step 2: Run LLM judge scoring
export OPENAI_API_KEY="your-api-key"
python eval.py judge \
    --predictions ./results/predictions.jsonl \
    --judge-base-url https://api.openai.com/v1 \
    --judge-model gpt-5.2-2025-12-11 \
    --output-dir ./results

# Step 3: Aggregate and display scores
python eval.py score --predictions ./results/judged.jsonl

You can use the standard transformers library to run inference. The model is designed to simulate environments (like a Linux terminal) based on specific system prompts and user actions.

For best results when using the model as an environment simulator, use the domain-specific system prompt templates provided in the prompts/ directory of the repository. Each domain folder contains a system_prompt.txt (for the world model) and a judge_system_prompt.txt (for evaluation).

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Qwen/Qwen-AgentWorld-35B-A3B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
)

messages = [
    {
        "role": "system",
        "content": "You are a language world model simulating a Linux terminal environment. "
                   "Given the user's command, predict the terminal output."
    },
    {
        "role": "user",
        "content": "Action: execute_bash\nCommand: ls -la /home/user/project/"
    }
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=2048, temperature=0.6)
response = tokenizer.decode(outputs[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True)
print(response)

AgentWorldBench data is stored in per-domain JSONL files (e.g., mcp_test.jsonl, search_test.jsonl, terminal_test.jsonl, swe_test.jsonl, android_test.jsonl, web_test.jsonl, os_test.jsonl).

Each entry in the JSONL file represents a trajectory turn and contains the following key fields:

• system_str: The specific world model system prompt for this sample. Note: The files in the prompts/ directory of this repository are templates only; always use the system_str from the data sample during evaluation.
• prompt / response: Lists containing the full trajectory of action prompts and ground-truth environment observations.
• current_prompt: The specific action prompt for the turn currently being evaluated.
• turn_idx: The 1-indexed position of the current turn.
• task: The domain name (e.g., mcp).
• id: A unique identifier for the sample.

{
    "task": "mcp",
    "id": 145256090131919,
    "prompt": ["### Turn 1\n**Action:**\n```json\n{...}\n```\n..."],
    "response": ["**Environment Observation:**\n{...}"],
    "current_prompt": "### Turn 1\n**Action:**\n...",
    "system_str": "# Role and Objective\n\nYou are a **Tool World Model** ...",
    "turn_idx": 1,
    "total_turns": 5
}