DOCSEARCHbyAlgolia
Submit Docs

Arena-Hard-Auto Documentation

repository·main·Indexed 21 days ago

https://github.com/lmarena/arena-hard-auto

An automated evaluation framework for instruction-tuned LLMs designed to approximate human preferences from Chatbot Arena. It features a challenging benchmark including 500 real-world user queries and 250 creative writing queries, utilizing advanced models like GPT-4.1 and Gemini-2.5 as automatic judges. The toolkit includes BenchBuilder for curating high-quality benchmarks through a two-stage annotation and filtering pipeline, style control to account for token length and markdown density, and support for multiple API endpoints including OpenAI, Anthropic, Vertex AI, and Amazon Bedrock.

Tokens
2.2K
Snippets
11
Records
13
Agent score
27%

What's inside Arena-Hard-Auto

  1. About Arena-Hard-Auto

    main

    Arena-Hard-Auto is an automatic evaluation tool for instruction-tuned LLMs designed to correlate highly with LMArena (Chatbot Arena) human preferences.

    Arena-Hard-v2.0-Preview features:

    • 500 challenging real-world user queries (software engineering, math, etc.).
    • 250 creative writing queries.
    • Automatic judges (GPT-4.1 and Gemini-2.5) acting as fast approximators to human preference.
    • Support for Style Control.

  2. How BenchBuilder works

    main

    BenchBuilder is an automatic two-stage pipeline designed to curate high-quality benchmarks from large datasets (like Chatbot Arena or WildChat).

    1. Annotation & Cheap Filtering: Prompts are annotated (e.g., using GPT-3.5-Turbo) and filtered to remove low-quality prompts or clusters that fall below specific score thresholds.
    2. High-Quality Curation: Remaining prompts are annotated with a more capable model (e.g., GPT-4-Turbo) and subjected to stricter quality and cluster-mean thresholds to minimize false positives.
    3. Sampling: After curation, the final benchmark is created by stratified sampling multiple prompts per cluster.

  3. Apply Style Control to model answers

    main

    Style Control allows you to account for attributes like token length and markdown density in evaluations.

    1. Add style attributes to answers

    If your model answers do not have style attributes (token length, number of headers, etc.), run add_markdown_info.py to append them. This script reads from --dir and saves to --output-dir.

    python add_markdown_info.py --dir data/arena-hard-v0.1/model_answer --output-dir data/arena-hard-v0.1/model_answer

    2. Control for style in results

    When running show_result.py, use the --control-features or -f flag to isolate or control for specific styles:

    • Full style control (markdown and length): python show_result.py -f markdown length
    • Markdown density only: python show_result.py -f markdown
    • Length control only: python show_result.py -f length
    python add_markdown_info.py --dir data/arena-hard-v0.1/model_answer --output-dir data/arena-hard-v0.1/model_answer

  4. Add a new model to Amazon Bedrock support

    main

    To support a new model hosted on Amazon Bedrock, you must perform two steps: update the configuration and register a handler.

    1. Update config/api_config.yaml

    Add a new entry for your model.

    Example Configuration:

    aws_nova_light_v1:
  model: aws_nova_light_v1
  model_id: us.amazon.nova-lite-v1:0
  endpoints: null
  api_type: aws_nova
  parallel: 8
  max_tokens: 4096
  temperature: 0.0

    Key Fields:

    • model: Internal alias for referencing the config.
    • model_id: The Bedrock-specific model identifier.
    • api_type: Must match the name used in the @register_api decorator in utils/completion.py.
    • endpoints: Set to null for default Bedrock endpoint, or provide a custom override.
    • parallel: Controls parallel inference calls (adjust for throughput).
    • max_tokens: Maximum output tokens.
    • temperature: Controls randomness (0.0 for deterministic).

    2. Register a Model Handler in utils/completion.py

    Create a new function decorated with @register_api("<api_type>"). This function defines how to format inputs, send requests via boto3 using the Bedrock invoke_model API, and parse the response.

    Implementation Checklist:

    • Ensure api_type in api_config.yaml matches the @register_api decorator.
    • Implement correct input formatting (prompt structure/message lists).
    • Map parameters correctly (temperature, max_tokens, model_id).
    • Implement correct response parsing (e.g., handling nested output.message.content).

  5. Evaluate models using Arena-Hard-Auto

    main

    The evaluation process follows a four-step pipeline: setting up API endpoints, generating model answers, generating judgments, and showing results.

    1. Set up API Endpoints

    Configure your model's API endpoint in config/api_config.yaml. Supported types include OpenAI-compatible servers, Anthropic, Vertex AI, and Amazon Bedrock. You can host models using vLLM or SGLang to provide an OpenAI-compatible interface.

    2. Generate Model Answers

    1. Add your model name to the model_list in config/gen_answer_config.yaml.
    2. Run the generation script:
    python gen_answer.py

    Note: Caching is enabled; it will skip prompts that already have existing answers/judgments (except when using the built-in SGLang server).

    3. Generate Judgments

    1. Add your model name to the model_list in config/arena-hard-v2.0.yaml.
    2. Configure the judge model in config/arena-hard-v2.0.yaml. GPT-4.1 is recommended for stability. To use Gemini-2.5, comment out the GPT-4.1 block and uncomment the Gemini-2.5 block.
    3. Run the judgment script:
    python gen_judgment.py

    4. Show Results

    Use show_result.py to output win rates. Use --judge-names to specify judges and --category for specific subsets.

    Example: Arena-Hard-v2.0-Preview (Hard Prompt, Style Control, GPT-4.1 as Judge)

    python show_result.py --judge-names gpt-4.1 --control-features markdown length

    Example: Arena-Hard-v2.0-Preview (Creative Writing, Ensemble GPT-4.1 and Gemini 2.5)

    python show_result.py --judge-names gpt-4.1 gemini-2.5 --category creative_writing
    # See content for multiple commands

  7. Install Arena-Hard-Auto dependencies

    main

    To set up the Arena-Hard-Auto environment, clone the repository and install the required Python packages. It is recommended to also install the optional dependencies if you plan to use specific model providers like Anthropic.

    git clone https://github.com/lmarena/arena-hard-auto.git
cd arena-hard
pip install -r requirements.txt
pip install -r requirements-optional.txt

  9. Download Arena-Hard datasets

    main

    You can download pre-generated model answers and judgments from Hugging Face using git-lfs. After downloading, copy the contents of the data directory into your local arena-hard directory.

    git lfs install
git clone git@hf.co:datasets/lmarena-ai/arena-hard-auto arena-hard-data
cp -r arena-hard-data/data .