DOCSEARCHbyAlgolia
Submit Docs

openai/simple-evals

repository·main·Indexed 26 days ago

https://github.com/openai/simple-evals

A lightweight library for evaluating language models using zero-shot and chain-of-thought prompting. It provides reference implementations for benchmarks including MMLU, MATH, GPQA, DROP, MGSM, HumanEval, SimpleQA, BrowseComp, and HealthBench. The library supports sampling interfaces for OpenAI and Anthropic APIs and includes specialized analysis scripts for HealthBench and Multilingual MMLU.

Tokens
6.2K
Snippets
13
Records
20
Agent score
89%

What's inside simple-evals

  1. Understand the purpose and scope of simple-evals

    main

    The simple-evals library is a lightweight tool for evaluating language models using a zero-shot, chain-of-thought prompting setting. It is designed to reflect realistic model performance by using simple instructions (e.g., "Solve the following multiple choice problem") rather than complex few-shot or role-playing prompts.

    Note: This repository is deprecated as of July 2025 and will no longer be updated for new models or benchmarks. It serves as a reference implementation for HealthBench, BrowseComp, and SimpleQA.

  3. Install dependencies for simple-evals

    main

    Because of optional dependencies, you must install the specific packages required for the evals and samplers you intend to use.

    For HumanEval (Python programming):

    git clone https://github.com/openai/human-eval
pip install -e human-eval

    For OpenAI API:

    pip install openai

    For Anthropic API:

    pip install anthropic

  4. Analyze HealthBench evaluation results

    main

    To analyze HealthBench results, you must first locate the output files generated by the simple-evals run. These files typically follow the naming pattern healthbench{OPTIONAL_SUBSET_NAME}_{MODEL_NAME}_{DATETIME}.json and *.allresults.json. By default, these are saved to your /tmp/ folder.

    Workflow:

    1. Move the .json and *allresults.json files to a local directory.
    2. In the analysis notebook, update the tmp_dir and results_dir variables to point to your local paths.
    3. Populate the main_filename_list (or other specific lists like hard_filename_list) with the names of your files.
    4. Run the analysis cells to generate descriptive statistics, LaTeX tables, and clustered bar plots.
    # Example configuration for analysis
tmp_dir = '/Users/rahul/Documents/healthbench/data/'
results_dir = '/Users/rahul/Documents/healthbench/results/'

main_filename_list = """healthbench_gpt-3.5-turbo-0125_20250507_0653.json
healthbench_gpt-4.1_20250507_0653.json""".splitlines()

  5. Run the Multilingual MMLU benchmark

    main

    To reproduce the Multilingual MMLU benchmark results, run the run_multilingual_mmlu module using Python. This evaluation uses a test set translated into 14 languages (including Arabic, Chinese, French, Swahili, and Yoruba) by professional human translators to ensure accuracy in low-resource languages.

    python -m simple-evals.run_multilingual_mmlu

  6. View Multilingual MMLU Benchmark Results

    main

    The benchmark evaluates various models across 14 languages. The results are presented as accuracy scores for different model versions such as o3-high, o1, gpt-4o, and others. The languages covered include Arabic, Bengali, Chinese (Simplified), French, German, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese (Brazil), Spanish, Swahili, and Yoruba.

    | Language             | o3-high     | o1    | o4-mini-high | o3-mini-high | gpt-4.5-preview-2025-02-27 | gpt-4.1-2025-04-14 | gpt-4o-2024-11-20 | gpt-4.1-mini-2025-04-14 | gpt-4o-mini-2024-07-18 | gpt-4.1-nano-2025-04-14 |
| :------------------: | :---------: | :---: | :----------: | :----------: | :------------------------: | :----------------: | :---------------: | :---------------------: | :--------------------: | :--------------------: |
| Arabic               | **0.904**   | 0.890 | 0.861        | 0.819        | 0.860                      | 0.844              | 0.831             | 0.795                   | 0.709                  | 0.659                   |
| Bengali              | **0.878**   | 0.873 | 0.840        | 0.801        | 0.848                      | 0.827              | 0.801             | 0.749                   | 0.658                  | 0.583                   |
| Chinese (Simplified) | **0.893**   | 0.889 | 0.869        | 0.836        | 0.870                      | 0.861              | 0.842             | 0.817                   | 0.731                  | 0.710                   |
| French               | **0.906**   | 0.893 | 0.874        | 0.837        | 0.878                      | 0.870              | 0.846             | 0.835                   | 0.766                  | 0.739                   |
| German               | **0.905**   | 0.890 | 0.867        | 0.808        | 0.853                      | 0.855              | 0.836             | 0.823                   | 0.743                  | 0.722                   |
| Hindi                | **0.898**   | 0.883 | 0.859        | 0.811        | 0.858                      | 0.842              | 0.819             | 0.780                   | 0.692                  | 0.629                   |
| Indonesian           | **0.898**   | 0.886 | 0.869        | 0.828        | 0.872                      | 0.859              | 0.840             | 0.816                   | 0.745                  | 0.714                   |
| Italian              | **0.912**   | 0.897 | 0.877        | 0.838        | 0.878                      | 0.869              | 0.845             | 0.835                   | 0.764                  | 0.734                   |
| Japanese             | **0.890**   | 0.889 | 0.869        | 0.831        | 0.869                      | 0.856              | 0.835             | 0.810                   | 0.726                  | 0.690                   |
| Korean               | **0.893**   | 0.882 | 0.867        | 0.826        | 0.860                      | 0.849              | 0.829             | 0.801                   | 0.720                  | 0.679                   |
| Portuguese (Brazil)  | **0.910**   | 0.895 | 0.878        | 0.841        | 0.879                      | 0.870              | 0.836             | 0.839                   | 0.768                  | 0.741                   |
| Spanish              | **0.911**   | 0.899 | 0.880        | 0.840        | 0.884                      | 0.876              | 0.843             | 0.839                   | 0.774                  | 0.748                   |
| Swahili              | **0.860**   | 0.854 | 0.813        | 0.738        | 0.820                      | 0.795              | 0.779             | 0.679                   | 0.619                  | 0.566                   |
| Yoruba               | **0.780**   | 0.754 | 0.708        | 0.637        | 0.682                      | 0.647              | 0.621             | 0.566                   | 0.458                  | 0.455                   |
| Average              | **0.888**   | 0.877 | 0.852        | 0.807        | 0.851                      | 0.837              | 0.814             | 0.785                   | 0.705                  | 0.669                   |

  7. Available Evals in simple-evals

    main

    The library currently includes implementations for the following benchmarks:

    • MMLU: Measuring Massive Multitask Language Understanding
    • MATH: Measuring Mathematical Problem Solving
    • GPQA: Graduate-Level Google-Proof Q&A
    • DROP: Reading Comprehension Requiring Discrete Reasoning
    • MGSM: Multilingual Grade School Math
    • HumanEval: Evaluating LLMs on Code
    • SimpleQA: Measuring short-form factuality
    • BrowseComp: Benchmark for Browsing Agents
    • HealthBench: Evaluating LLMs for Human Health

  8. Analyze HealthBench performance vs. cost

    main

    The HealthBench analysis scripts allow you to evaluate the relationship between model performance (rubric scores) and inference cost. You can calculate the average cost per model by multiplying average input and output token counts by their respective dollar costs per million tokens.

    To visualize this, use plot_dollar_cost_scatter to generate scatter plots showing the performance-cost frontier. You can choose between a 'linear' or 'log' scale for the x-axis (cost).

    Key data points used in this analysis include:

    • rubric_score_cost: A list containing [avg_cost_per_model, avg_rubric_score, model_name].
    • MODEL_FAMILIES: A grouping of related models (e.g., low, standard, and high versions) to facilitate comparison in plots.
    # Example of plotting the performance-cost frontier
plot_dollar_cost_scatter(
    rubric_score_cost, 
    title="HealthBench performance-cost frontier", 
    x_label="Inference cost per example ($)", 
    y_label="HealthBench score", 
    model_families=MODEL_FAMILIES, 
    scale='log'
)

# Exporting the cost-performance data to CSV
cost_perf_data = pd.DataFrame(rubric_score_cost, columns=['cost_usd', 'performance_pct', 'model'])[['model', 'cost_usd', 'performance_pct']]
save_csv_and_print(cost_perf_data, 'cost_perf_data.csv')

  9. Run evaluations for a specific model

    main

    To execute evaluations, specify the target model and the number of examples to run using the --model and --examples flags. This command launches evaluations through the OpenAI API.

    python -m simple-evals.simple_evals --model <model_name> --examples <num_examples>

  11. Load and process HealthBench results

    main

    The analysis script uses several helper functions to load and organize evaluation data from JSON files into Pandas DataFrames for plotting and statistical analysis.

    • load_results(filename): Loads a single JSON result file from tmp_dir.
    • get_results_by_filename(filenames): Returns a dictionary mapping model names (extracted from filenames) to their respective result dictionaries.
    • get_df_from_results_by_model(results_by_model): Converts the dictionary of results into a structured Pandas DataFrame indexed by ['model', 'metric']. This DataFrame includes value, bootstrap_std, and n_samples for each metric.
    def load_results(filename: str) -> dict:
    fp = os.path.join(tmp_dir, filename)
    results = json.load(open(fp))
    print(f'Loaded {filename}')
    return results

def get_results_by_filename(filenames: list[str]) -> dict:
    results_by_model = {
        filename_to_model(f): load_results(f)
        for f in filenames
    }
    return results_by_model

  12. Plot worst-case score at k samples

    main

    The plot_fail_at_k function visualizes the 'worst-at-k' metric, which shows the failure rate or score when selecting the worst performing sample out of $k$ attempts. This is useful for understanding model reliability and variance.

    # worst_at_k_list is a list of (score, k) tuples
results_worst_at_k = get_results_by_filename(many_replicate_filename_list)
# ... (processing logic to compute worst_at_k_per_model) ...

worst_at_k_list = [(v, k) for k, v in worst_at_k_per_model.items()]
fail_at_k_data = plot_fail_at_k(
    worst_at_k_list, 
    x_label="Number of samples (k)", 
    y_label="Score", 
    title="Worst-case HealthBench score at k samples"
)

save_csv_and_print(fail_at_k_data, 'fail_at_k_data.csv')