DOCSEARCHbyAlgolia
Submit Docs

HumanEval Documentation

repository·master·Indexed 25 days ago

https://github.com/openai/human-eval

An evaluation harness for the HumanEval problem-solving dataset used to measure the ability of large language models to solve coding tasks. Includes instructions for installation, generating completions in JSON Lines format, and evaluating functional correctness using the evaluate_functional_correctness command.

Tokens
689
Snippets
2
Records
4
Agent score
36%

What's inside HumanEval

  1. Install HumanEval

    master

    Ensure you are using Python 3.7 or later. You can set up a dedicated environment using conda and install the repository in editable mode using pip.

    $ conda create -n codex python=3.7
$ conda activate codex

$ git clone https://github.com/openai/human-eval
$ pip install -e human-eval

  2. Generate HumanEval samples

    master

    To evaluate a model, you must first generate completions and save them in a JSON Lines (jsonl) format. Each line must contain a task_id and the completion (the code generated without the prompt).

    Format:

    {"task_id": "Corresponding HumanEval task ID", "completion": "Completion only without the prompt"}

    Example usage with human_eval.data utilities:

    from human_eval.data import write_jsonl, read_problems

problems = read_problems()

num_samples_per_task = 200
samples = [
    dict(task_id=task_id, completion=generate_one_completion(problems[task_id]["prompt"]))
    for task_id in problems
    for _ in range(num_samples_per_task)
]
write_jsonl("samples.jsonl", samples)

  3. Troubleshoot memory errors in HumanEval

    master
    If you encounter the error malloc: can't allocate region, it indicates the system is running out of RAM. Because this can cause correct programs to fail the evaluation, it is recommended to free up system memory and retry the evaluation.

  4. Evaluate functional correctness

    master

    Use the evaluate_functional_correctness command to run the test suites against your generated samples.

    Security Warning: This program executes untrusted model-generated code. It is strongly recommended to run this within a robust security sandbox. The execution call in human_eval/execution.py is commented out by default to prevent accidental unsafe execution; you must enable it manually after reading the security disclaimers in that file.

    Command Syntax: evaluate_functional_correctness <samples_path> [--problem_file=<path>] [--k=<comma-separated-values>]

    • <samples_path>: The path to your .jsonl samples file.
    • --problem_file: Path to the problem file (required if using custom problems).
    • --k: Specify other $k$ values for pass@k estimation (e.g., --k=1,10,100).

    Results are written to a new file named <input_path>_results.jsonl. Each row in the results file contains:

    • passed: Boolean indicating if the completion passed.
    • result: One of "passed", "timed out", or "failed".