CRUXEval Benchmark

The dataset contains 800 Python functions and input-output pairs for two tasks: CRUXEval-I (input prediction) and CRUXEval-O (output prediction).

• Local file: data/cruxeval.jsonl
• HuggingFace: cruxeval-org/cruxeval

Each sample includes code, input, and output fields.

Depending on your use case, install the appropriate requirements file:

• Full installation: pip install -r requirements.txt
• Scoring only: pip install -r requirements-base.txt
• OpenAI models: pip install -r requirements-openai.txt
• HuggingFace inference: pip install -r requirements-inference.txt

Tested with Python 3.9 and CUDA 12.1.

To evaluate model outputs, first prepare a JSON file (e.g., generations.json) where keys are sample IDs and values are lists of strings representing the model's predictions.

Format for CRUXEval-I (Input Prediction):

{
    "sample_0": ["f([1, 1, 1, 1, 3, 3])", "f([])"],
    "sample_799": ["f('~neqe-;ew22')", "f('~neqe-;ew22')"]
}

Evaluation Steps:

1. Navigate to the evaluation directory.
2. Run evaluate_generations.py using the --mode flag:
   • Use --mode input for CRUXEval-I.
   • Use --mode output for CRUXEval-O.

Example Command:

python evaluate_generations.py \
    --generations_path generations.json \
    --scored_results_path generations_scored.json \
    --mode input

The output file at --scored_results_path will contain raw_generations, raw_scored_generations, and the overall pass_at_1 and pass_at_5 scores.

To evaluate multiple models in parallel using SLURM:

1. Use the scripts in the evaluation directory:
   • evaluate_all_predictions_input.sh (for CRUXEval-I)
   • evaluate_all_predictions_output.sh (for CRUXEval-O)
2. You may need to update --output, --error, --partition, and run_names.
3. Use evaluation/print_evaluation_directories.py to automatically find directories in model_generations to populate run_names.
4. To view results in a table, run evaluation/read_results.py.

Results (including pass@1 and pass@5) are stored in evaluation/evaluation_results.

python evaluation/read_results.py

To run inference using OpenAI models:

1. Set your API key in the environment:

   export OPENAI_API_KEY=YOUR_KEY

2. Navigate to the openai directory.
3. Run the inference script:

   python openai_run.py

Generations will be saved to ../model_generations/{MODEL_INFO}/generations.json.

export OPENAI_API_KEY=YOUR_KEY
cd openai
python openai_run.py

To use CRUXEval, clone the repository and navigate to the directory:

git clone git@github.com:facebookresearch/cruxeval.git
cd cruxeval

You can load the CRUXEval dataset locally by reading the data/cruxeval.jsonl file. Each line in the file is a JSON object containing the code, input, and output fields required for evaluation.

import json
data = [json.loads(l) for l in open("data/cruxeval.jsonl", "r").readlines()]

Alternatively, you can load the dataset directly from the HuggingFace Hub using the datasets library. The evaluation data is located in the test split of the cruxeval-org/cruxeval dataset.

from datasets import load_dataset
data = load_dataset("cruxeval-org/cruxeval")["test"]

To evaluate if a piece of code correctly handles a specific input and produces the expected output, use the check_correctness function from evaluation.utils_execute.

Construct a test string by appending an assert statement to the source code. The format should be: {code}\nassert f({input}) == {output}.

Pass this string to check_correctness along with a timeout value (in seconds) to prevent infinite loops during execution.

from evaluation.utils_execute import check_correctness

# Example construction
code_to_check = f"{code}\nassert f({input}) == {output}"

# Execute verification
result = check_correctness(code_to_check, timeout=3)
print(result)