BigCodeBench

BigCodeBench provides different ways to slice the benchmark tasks based on the model type and difficulty:

Splits

• complete: Designed for code completion tasks using comprehensive docstrings.
• instruct: Designed for instruction-tuned and chat models. Models generate code based on natural language instructions requiring complex reasoning.

Subsets

• full: The complete set of tasks.
• hard: A subset of 148 tasks aligned with real-world programming tasks.

If you want to work with the source code directly, clone the repository, add the current directory to your PYTHONPATH, and install it in editable mode.

git clone https://github.com/bigcode-project/bigcodebench.git
cd bigcodebench
export PYTHONPATH=$PYTHONPATH:$(pwd)
pip install -e .

To use BigCodeBench, you should set up an isolated environment. You need two sets of dependencies: one for the core library and one for local evaluation.

1. Install evaluation requirements: Required if you intend to run evaluations locally.
2. Install BigCodeBench: The main library.

To install the nightly version, use the git URL directly.

# Install evaluation requirements in an isolated environment
pip install -I -r https://raw.githubusercontent.com/bigcode-project/bigcodebench/main/Requirements/requirements-eval.txt

# Install BigCodeBench dependencies
pip install bigcodebench --upgrade

# OR: Install nightly version
pip install "git+https://github.com/bigcode-project/bigcodebench.git" --upgrade

To use BigCodeBench for evaluation, install the package via pip. It is also recommended to install flash-attn for code sample generation. If you encounter installation issues with flash-attn, consider using pre-built wheels from the Dao-AILAB releases.

To install the nightly version (required to use bigcodebench.generate), install directly from the GitHub repository.

# Standard installation
pip install bigcodebench --upgrade

# Recommended for generating code samples
pip install packaging ninja
pip install flash-attn --no-build-isolation

# Install nightly version to use bigcodebench.generate
pip install "git+https://github.com/bigcode-project/bigcodebench.git" --upgrade

BigCodeBench provides a script to replicate performance analysis, including Elo Rating and Task Solve Rate.

Steps to run analysis:

1. Place all samples_eval_results.json files into a folder named results.
2. Ensure the results folder is in the same directory as the analysis script.
3. Run the script:

cd analysis
python get_results.py

To use specific model backends for evaluation, you must set the corresponding API keys in your environment variables.

export E2B_API_KEY=<your_e2b_api_key>
export OPENAI_API_KEY=<your_openai_api_key>
export ANTHROPIC_API_KEY=<your_anthropic_api_key>
export MISTRAL_API_KEY=<your_mistral_api_key>
export GOOGLE_API_KEY=<your_google_api_key>
export HF_INFERENCE_API_KEY=<your_hf_api_key>

Memory/Import Errors

If you encounter ImportError: ... failed to map segment from shared object (specifically related to matplotlib), this is likely due to the Docker container's memory limit. Increase the container's memory limit using the Docker flags mentioned in the Docker guide.

Tokenizer Issues

If you notice unexpected behavior due to Hugging Face tokenizer updates, try using the --tokenizer_legacy flag during generation.

Network/Proxy Issues

If you are behind a proxy and cannot access the internet, use Remote Evaluation instead of local evaluation to ensure accurate results.

Performance/Timeout

• LLM solutions are marked as failed on timeout or OOM (Out of Memory).
• Dynamic timeouts are set based on the ground-truth solution's runtime.
• Avoid over-stressing the machine (e.g., using --parallel 64 on a 4-core machine) to prevent false timeouts.

To understand why certain samples failed, use the bigcodebench.inspect command. This will analyze the evaluation results and save the inspection details to an inspect/ directory.

Inspect results:

bigcodebench.inspect --eval_results sample-sanitized-calibrated_eval_results.json --split complete --subset hard

Re-run inspection in place:

bigcodebench.inspect --eval_results sample-sanitized-calibrated_eval_results.json --split complete --subset hard --in_place

The bigcodebench.evaluate command accepts the following arguments:

• --model: The model identifier (e.g., meta-llama/Meta-Llama-3.1-8B-Instruct).
• --execution: The execution environment. Options: [e2b|gradio|local].
• --split: The task split. Options: [complete|instruct].
• --subset: The task subset. Options: [full|hard].
• --backend: The model provider/backend. Options: [vllm|openai|anthropic|google|mistral|hf|hf-inference].
• --bs: Batch size (use 1 for more deterministic greedy decoding).
• --direct_completion: Use this for base models that have a tokenizer.chat_template to avoid chat-mode evaluation.