RepoBench

RepoBench focuses on next-line prediction tasks to benchmark repository-level code auto-completion systems.

Datasets are categorized by prompt length (number of tokens) using OpenAI's GPT-4 tokenizer via tiktoken. The following levels are used to organize data points:

RepoBench evaluates code auto-completion across three distinct settings for each task:

• cross_file_first: Masks the line where a module from a different file is used for the first time.
• cross_file_random: Masks a random line where a module from a different file is used (excluding the first usage).
• in_file: Masks a random line that has no cross-file dependency.

You can load the RepoBench v1.1 datasets directly using the datasets library. The datasets are available for both Python and Java on the Hugging Face Hub.

from datasets import load_dataset

dataset = load_dataset("tianyang/repobench_python_v1.1", ignore_verifications=True)

To use RepoBench, clone the repository and navigate to the directory. If you intend to reproduce the results presented in the paper, install the dependencies listed in requirements.txt. If you only need to access the data, you can skip the dependency installation.

git clone https://github.com/Leolty/repobench.git
cd repobench

You can load the RepoBench v1.1 datasets (Python and Java) directly from the HuggingFace Hub using the datasets library. These datasets include data collected from GitHub between October 6, 2023, and December 31, 2023, and have undergone deduplication to mitigate data leakage.

from datasets import load_dataset

# Load the Python dataset
python_dataset = load_dataset("tianyang/repobench_python_v1.1")

# Load the Java dataset
java_dataset = load_dataset("tianyang/repobench_java_v1.1")

If you use RepoBench in your research, please cite the following paper from ICLR 2024:

@misc{liu2023repobench,
      title={RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems}, 
      author={Tianyang Liu and Canwen Xu and Julian McAuley},
      year={2024},
      url={https://arxiv.org/abs/2306.03091},
      booktitle={International Conference on Learning Representations}
}

After generating completions, use eval.py to calculate performance metrics. The script computes Exact Match (EM), Edit Similarity (ES), and CodeBLEU (CB) scores for each setting (cross_file_first, cross_file_random, in_file) and provides weighted averages across all levels.

python eval.py --path "results/deepseek-coder-1.3b-base-python" --language "python"

Use the run.py script to run experiments on the RepoBench v1.1 dataset. The script utilizes the 🤗 Transformers library. You can customize parameters such as model name, dataset name, language, and context window levels.

CUDA_VISIBLE_DEVICES=0 python run.py --model_name "deepseek-ai/deepseek-coder-1.3b-base" \
               --dataset_name "tianyang/repobench_python_v1.1" \
               --start_date "2023-12-01" \
               --end_date "2023-12-31" \
               --language "python" \
               --max_token_nums 15800 \
               --levels "2k" "4k" "8k" "12k" "16k" \
               --temperature 0.2 \
               --top_p 0.95 \
               --max_new_tokens 128 \
               --batch_size 1