DOCSEARCHbyAlgolia
Submit Docs

CrossCodeEval

repository·main·Indexed 41 days ago

https://github.com/amazon-science/cceval

A benchmark for evaluating cross-file code completion in diverse and multilingual settings. It provides data and inference scripts for various retrieval settings and models, supporting vLLM for public models (e.g., StarCoder, DeepSeek-Coder), Accelerate for non-vLLM models, and OpenAI API integration.

Tokens
1.9K
Snippets
6
Records
6
Agent score
2%

What's inside CrossCodeEval

  1. Calculate evaluation metrics

    main

    Once generation is complete, use scripts/eval.py to calculate the final metrics. Use the --only_compute_metric flag to skip other processes and focus on metric calculation.

    Required arguments:

    • --prompt_file: Path to the .jsonl prompt file.
    • --output_dir: Directory containing the generated outputs.
    • --ts_lib: Path to the compiled language parser (e.g., ./build/python-lang-parser.so).
    • --language: The programming language.
    • --only_compute_metric: Flag to trigger metric calculation.
    export language=python
export ts_lib=./build/${language}-lang-parser.so; 
export task=line_completion_oracle_unixcoder_cosine_sim
export prompt_file=./data/${language}/${task}.jsonl 
export output_dir=./tmp/crosscodeeval_testrun/;  
python scripts/eval.py \
  --prompt_file $prompt_file \
  --output_dir $output_dir \
  --ts_lib $ts_lib \
  --language $language \
  --only_compute_metric

  2. Generate retrieval augmented prompts

    main

    To generate retrieval augmented prompts, follow these three steps:

    1. Obtain raw repositories: Contact the maintainers via email to request access to the raw software repositories.
    2. Configure the root directory: Set the repository_root environment variable to the absolute path of the directory containing the raw software repositories.
    3. Execute the generation script: Run the provided bash script to start the process.

    Note: Access to the raw data is required before the script can be successfully executed.

    # Example setup (replace with your actual path)
export repository_root="/path/to/your/raw_repositories"
./run.sh

  3. Generate completions using vLLM for public models

    main

    For fast and distributed inference on publicly available models (e.g., StarCoder, DeepSeek-Coder), use the vllm_inference.py script.

    Required environment variables:

    • gpus: Number of GPUs to use.
    • model: The model identifier (e.g., bigcode/starcoder2-3b).
    • language: The programming language (e.g., python).
    • task: The specific task identifier (e.g., line_completion_rg1_unixcoder_cosine_sim).
    • output_dir: Directory to save results.

    Use the --use_crossfile_context flag to enable cross-file context during inference. For additional arguments like top_p or context length, run python vllm_inference.py --help.

    export gpus=2
export model=bigcode/starcoder2-3b
export language=python
export task=line_completion_rg1_unixcoder_cosine_sim
export output_dir=./tmp/crosscodeeval_testrun/
python scripts/vllm_inference.py \
  --tp $gpus \
  --task $task \
  --language $language \
  --model $model \
  --output_dir $output_dir \
  --use_crossfile_context

  4. Generate completions using non-vLLM (Accelerate)

    main

    If you prefer not to use vLLM, you can use the eval.py script with accelerate.

    Prerequisites:

    • Configure accelerate using accelerate config. A reference configuration is available at cceval_config.yaml.
    • Ensure the tree-sitter library (ts_lib) is built for the target language.

    Key Arguments:

    • --model_type: Use codelm_cfc for cross-file context or codelm for no cross-file context.
    • --prompt_file: Path to the .jsonl file corresponding to the retrieval method/setting.
    • --cfc_seq_length: Sequence length for cross-file context.
    • --ts_lib: Path to the compiled language parser (e.g., ./build/python-lang-parser.so).

    Sampling Options: To enable sampling, add the following flags:

    • --do_sample
    • --top_p <float>
    • --temperature <float>
    • --num_return_sequences <int>
    export model_type=codelm_cfc
export model_name=Salesforce/codegen-350M-mono
export language=python
export ts_lib=./build/${language}-lang-parser.so
export dtype=bf16
export prompt_file=./data/crosscodeeval_data/${language}/line_completion_rg1_unixcoder_cosine_sim.jsonl
export max_seq_length=2048
export cfc_seq_length=512 
export batch_size=16
export output_dir=./tmp/crosscodeeval_testrun/

accelerate launch eval.py \
        --model_type $model_type \
        --model_name_or_path $model_name \
        --cfc_seq_length $cfc_seq_length \
        --prompt_file $prompt_file \
        --gen_length 50 \
        --max_seq_length $max_seq_length \
        --batch_size $batch_size \
        --output_dir $output_dir \
        --dtype $dtype \
        --num_return_sequences 1 \
        --overwrite_cache True \
        --ts_lib $ts_lib \
        --language $language

  5. Setup CrossCodeEval environment

    main

    To use CrossCodeEval, you must uncompress the dataset, install Python dependencies, and build the tree-sitter parser libraries.

    1. Uncompress data: Use tar to extract the data into the data/ directory.
    2. Install dependencies: Use pip to install required packages from requirements.txt.
    3. Build tree-sitter: Run the provided shell script to build the language parsers.

    Note: The data contains various settings (baseline, retrieval, retrieval w/ ref.) across different retrievers (bm25, UniXCoder, OpenAI Ada).

    tar -xvJf data/crosscodeeval_data.tar.xz -C data/
pip install -r requirements.txt
bash scripts/build_treesitter.sh

  6. Generate completions using OpenAI models

    main

    OpenAI models are accessed via API using the openai_inference.py script.

    Required environment variables:

    • model: The OpenAI model name (e.g., gpt-3.5-turbo-0125).
    • language: The programming language.
    • task: The task identifier.
    • output_dir: Directory to save results.

    Use the --use_crossfile_context flag to enable cross-file context.

    export model=gpt-3.5-turbo-0125 
export language=python
export task=line_completion_rg1_unixcoder_cosine_sim
export output_dir=./tmp/crosscodeeval_openai_testrun/
python scripts/openai_inference.py \
  --task $task \
  --language $language \
  --model $model \
  --output_dir $output_dir \
  --use_crossfile_context