DOCSEARCHbyAlgolia
Submit Docs

Microsoft Code Pretraining Models

repository·master·Indexed 25 days ago

https://github.com/microsoft/codebert

A collection of Microsoft's code pretraining models, including CodeBERT, GraphCodeBERT, UniXcoder, CodeReviewer, CodeExecutor, and LongCoder. These models support various programming language tasks such as code understanding, generation, execution trace prediction, code search, and code review tasks including quality estimation, comment generation, and code refinement.

Tokens
19.1K
Snippets
72
Records
87
Agent score
83%

What's inside microsoft-codebert

  1. Understand CodeReviewer tasks and data formats

    master

    CodeReviewer supports three primary tasks. All tasks utilize a data format containing old_file (the original code), diff_hunk (the code change), and optionally a comment or target.

    • Quality Estimation (cls): Predicts whether a code change is poor and requires a comment. Input: old_file, diff_hunk.
    • Comment Generation (msg): Generates a review comment for a change. Input: old_file, diff_hunk. Expected output: comment.
    • Code Refinement (ref): Changes the code based on a review comment. Input: old_file, diff_hunk, comment. Expected output: target (the refined code).

    Example data structure:

    {
    "old_file": "import torch",
    "diff_hunk": "@@ -1 +1,2 @@\n import torch\n +import torch.nn as nn",
    "comment": "I don't think we need to import torch.nn here.",
    "target": "import torch"
}
    {
    "old_file": "import torch",  # f1
    "diff_hunk": "@@ -1 +1,2 @@\n import torch\n +import torch.nn as nn",  # f1->f2
    "comment": "I don't think we need to import torch.nn here.",  # requirements for f2->f3
    "target": "import torch"  # f3
}

  4. Run evaluation and testing for Code Generation

    master

    To test the fine-tuned model on the test set, use run.py with the --do_test flag. Prediction results are saved to saved_models/predictions.txt. To obtain official scores, you must send these predictions to codexglue@microsoft.com.

    python run.py \
	--do_test \
	--model_name_or_path microsoft/unixcoder-base \
	--test_filename dataset/test.json \
	--output_dir saved_models \
	--max_source_length 350 \
	--max_target_length 150 \
	--beam_size 3 \
	--train_batch_size 32 \
	--eval_batch_size 32 \
	--learning_rate 5e-5 \
	--gradient_accumulation_steps 1 \
	--num_train_epochs 30

  6. Download and prepare Code Summarization data

    master

    To prepare the dataset for code summarization, download the CodeXGLUE dataset, extract the language-specific archives (Python, Java, Ruby, JavaScript, Go, PHP), and run the preprocessing script. Note that the preprocessing script preprocess.py should be run from within the dataset directory after downloading the files.

    wget https://github.com/microsoft/CodeXGLUE/raw/main/Code-Text/code-to-text/dataset.zip
unzip dataset.zip
rm dataset.zip
cd dataset
wget https://zenodo.org/record/7857872/files/python.zip
wget https://zenodo.org/record/7857872/files/java.zip
wget https://zenodo.org/record/7857872/files/ruby.zip
wget https://zenodo.org/record/7857872/files/javascript.zip
wget https://zenodo.org/record/7857872/files/go.zip
wget https://zenodo.org/record/7857872/files/php.zip

unzip python.zip
unzip java.zip
unzip ruby.zip
unzip javascript.zip
unzip go.zip
unzip php.zip
rm *.zip
rm *.pkl

python preprocess.py
rm -r */final
cd ..

  9. Install CodeReviewer dependencies

    master

    Install the required dependencies using conda to set up the environment for CodeReviewer:

    conda install nltk
conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch
conda install transformers
    conda install nltk
conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch
conda install transformers

  10. Initialize the UniXcoder model

    master

    First, download the unixcoder.py implementation file. Then, you can initialize the model using the UniXcoder class. You can specify different pre-trained weights such as microsoft/unixcoder-base.

    import torch
from unixcoder import UniXcoder

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = UniXcoder("microsoft/unixcoder-base")
model.to(device)

  12. Fine-tune CodeBERT for Code Search

    master

    Fine-tune a language-specific model for code search using run_classifier.py. You can specify the programming language, the pretrained model (e.g., microsoft/codebert-base or roberta-base), and training hyperparameters. The script requires a data_dir containing the preprocessed training and validation files for the target language.

    cd codesearch

lang=php #fine-tuning a language-specific model for each programming language 
pretrained_model=microsoft/codebert-base  #Roberta: roberta-base

python run_classifier.py \
--model_type roberta \
--task_name codesearch \
--do_train \
--do_eval \
--eval_all_checkpoints \
--train_file train.txt \
--dev_file valid.txt \
--max_seq_length 200 \
--per_gpu_train_batch_size 32 \
--per_gpu_eval_batch_size 32 \
--learning_rate 1e-5 \
--num_train_epochs 8 \
--gradient_accumulation_steps 1 \
--overwrite_output_dir \
--data_dir ../data/codesearch/train_valid/$lang \
--output_dir ./models/$lang  \
--model_name_or_path $pretrained_model