DOCSEARCHbyAlgolia
Submit Docs

code-docstring-corpus

repository·master·Indexed 42 days ago

https://github.com/edinburghnlp/code-docstring-corpus

A collection of preprocessed Python function triples (declarations, docstrings, and bodies) for training and evaluating models on automated code documentation (code2doc) and code generation (doc2code) tasks. Includes a main parallel corpus, code-only corpora, and a V2 version featuring class declarations, module docstrings, and commit SHAs.

Tokens
1.7K
Snippets
2
Records
11
Agent score
54%

What's inside code-docstring-corpus

  1. Overview of the code-docstring-corpus datasets

    master

    The corpus provides preprocessed Python function declarations, docstrings, and bodies for code documentation (code2doc) and code generation (doc2code) tasks.

    Available Corpora

    directorydescription
    parallel-corpusMain parallel corpus (150,370 triples). Includes a canonical split (109,108 training, 2,000 validation, 2,000 test). Contains reassembled pairs for specific tasks: (declaration+body, docstring) for code documentation and (declaration+docstring, body) for code generation.
    code-only-corpusA code-only corpus of 161,630 pairs of function declarations and function bodies.
    backtranslations-corpusDocstrings automatically generated from the code-only-corpus using NMT for data augmentation.
    nmt-outputsTest and validation outputs from baseline NMT models.
    repo_split.parallel-corpusAn alternate train/validation/test split where no repository is shared between sets (repository-consistent).
    repo_split.code-only-corpusA repository-consistent filtered version of the code-only corpus.
    V2Version 2 of the corpus, which includes class declarations, class methods, module docstrings, and commit SHAs.
    scriptsPreprocessing scripts used to generate the corpora.

  2. Overview of code-docstring-corpus V2

    master
    The code-docstring-corpus V2 is a dataset designed for automated code documentation and code generation tasks. It provides a large-scale collection of Python code elements paired with their corresponding docstrings. The V2 version specifically includes class declarations, class methods, module docstrings, and commit SHAs to ensure data integrity and context.

  3. Understand the metadata format for the main parallel corpus

    master

    The main parallel corpus uses specific metadata formats to identify the location and declaration lines of code elements. This metadata is used to map code snippets to their source in GitHub.

    • For top-level functions: The format is github/[repo]_[commit_SHA]/[file_path] [declaration_line]
    • For class methods: The format is github/[repo]_[commit_SHA]/[file_path] [declaration_line] [class_declaration_line]

  4. Explore the structure of the code-docstring-corpus V2

    master

    The corpus is organized into several directories depending on whether you need parallel (code-docstring pairs) or monolingual (code-only) data, or specific data splits for machine learning training:

    • classes_and_modules: Contains class declarations and docstrings, as well as method docstrings.
    • parallel: The main parallel corpus containing paired code and documentation.
    • mono: A monolingual code-only corpus containing 223,033 top-level functions and 1,121,304 class methods.
    • repo_split: Provides repository-consistent train-valid-test splits of the main parallel corpus and the filtered code-only repository corresponding to the training repositories.

  5. Handle 'd' and 'q' escapes in the data_ps.all dataset

    master

    The data_ps.all dataset file uses d' and 'd as markers for the start and end of descriptions. To prevent these markers from interfering with the data, the following escaping convention is applied:

    1. Escape d: All instances of the letter d are replaced with qz.
    2. Escape q: To ensure the process is reversible, all instances of q are replaced with qq.

    To restore the original text after splitting the dataset into columns, you must reverse these transformations using the 'd and d' tokens.

  6. Understand special tokens in the parallel corpus

    master

    The dataset uses specific special tokens to preserve Python's structural whitespace and line breaks, which are often collapsed by standard seq2seq tokenizers.

    • DCNL: Represents a newline.
    • DCSP: Represents a space used in leading indentation (one token per nesting level) or inside string constants.

    These tokens ensure that multi-line code snippets and semantic whitespace (like indentation) are preserved during training and inference.

  7. Install the code-docstring-corpus dependencies

    master

    To use the preprocessing scripts or run the corpus, install the required Python dependencies using pip:

    pip install -r requirements.txt

    Additional External Requirements:

  8. Understand the schema for class and module information

    master

    The corpus provides structured information about classes and modules using specific keys. Use these keys to access declarations, descriptions, and metadata:

    • class_decl: Contains the actual class declarations.
    • class_desc: Contains class docstrings or DCNA (Document Content Not Available) if a docstring is missing.
    • class_meta: Contains class metadata. The format is github/[repo]_[commit_SHA]/[file_path] [declaration_line] [parent_declaration_line]. For top-level classes, parent_declaration_line is set to -1; for nested classes, it indicates the line of the parent declaration.
    • module_desc: Contains module docstrings or DCNA if not present.
    • module_meta: Contains module metadata. The format is github/[repo]_[commit_SHA]/[file_path].
    |file | description |
|---  |---          |
|class_decl | Class declarations |
|class_desc | Class docstrings or DCNA if not present |
|class_meta | Class metadata. Format: `github/[repo]_[commit_SHA]/[file_path] [declaration_line] [parent_declaration_line]` |
|module_desc | Module docstrings or DCNA if not present|
|module_meta | Module metadata. Format: `github/[repo]_[commit_SHA]/[file_path]`|

  10. Baseline NMT model results

    master

    Baseline Neural Machine Translation models were trained using Nematus on sub-tokenized data from the parallel-corpus. The following BLEU scores represent the baseline performance for the code2doc and doc2code tasks:

    ModelValidation BLEUTest BLEU
    declbodies2desc.baseline14.0313.84
    decldesc2bodies.baseline10.3210.24
    decldesc2bodies.backtransl10.8510.90

    BLEU scores are computed using the Moses multi-bleu.perl script.