CodeGeeX Documentation

HumanEval-X is a multilingual code generation benchmark designed to measure the functional correctness of generated code across multiple languages. It contains 820 high-quality hand-written samples covering Python, C++, Java, JavaScript, and Go.

Supported Tasks

• Code Generation: Input consists of a function declaration and a docstring; the output is the function implementation.
• Code Translation: Input consists of function declarations in two languages and the implementation in the source language; the output is the implementation in the target language.

Evaluation Metric

It uses the unbiased pass@k metric (as used in Codex) where $n=200$ and $k \in {1, 10, 100}$: $$\text{pass}@k := \mathbb{E}[1-\frac{\binom{n-c}{k}}{\binom{n}{k}}]$$

CodeGeeX is a 13 billion parameter multilingual code generation model based on a transformer architecture. It functions as a left-to-right autoregressive decoder that predicts the next token based on code and natural language input.

Key Specifications:

• Layers: 40 transformer layers
• Hidden Size (Self-attention): 5,120
• Hidden Size (Feed-forward): 20,480
• Max Sequence Length: 2,048
• Vocabulary Size: 50,400 tokens (using a GPT-2 based tokenizer with whitespace as extra tokens)
• Training Data: 158.7B tokens across 23 programming languages.

HumanEval-X is a benchmark designed to evaluate the functional correctness of multilingual code generation models. It contains 820 high-quality samples across Python, C++, Java, JavaScript, and Go.

Supported Tasks:

• Code Generation: Input consists of a function declaration and a docstring to generate the implementation (solution).
• Code Translation: Input consists of declarations in both source and target languages, plus the solution in the source language, to generate the solution in the target language.

For detailed usage instructions and the standard format for contributing new problems, refer to the codegeex/benchmark/README.md file.

The evaluation requires specific versions of various compilers and runtimes (Python 3.8.12, JDK 18.0.2.1, Node.js 16.14.0, C++ 11, g++ 7.5.0, Boost 1.71.0, OpenSSL 3.0.0, and Go 1.18.4). To simplify setup, use the provided Docker image.

Option 1: Pull pre-built image

docker pull rishubi/codegeex:latest

Option 2: Build from Dockerfile

If you need to customize the environment, build the image from the repository's Dockerfile:

cd codegeex/docker
docker build [OPTIONS] .

Run the Container

Use the following command to create a container with GPU support and bind mounts:

docker run -it --gpus all --mount type=bind,source=<LOCAL PATH>,target=<PATH IN CONTAINER> [OPTIONS] <IMAGE NAME:TAG>

To use the CodeGeeX API, you must first apply for access through the Tianqi API Open Platform.

Follow these steps to create an API application:

1. Access the Tianqi platform experience entry from the homepage.
2. Navigate to 'API 应用' (API Applications).
3. Enter an arbitrary name to create a new API application.
4. Once created, you will receive an API Key and API Secret, which are required for all API calls.
5. You can find the request URLs for code generation and code translation, along with detailed usage documentation, within the API information section of the platform.

To use CodeGeeX, search for "codegeex" in the VS Code Marketplace and install it.

Requirements:

• VS Code version must be 1.68.0 or higher.

CodeGeeX is available as a free plugin for several IDEs:

VS Code

Search for "codegeex" in the VS Code Marketplace.

JetBrains

Search for "codegeex" in the JetBrains Plugins market.

• Requirement: IDE version 2021.1 or higher.
• Supported IDEs: IntelliJ IDEA, PyCharm, GoLand, CLion, Android Studio, AppCode, Aqua, DataSpell, DataGrip, Rider, RubyMine, WebStorm.

Interactive Mode allows you to view multiple code candidates in a side window.

1. Press Ctrl + Enter to activate the mode.
2. CodeGeeX will generate X candidates (where X is defined by the Candidate Num setting).
3. The candidates will appear in a side window. Click use code above a candidate to insert it at your current cursor position.

Model weights must be requested via this link. Once approved, you will receive a urls.txt file containing temporary download links.

It is recommended to use aria2 for downloading. Ensure you have approximately 26GB of disk space available. After downloading the split files, you must concatenate and extract them.

# Download using aria2
aria2c -x 16 -s 16 -j 4 --continue=true -i urls.txt 

# Concatenate and extract
cat codegeex_13b.tar.gz.* > codegeex_13b.tar.gz
tar xvf codegeex_13b.tar.gz

After obtaining your API Key and API Secret, you can use the provided implementation examples to integrate the API into your projects:

• For Python implementations, refer to the directory: api/codegeex-api-example-python
• For Java implementations, refer to the project: api/codegeex-api-example-java

CodeGeeX is a multilingual code generation model. To use it in VS Code:

1. Ensure your VS Code version is >= 1.68.0.
2. Search for "codegeex" in the VS Code Marketplace and install the extension.
3. Activate the extension globally to enable its four primary modes: Invisble Mode, Interactive Mode, Translation Mode, and Prompt Mode (experimental).

Prompt mode uses few-shot prompting to perform tasks like code explanation, summarization, or style enforcement.

• Trigger: Select the code and press Alt/Option + t.
• Action: Choose a template from the list to generate the output.
• Customization: You can add your own templates in the extension settings under Prompt Templates. Each template entry requires a key (the name shown in the list) and a value (the file path to the template, e.g., .txt, .py).