DOCSEARCHbyAlgolia
Submit Docs

Code Llama

repository·main·Indexed 12 days ago

https://github.com/meta-llama/codellama

A family of large language models based on Llama 2 architecture, optimized for code generation, infilling, and instruction following. Available in 7B, 13B, 34B, and 70B parameter sizes across three variants: base Code Llama, Code Llama - Python, and Code Llama - Instruct. Supports context windows up to 100K tokens at inference time.

Tokens
1.4K
Snippets
6
Records
9
Agent score
47%

What's inside Code Llama

  1. Understand Code Llama architecture and context window

    main

    Code Llama models are autoregressive language models using optimized transformer architectures.

    Key technical capabilities:

    • Infilling: The 7B, 13B, and 70B models support infilling text generation.
    • Context Window: Most models (excluding Code Llama - Python 70B and Code Llama - Instruct 70B) were fine-tuned with up to 16K tokens and support up to 100K tokens at inference time.
    • Input/Output: Models process and generate text only.

  2. Select a Code Llama model variant

    main

    Code Llama is available in three distinct variants depending on your use case:

    1. Code Llama: Base models designed for general code synthesis and understanding.
    2. Code Llama - Python: Specialized models designed specifically for the Python programming language.
    3. Code Llama - Instruct: Models fine-tuned for instruction following, intended for safer deployment in code assistance and generation applications.

    All variants are available in the following parameter sizes: 7B, 13B, 34B, and 70B.

  3. Review Code Llama licensing and usage terms

    main

    Code Llama is released under a custom commercial license. You can find the full details at: https://ai.meta.com/resources/models-and-libraries/llama-downloads/.

    Intended Use:

    • Commercial and research use in English and relevant programming languages.
    • Base models can be adapted for various synthesis/understanding tasks.
    • Python variant for Python-specific tasks.
    • Instruct variant for safer code assistance.

    Out-of-Scope Uses:

    • Use in any manner violating laws or regulations.
    • Use in languages other than English.
    • Any use prohibited by the Acceptable Use Policy and Licensing Agreement.

  4. Download Code Llama model weights

    main

    To download model weights and tokenizers, you must first request access via the Meta website and accept the license. Once approved, you will receive a signed URL via email.

    To perform the download:

    1. Ensure wget and md5sum are installed.
    2. Run the provided download script: bash download.sh.
    3. When prompted, paste the signed URL.

    Important: Copy the raw URL text. Do not use the 'Copy link address' context menu option. A valid URL should start with https://download.llamameta.net. If it starts with https://l.facebook.com, the copy failed.

    Note: Links expire after 24 hours or a limited number of downloads. If you encounter a 403: Forbidden error, re-request a link.

    bash download.sh

  5. Install Code Llama repository

    main

    To set up the repository for inference, use a conda environment with PyTorch and CUDA available. Clone the repository and install the package in editable mode from the top-level directory.

    pip install -e .

  6. Run Code Infilling with Code Llama

    main

    The 7B and 13B variants of Code Llama and Code Llama - Instruct support infilling (filling in code based on surrounding context). Use example_infilling.py to implement this.

    Example running CodeLlama-7b for infilling:

    torchrun --nproc_per_node 1 example_infilling.py \
    --ckpt_dir CodeLlama-7b/ \
    --tokenizer_path CodeLlama-7b/tokenizer.model \
    --max_seq_len 192 --max_batch_size 4

  7. Run inference with Pretrained Code Models

    main

    Pretrained models (CodeLlama-{size} and CodeLlama-{size}-Python) are not fine-tuned for instructions. They function via text completion; prompt them so that the desired output is the natural continuation of the input.

    When running inference, you must set --nproc_per_node to the Model Parallel (MP) value corresponding to your model size:

    • 7B: MP 1
    • 13B: MP 2
    • 34B: MP 4
    • 70B: MP 8

    Example running CodeLlama-7b using example_completion.py:

    torchrun --nproc_per_node 1 example_completion.py \
    --ckpt_dir CodeLlama-7b/ \
    --tokenizer_path CodeLlama-7b/tokenizer.model \
    --max_seq_len 128 --max_batch_size 4

  8. Use Fine-tuned Instruction Models

    main

    The Code Llama - Instruct models are designed to follow instructions. To achieve optimal performance, you must follow a specific prompt format involving INST and <<SYS>> tags, BOS and EOS tokens, and specific whitespace/linebreaks.

    Recommended Approach: Use the chat_completion() function directly. It automatically handles the required formatting for all instruct models.

    Note for 70B: CodeLlama-70b-Instruct requires a separate turn-based prompt format defined in dialog_prompt_tokens().

    Example running CodeLlama-7b-Instruct using example_instructions.py:

    torchrun --nproc_per_node 1 example_instructions.py \
    --ckpt_dir CodeLlama-7b-Instruct/ \
    --tokenizer_path CodeLlama-7b-Instruct/tokenizer.model \
    --max_seq_len 512 --max_batch_size 4

  9. Reference: Model Parallel (MP) values and Sizes

    main

    The following table maps model sizes to the required Model Parallel (MP) values for inference and provides approximate weight sizes.

    | Model | Size     | MP |
|-------|----------|----|
| 7B    | ~12.55GB | 1  |
| 13B   | 24GB     | 2  |
| 34B   | 63GB     | 4  |
| 70B   | 131GB    | 8  |