DOCSEARCHbyAlgolia
Submit Docs

CTranslate2 Documentation

repository·master·Indexed 26 days ago

https://github.com/opennmt/ctranslate2

A high-performance C++ and Python library for efficient inference of Transformer models on CPU and GPU. It supports encoder-decoder, decoder-only, and encoder-only architectures, including Llama, Mistral, T5, and BERT. The library optimizes execution via quantization, layer fusion, and batch reordering, and provides a Python API and the ct2-translator CLI for translation and scoring tasks.

Tokens
17.9K
Snippets
65
Records
101
Agent score
89%

What's inside CTranslate2

  1. Enable dynamic vocabulary reduction with vmap

    master

    Dynamic vocabulary reduction increases translation speed by limiting the target vocabulary to a subset of candidates based on the source N-grams in the current batch.

    1. Generate a vmap file: Create a text file where each line follows the format: src_1 src_2 ... src_N<TAB>tgt_1 tgt_2 ... tgt_K (If the source N-gram is empty, the associated target tokens are always included).
    2. Convert the model: Pass the mapping file to the converter using the --vocab_mapping option.
    3. Translate: Enable the reduction during translation using the use_vmap option.

  2. Stream tokens with generate_tokens

    master

    The generate_tokens method allows you to stream tokens as they are generated, which is useful for interactive environments.

    Important: If you break out of the loop manually, the generation will continue to run in the background. To stop generation early, you must explicitly call step_results.close() on the iterator returned by the method.

    Note that generate_tokens uses the callback argument of generate_batch internally to implement streaming.

    import ctranslate2
import sentencepiece as spm

generator = ctranslate2.Generator("ct2_model/")
sp = spm.SentencePieceProcessor("tokenizer.model")

prompt = "What is the meaning of life?"
prompt_tokens = sp.encode(prompt, out_type=str)

step_results = generator.generate_tokens(
    prompt_tokens,
    sampling_temperature=0.8,
    sampling_topk=20,
    max_length=1024,
)

output_ids = []

for step_result in step_results:
    is_new_word = step_result.token.startswith(" ")

    if is_new_word and output_ids:
        word = sp.decode(output_ids)
        print(word, end=" ", flush=True)
        output_ids = []

    output_ids.append(step_result.token_id)

if output_ids:
    word = sp.decode(output_ids)
    print(word)

  3. Use M2M-100 Fairseq models in CTranslate2

    master

    M2M-100 models require the --fixed_dictionary flag during conversion because they use a single vocabulary file.

    For translation, you must prefix both the source and target sequences with language tokens in the format __X__ (e.g., __en__, __de__). Refer to the end of the model's fixed dictionary file for the list of supported language codes.

    # Conversion example
ct2-fairseq-converter --data_dir . --model_path 418M_last_checkpoint.pt \
    --fixed_dictionary model_dict.128k.txt \
    --output_dir m2m_100_418m_ct2
    import ctranslate2
import sentencepiece as spm

sp = spm.SentencePieceProcessor()
sp.load("spm.128k.model")

source = ["__en__"] + sp.encode("Hello world!", out_type=str)
target_prefix = ["__de__"]

translator = ctranslate2.Translator("m2m_100_418m_ct2")
result = translator.translate_batch([source], target_prefix=[target_prefix])

output = sp.decode(result[0].hypotheses[0][1:])
print(output)

  4. Configure data parallelism for multiple workers

    master

    You can process multiple batches in parallel using inter_threads (to run multiple workers) or device_index (to distribute workers across specific GPUs).

    To enable parallel execution, you must submit batches concurrently using one of the following methods:

    • Calling methods from multiple Python threads (computation methods release the GIL).
    • Calling methods with asynchronous=True.
    • Using file-based or stream-based methods.
    • Setting max_batch_size: the input is split into sub-batches that execute in parallel.

    When workers run on the same device, model weights are shared to save memory.

    # Create a CPU translator with 4 workers each using 1 intra-op thread:
translator = ctranslate2.Translator(model_path, device="cpu", inter_threads=4, intra_threads=1)

# Create a GPU translator with 4 workers each running on a separate GPU:
translator = ctranslate2.Translator(model_path, device="cuda", device_index=[0, 1, 2, 3])

# Create a GPU translator with 4 workers each using a different CUDA stream:
translator = ctranslate2.Translator(model_path, device="cuda", inter_threads=4)

  5. Convert models to CTranslate2 format

    master

    CTranslate2 uses a framework-agnostic core. To use models from other frameworks, you must perform a conversion step that loads the framework-specific model into a unified CTranslate2 representation. This process optionally quantizes weights and saves them into an optimized binary format.

    Supported frameworks include:

    • Fairseq
    • Marian
    • OpenNMT-py
    • OpenNMT-tf
    • OPUS-MT
    • Transformers

    Conversion can be performed using the Python conversion API or dedicated conversion scripts.

  7. Use 4-bit AWQ quantization

    master

    AWQ quantization is supported on NVIDIA GPUs with Compute Capability >= 7.5. In this mode, weights are stored in half precision and layers run in half precision, while scale and zero parameters are stored in int32.

    To use AWQ:

    1. Obtain an AWQ quantized model (e.g., from Hugging Face).
    2. Convert it to CTranslate2 format using ct2-transformers-converter.
    3. Run inference using ctranslate2.Generator.

  9. Implement a custom model converter

    master

    To add support for a new framework, you must write a converter that populates a model specification with trained weights. The architecture must be supported by CTranslate2.

    Model Specification via LayerSpec

    In Python, the model specification is represented using nested LayerSpec objects.

    • Intermediate objects: Define weight scopes.
    • Leaf objects: Define the weight name and value.

    The structure determines the weight names used by the C++ engine. For example, a weight accessed via root.encoder.embeddings.weight in Python will be named encoder/embeddings/weight in the serialized model.

  10. Convert and use GPTBigCode (StarCoder) models

    master

    To convert GPTBigCode models like StarCoder, use the --revision and --quantization flags. For inference, you can use Fill-In-The-Middle (FIM) tokens in your prompt.

    ct2-transformers-converter --model bigcode/starcoder --revision main --quantization float16 --output_dir starcoder_ct2

  11. Convert OPUS-MT models to CTranslate2 format

    master

    OPUS-MT models are Transformer models trained with Marian and are compatible with CTranslate2. You can use the ct2-opus-mt-converter utility to convert these models into the CTranslate2 format. Provide the directory containing the original model files via --model_dir and specify the destination directory via --output_dir.

    ct2-opus-mt-converter --model_dir opus_model --output_dir ct2_model