Yi Documentation

The Yi series are bilingual (English/Chinese) large language models developed by 01.AI. The series includes base models in two primary parameter sizes:

• 6B: Available as Yi-6B (4K sequence length, extendable to 32K during inference) and Yi-6B-200K (200K context length).
• 34B: Available as Yi-34B (4K sequence length, extendable to 32K during inference) and Yi-34B-200K (200K context length).

The Yi series consists of next-generation open-source bilingual large language models trained from scratch by 01.AI. These models are trained on a 3T multilingual corpus and excel in language cognition, common sense reasoning, and reading comprehension.

Key characteristics:

• Bilingual Capability: Strong performance in both Chinese and English.
• Architecture: Uses the Transformer architecture, specifically following the Llama architecture style for compatibility with existing tools and libraries, though they are not derivatives of Llama and do not use Llama weights.
• Performance: Models like Yi-34B-Chat have demonstrated performance comparable to GPT-4 Turbo on benchmarks like AlpacaEval.

The Yi series consists of next-generation, open-source, bilingual (English/Chinese) large language models developed by 01.AI. These models are trained from scratch on a 3T multilingual corpus and are designed for high performance in language understanding, commonsense reasoning, and reading comprehension.

Key characteristics:

• Architecture: Based on the Transformer structure and adopts the Llama model architecture, ensuring compatibility with the existing AI ecosystem tools.
• Independence: While using the Llama architecture, Yi models are not derivatives of Llama; they use entirely independent training datasets, pipelines, and infrastructure.
• Performance: Models like Yi-34B-Chat have demonstrated competitive performance against models like GPT-4 Turbo on benchmarks such as AlpacaEval.

To run Yi models using 4-bit integer quantization, you need to build the llama.cpp library. If you haven't built it yet, clone the repository and use make to compile it.

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make

To perform AWQ quantization, use the swift export command. AWQ requires a dataset for the quantization process. You can specify multiple datasets (e.g., alpaca-zh, alpaca-en, sharegpt-gpt4:default).

After quantization, you can perform inference using the swift infer command by pointing to the newly created quantized model path.

# Quantize using AWQ
CUDA_VISIBLE_DEVICES=0 swift export \
    --model_type yi-1_5-6b-chat --quant_bits 4 \
    --dataset alpaca-zh alpaca-en sharegpt-gpt4:default --quant_method awq

# Infer using the quantized model
CUDA_VISIBLE_DEVICES=0 swift infer \
    --model_type yi-1_5-6b-chat \
    --model_id_or_path yi-1_5-6b-chat-awq-int4

Use the swift infer command to interact with your fine-tuned model.

Arguments:

• --ckpt_dir: The directory of the merged model checkpoint.
• --eval_human: Set to true to enable interactive human-machine mode.
• --stop_words: A string used to control output length.
• --infer_backend: The inference engine to use (e.g., pt for PyTorch).

swift infer \
    --ckpt_dir /content/yi-1-5-6b-chat/v0-20240717-024536/checkpoint-682-merged \
    --eval_human true \
    --stop_words "Observation:" \
    --infer_backend pt

To run Yi-34B-Chat locally using Docker (requires NVIDIA A800 GPU or 4*4090), ensure docker and nvidia-container-toolkit are installed. Mount your model path to /models inside the container.

docker run -it --gpus all \
-v <your-model-path>: /models \
ghcr.io/01-ai/yi:latest

GPT-Q is a Post-Training Quantization (PTQ) method. Use the quant_autogptq.py script to perform quantization and eval_quantized_model.py to verify the results.

# Perform quantization
python quantization/gptq/quant_autogptq.py \
  --model /base_model \
  --output_dir /quantized_model \
  --bits 4 --group_size 128 --trust_remote_code

# Evaluate the quantized model
python quantization/gptq/eval_quantized_model.py \
  --model /quantized_model \
  --trust_remote_code

After fine-tuning, you can perform direct inference using the checkpoint directory. Alternatively, you can merge the LoRA weights into the base model using swift export with --merge_lora true and then run inference on the resulting merged checkpoint.

# Direct inference
CUDA_VISIBLE_DEVICES=0 swift infer \
    --ckpt_dir output/yi-vl-6b-chat/vx-xxx/checkpoint-xxx \
    --load_dataset_config true

# Merge LoRA and then inference
CUDA_VISIBLE_DEVICES=0 swift export \
    --ckpt_dir output/yi-vl-6b-chat/vx-xxx/checkpoint-xxx \
    --merge_lora true

CUDA_VISIBLE_DEVICES=0 swift infer \
    --ckpt_dir output/yi-vl-6b-chat/vx-xxx/checkpoint-xxx-merged \
    --load_dataset_config true

LM Studio is a desktop application used to experiment with and deploy local open-source large language models (LLMs).

To run Yi models locally:

1. Download and install LM Studio from the official website based on your operating system.
2. Open LM Studio and use the search bar to find models such as yi1.5-6b-chat.
3. LM Studio will provide an assessment of whether your local hardware can run the selected model to prevent memory issues.
4. Select the desired model and click "download" to begin using it locally.

To use the yi-large model, configure the OpenAI client with the following settings:

• api_key: Your actual API key.
• base_url: https://api.01.ai/v1

When making chat completion requests, use model="yi-large".

from openai import OpenAI

API_BASE = "https://api.01.ai/v1"
API_KEY = "Your API Key"  # Replace with your actual API key
client = OpenAI(
    api_key=API_KEY,
    base_url=API_BASE
)

# Example usage in a completion call
completion = client.chat.completions.create(
    model="yi-large",
    messages=[...] 
)