ChatGLM-6B

The ChatGLM family has evolved through several key releases:

• CodeGeeX2: A code generation model based on ChatGLM2-6B. It features significantly improved coding capabilities (e.g., +321% in Rust on HumanEval-X), supports up to 8192 sequence length, and can run on 6GB VRAM after quantization.
• ChatGLM2-6B: An upgrade to the original ChatGLM-6B. It features:
  • Higher Performance: Significant improvements in MMLU, CEval, GSM8K, and BBH benchmarks.
  • Longer Context: Uses FlashAttention to extend context length from 2K to 32K (with 8K used during dialogue training).
  • Efficient Inference: Uses Multi-Query Attention to increase inference speed by 42% and improve VRAM efficiency (8K dialogue length supported with 6GB VRAM via INT4).
• VisualGLM-6B: A multimodal model for image understanding. Requires SwissArmyTransformer and torchvision to run via cli_demo_vision.py or web_demo_vision.py.
• WebGLM: A research work supporting long answers with accurate citations using web information.

ChatGLM-6B is a small-scale model (6B parameters) and has several known limitations that developers should account for in their applications:

• Limited Model Capacity: Due to its size, it has weaker memory and language capabilities compared to larger models. It may generate incorrect factual information and struggles with complex logic tasks such as mathematics and programming.
• Potential for Harmful or Biased Content: As a model only partially aligned with human intent, it may generate offensive, biased, or harmful content.
• Weak English Proficiency: The training data is predominantly Chinese. English instructions may result in lower quality responses, contradictions with Chinese context, or mixed Chinese-English output.
• Susceptibility to Misleading Prompts: The model has weaker conversational stability and 'self-awareness.' It can be easily misled into providing incorrect information regarding its own identity or facts.

To deploy a model using a P-Tuning v2 checkpoint (which only contains PrefixEncoder parameters), follow these steps:

1. Load the Tokenizer:

from transformers import AutoConfig, AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)

2. Load the Base Model and Inject Prefix Weights: Note: Replace pre_seq_len with your actual training value and THUDM/chatglm-6b with your local model path if necessary.

import torch
import os
from transformers import AutoConfig, AutoModel

config = AutoConfig.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True, pre_seq_len=128)
model = AutoModel.from_pretrained("THUDM/chatglm-6b", config=config, trust_remote_code=True)

# Load PrefixEncoder weights
prefix_state_dict = torch.load(os.path.join(CHECKPOINT_PATH, "pytorch_model.bin"))
new_prefix_state_dict = {}
for k, v in prefix_state_dict.items():
    if k.startswith("transformer.prefix_encoder."):
        new_prefix_state_dict[k[len("transformer.prefix_encoder."):]] = v
model.transformer.prefix_encoder.load_state_dict(new_prefix_state_dict)

3. Quantization and Inference:

# Comment out the following line if you don't use quantization
model = model.quantize(4)
model = model.half().cuda()
model.transformer.prefix_encoder.float()
model = model.eval()

response, history = model.chat(tokenizer, "你好", history=[])

If you are using a full parameter checkpoint or a full finetune, load it directly:

model = AutoModel.from_pretrained(CHECKPOINT_PATH, trust_remote_code=True)

from transformers import AutoConfig, AutoModel, AutoTokenizer

# 1. Load Tokenizer
tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)

# 2. Load P-Tuning v2 Checkpoint
config = AutoConfig.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True, pre_seq_len=128)
model = AutoModel.from_pretrained("THUDM/chatglm-6b", config=config, trust_remote_code=True)
prefix_state_dict = torch.load(os.path.join(CHECKPOINT_PATH, "pytorch_model.bin"))
new_prefix_state_dict = {}
for k, v in prefix_state_dict.items():
    if k.startswith("transformer.prefix_encoder."):
        new_prefix_state_dict[k[len("transformer.prefix_encoder."):]] = v
model.transformer.prefix_encoder.load_state_dict(new_prefix_state_dict)

# 3. Quantize and Chat
model = model.quantize(4)
model = model.half().cuda()
model.transformer.prefix_encoder.float()
model = model.eval()

response, history = model.chat(tokenizer, "你好", history=[])