GLM-4.5 Foundation Models

You can benchmark the serving performance of GLM-4.6 models using the sglang.bench_serving module. This is useful for measuring throughput, latency (E2E and TTFT), and concurrency limits on your hardware (e.g., Ascend NPU).

Note: The provided examples use sglang version 0.4.10 and the sglang-oai backend. Ensure your model server is running at the specified --base-url before executing the benchmark.

# Example: Single concurrency short text benchmark
python3 -m sglang.bench_serving   
 --backend sglang-oai   
 --base-url http://127.0.0.1:30000   
 --model zai-org/GLM-4.6    
 --dataset-name sharegpt   
 --num-prompts 8 \   
 --random-input-len 8192 \
 --random-output-len 1024  \
 --request-rate 1 \
 --max-concurrency 1 \
 --disable-stream   

For non-NVIDIA hardware, use the following specialized deployment guides:

• Ascend NPU: Use xLLM for fast inference. Refer to the Ascend NPU Deployment Guide.
• AMD GPU: Refer to the AMD GPU Deployment Guide.

Install sglang on your server and launch the GLM-4.5 model service using the sglang.launch_server module. This configuration uses the glm45 parser for tool calls and reasoning, and enables the EAGLE speculative decoding algorithm.

# Install sglang
pip install sglang

# Launch the server
python3 -m sglang.launch_server \
  --model-path zai-org/GLM-4.5 \
  --tp-size 16 \
  --tool-call-parser glm45  \
  --reasoning-parser glm45 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --mem-fraction-static 0.7 \
  --served-model-name glm-4.5 \
  --port 8000 \
  --host 0.0.0.0

To serve the zai-org/GLM-4.7-FP8 model using SGLang, use the sglang.launch_server module. For GLM-4.7, set --tool-call-parser to glm47. This example includes configuration for the EAGLE speculative decoding algorithm.

python3 -m sglang.launch_server \
  --model-path zai-org/GLM-4.7-FP8 \
  --tp-size 8 \
  --tool-call-parser glm47  \
  --reasoning-parser glm45 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --mem-fraction-static 0.8 \
  --served-model-name glm-4.7-fp8 \
  --host 0.0.0.0 \
  --port 8000

To connect your local Claude Code environment to the remote GLM-4.5 server, follow these configuration steps:

1. Edit config.example.json: Change api_base_url to your server's accessible address (e.g., http://<your-server-ip>:8000).
2. Rename the file to config.json.
3. Move the file to ~/.claude-code-router/config.json.
4. Restart the router service using the ccr restart command.

Install the quantization tools by running bash install.sh within the msit/msmodelslim/ directory. To quantize GLM-4.5, copy the quant_qwen_moe_w8a8.py script to quant_glm_moe_w8a8.py and modify the disable_names logic in the script to target specific layers (e.g., model.layers.{ids}.mlp.down_proj and model.layers.{ids}.self_attn.o_proj for the first three layers).

cd msit/msmodelslim/
bash install.sh 

# Modify quantization script based on Qwen-MOE
cd msmodelslim/example/Qwen3-MOE
cp quant_qwen_moe_w8a8.py quant_glm_moe_w8a8.py 
# Edit quant_glm_moe_w8a8.py to update disable_names logic

Fine-tuning can be performed using the Llama Factory or Swift frameworks.

Using Llama Factory

Using Swift

For non-NVIDIA hardware:

• Ascend A3 Devices: Use xLLM for high-speed inference. Refer to the Ascend NPU Deployment Guide.
• AMD GPUs: Refer to the AMD GPU Deployment Guide.

Download the appropriate Docker image based on your Ascend hardware architecture (A2 x86, A2 arm, or A3 arm) to set up the environment for GLM-4.5/4.6/4.7 inference.

# A2 x86
docker pull quay.io/jd_xllm/xllm-ai:xllm-dev-hb-rc2-x86
# A2 arm
docker pull quay.io/jd_xllm/xllm-ai:xllm-dev-hb-rc2-arm
# A3 arm
docker pull quay.io/jd_xllm/xllm-ai:xllm-dev-hc-rc2-arm