Kimi-VL Documentation

Use the vllm serve command to deploy Kimi-VL as an API server.

Key CLI Flags:

• --trust-remote-code: Required to load the model.
• --served-model-name: The name used to identify the model in API calls.
• --max-model-len: Set this (e.g., 32768 or 131072) to adjust the context window.
• --max-num-batched-tokens: Adjusts the batch size for tokens.
• --limit-mm-per-prompt: Controls the number of images allowed per prompt (e.g., image=64).
• --tensor-parallel-size: Number of GPUs for tensor parallelism.

# kimi-vl-thinking-2506
vllm serve moonshotai/Kimi-VL-A3B-Thinking-2506 --served-model-name kimi-vl-thinking-2506 --trust-remote-code --tensor-parallel-size 1 --max-num-batched-tokens 32768 --max-model-len 32768 --limit-mm-per-prompt image=64

# kimi-vl-instruct
vllm serve moonshotai/Kimi-VL-A3B-Instruct --served-model-name kimi-vl --trust-remote-code --tensor-parallel-size 1 --max-num-batched-tokens 32768 --max-model-len 32768 --limit-mm-per-prompt image=64

Use the Hugging Face transformers library to run inference with the Kimi-VL-A3B-Instruct model.

Recommended Environment:

• Python: 3.10
• PyTorch: 2.5.1
• Transformers: 4.51.3

Optimization Tip: If flash-attn is installed, use torch_dtype=torch.bfloat16 and attn_implementation="flash_attention_2" in from_pretrained to save memory and increase speed.

import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoProcessor

model_path = "moonshotai/Kimi-VL-A3B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)

processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)

image_path = "./figures/demo.png"
image = Image.open(image_path)
messages = [
    {"role": "user", "content": [{"type": "image", "image": image_path}, {"type": "text", "text": "What is the dome building in the picture? Think step by step."}]}
]
text = processor.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
inputs = processor(images=image, text=text, return_tensors="pt", padding=True, truncation=True).to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=512)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
response = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)[0]
print(response)

Once the vLLM server is running, you can interact with it using the openai Python client. Images must be provided as base64 encoded strings within the image_url field of the message content. The message structure follows the OpenAI vision format: {"role": "user", "content": [{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}, {"type": "text", "text": "..."}]}.

import base64
from PIL import Image
from io import BytesIO
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="token-abc123",
)

image_path = "./figures/demo.png"
image = Image.open(image_path).convert("RGB")

buffered = BytesIO()
image.save(buffered, format="JPEG")
img_b64_str = base64.b64encode(buffered.getvalue()).decode("utf-8")
base64_image_url = f"data:image/jpeg;base64,{img_b64_str}"

messages = [
    {"role": "user", "content": [{"type": "image_url", "image_url": {"url": base64_image_url}}, {"type": "text", "text": "What is the dome building in the picture? Think step by step."}]}
]

completion = client.chat.completions.create(
  model="kimi-vl-thinking-2506", # or kimi-vl
  messages=messages
)

print(completion.choices[0].message)

The Kimi-VL-A3B-Thinking-2506 variant is optimized for multimodal reasoning and supports multi-image inputs.

Key Configuration:

• Temperature: It is recommended to use Temperature = 0.8 for Thinking models.
• Max Tokens: For complex reasoning, you may need a higher max_new_tokens value (e.g., 32768).

Optimization Tip: If flash-attn is installed, use torch_dtype=torch.bfloat16 and attn_implementation="flash_attention_2" in from_pretrained to save memory and increase speed.

import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoProcessor

model_path = "moonshotai/Kimi-VL-A3B-Thinking-2506"
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)
# If flash-attn has been installed, it is recommended to set torch_dtype=torch.bfloat16 and attn_implementation="flash_attention_2"
# model = AutoModelForCausalLM.from_pretrained(
#     model_path,
#     torch_dtype=torch.bfloat16,
#     device_map="auto",
#     trust_remote_code=True,
#     attn_implementation="flash_attention_2"
# )
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)

image_paths = ["./figures/demo1.png", "./figures/demo2.png"]
images = [Image.open(path) for path in image_paths]
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image_path} for image_path in image_paths
        ] + [{"type": "text", "text": "Please infer step by step who this manuscript belongs to and what it records"}],
    },
]
text = processor.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
inputs = processor(images=images, text=text, return_tensors="pt", padding=True, truncation=True).to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=32768, temperature=0.8)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
response = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)[0]
print(response)

You can use the vLLM main branch to run Kimi-VL models for offline inference. This requires PIL, transformers, and vllm. You must set trust_remote_code=True when initializing both the LLM and the AutoProcessor. The input format for llm.generate requires a dictionary containing the prompt (processed via processor.apply_chat_template) and multi_modal_data containing the image.

from PIL import Image
from transformers import AutoProcessor
from vllm import LLM, SamplingParams

model_path = "moonshotai/Kimi-VL-A3B-Instruct"  # or "moonshotai/Kimi-VL-A3B-Thinking-2506"
llm = LLM(
    model_path,
    trust_remote_code=True,
)

processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)

image_path = "./figures/demo.png"
image = Image.open(image_path)
messages = [
    {"role": "user", "content": [{"type": "image", "image": image_path}, {"type": "text", "text": "What is the dome building in the picture? Think step by step."}]}
]
text = processor.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
outputs = llm.generate([{"prompt": text, "multi_modal_data": {"image": image}}], sampling_params = SamplingParams(max_tokens=512))

print("-" * 50)
for o in outputs:
    generated_text = o.outputs[0].text
    print(generated_text)
    print("-" * 50)