DOCSEARCHbyAlgolia
Submit Docs

Kwai Keye-VL Documentation

repository·main·Indexed 21 days ago

https://github.com/kwai-keye/keye

A family of high-performance multimodal large language models for video understanding, visual reasoning, and agentic tasks. Includes documentation for deploying Keye-VL-671B-A37B via SGLang, using the multi-modal chat completions API, and performing evaluations using the KC-MMBench and CC-OCR benchmarks via VLMEvalKit.

Tokens
15K
Snippets
38
Records
53
Agent score
74%

What's inside Kwai Keye-VL

  1. Overview of KC-MMBench Evaluation Datasets

    main

    KC-MMBench is an evaluation suite built on the VLMEvalKit framework. It consists of 6 specialized datasets designed to evaluate Vision-Language Models (VLMs) on short video data and e-commerce tasks. The datasets are available on Hugging Face.

    Available Tasks

    TaskDescription
    CPVPredicting product attributes in e-commerce.
    Hot_Videos_AggregationDetermining if multiple videos belong to the same topic.
    Collection_OrderDetermining the logical order between multiple videos with the same topic.
    Pornographic_CommentDetecting pornographic content in short video comments.
    High_LikeBinary classification of short video like rates.
    SPUDetermining if two items are the same product in e-commerce.

  2. Overview of Kwai Keye-VL

    main

    Kwai Keye-VL is a series of cutting-edge multimodal large language models (MLLMs) developed by the Kwai Keye Team at Kuaishou. The models excel in video understanding, visual perception, and reasoning tasks.

    Key features include:

    • Long-Video Understanding: Utilizing DSA-Native (DeepSeek Sparse Attention) architecture for efficient, nearly lossless reasoning over ultra-long contexts (up to 256K tokens).
    • Agent Capabilities: Built-in support for Search, Tool, and Code agent workflows.
    • High Performance: Competitive performance against top-tier closed-source models in temporal grounding, fine-grained perception, and complex reasoning.
    • Scalable Models: The series includes various scales, such as the 30B-class flagship Keye-VL-2.0-30B-A3B and the massive Keye-VL-671B-A37B.

  3. How Prompt Construction Works

    main

    VLMEvalKit uses a hierarchical approach to construct input prompts:

    1. dataset.build_prompt(): Each dataset class has a method to format questions (e.g., combining hints, questions, and options for MCQ).
    2. model.build_prompt(): If defined on the model, this overrides the dataset's prompt construction. This is useful for models with specific formatting requirements.

    Customizing Prompt Logic

    You can control which benchmarks use a model's custom build_prompt() by implementing the use_custom_prompt(self, dataset: str) -> bool function within your model class. If this returns True, the model's custom prompt logic is used; otherwise, the dataset's default is used.

    def use_custom_prompt(self, dataset: str) -> bool:
    from vlmeval.dataset import DATASET_TYPE, DATASET_MODALITY
    dataset_type = DATASET_TYPE(dataset, default=None)
    if not self._use_custom_prompt:
        return False
    if listinstr(['MMVet'], dataset):
        return True
    if dataset_type == 'MCQ':
        return True
    if DATASET_MODALITY(dataset) == 'VIDEO':
        return False
    return False

  4. How Model Splitting and GPU Allocation Works

    main

    VLMEvalKit supports automatic GPU resource allocation and model splitting for lmdeploy or transformers backends.

    • python command: The model is allocated to all available GPUs. Use CUDA_VISIBLE_DEVICES to restrict which GPUs are visible to the process.
    • torchrun command: Each model instance is allocated N_GPU // N_PROC GPUs, where N_PROC is the --nproc-per-node value.
      • If CUDA_VISIBLE_DEVICES is set, N_GPU is the count of GPUs in that variable.
      • If not set, N_GPU is the total number of available GPUs.

    Note: This feature is not compatible with the vllm backend. For vllm, use the python command, and all visible GPUs will be used by the single instance.

    # Launch two model instances in data parallel, each using 4 GPUs (on an 8-GPU machine)
torchrun --nproc-per-node=2 run.py --data MMBench_DEV_EN --model InternVL3-78B

# Launch one model instance using all 8 GPUs
python run.py --data MMBench_DEV_EN --model InternVL3-78B

# Launch three model instances, each using 2 GPUs (using specific GPUs 1-6)
CUDA_VISIBLE_DEVICES=1,2,3,4,5,6 torchrun --nproc-per-node=3 run.py --data MMBench_DEV_EN --model InternVL3-38B

  5. Getting started with VLMEvalKit

    main

    To begin using VLMEvalKit, follow the recommended workflow based on your goals:

    1. For basic usage: Follow the Quickstart guide to set up your environment and run a mini-experiment. This is the best way to familiarize yourself with the core process.
    2. For customization: If you need to add new datasets or models, refer to the Advanced Tutorial, which covers development and the configuration system.

  6. Run CC-OCR inference and evaluation in batch

    main

    To perform inference and evaluation for the CC-OCR benchmark, execute the run.py script from the root directory of VLMEvalKit followed by the CC-OCR evaluator script. The process is divided into four main subsets: multi_scene_ocr, multi_lan_ocr, doc_parsing, and kie.

    For each subset, you must specify the --data flag with the relevant dataset names, the --model flag with your model name, and the --work-dir flag to define the output directory. After running the inference, run python vlmeval/dataset/utils/ccocr_evaluator/common.py ${SUB_OUTPUT_DIR} to process the results.

    # Example for the KIE subset
MODEL_NAME="QwenVLMax"
OUTPUT_DIR="/your/path/to/output_dir"
SUB_OUTPUT_DIR=${OUTPUT_DIR}/kie

# 1. Run inference
python run.py --data CCOCR_Kie_Sroie2019Word CCOCR_Kie_Cord CCOCR_Kie_EphoieScut CCOCR_Kie_Poie CCOCR_Kie_ColdSibr CCOCR_Kie_ColdCell --model ${MODEL_NAME} --work-dir ${SUB_OUTPUT_DIR} --verbose

# 2. Run evaluation
python vlmeval/dataset/utils/ccocr_evaluator/common.py ${SUB_OUTPUT_DIR}

  7. Configure API Keys for Proprietary VLMs and Judges

    main

    To use proprietary VLMs (like GPT-4v, Gemini) or to use an LLM as a judge/choice extractor, you must provide API keys. You can set these as environment variables or create a .env file located under the $VLMEvalKit directory.

    If no API key is provided for a judge, VLMEvalKit defaults to exact matching mode (searching for 'Yes', 'No', 'A', 'B', etc., in the output). Note that exact matching is only applicable to Yes-or-No and Multi-choice tasks.

    # Example .env file content under $VLMEvalKit
DASHSCOPE_API_KEY=
GOOGLE_API_KEY=
OPENAI_API_KEY=
OPENAI_API_BASE=
STEPAI_API_KEY=
REKA_API_KEY=
GLMV_API_KEY=
CW_API_BASE=
CW_API_KEY=
SENSENOVA_API_KEY=
HUNYUAN_SECRET_KEY=
HUNYUAN_SECRET_ID=
LMDEPLOY_API_BASE=
EVAL_PROXY=

  8. Use the Config System for flexible evaluations

    main

    While VLMEvalKit allows simple evaluations using --model and --data arguments in run.py, the Config System provides a more flexible way to evaluate multiple models and datasets with different settings simultaneously.

    To use the config system, create a JSON file defining your model and data configurations and pass it to the run.py script using the --config flag.

    python run.py --config config.json

  10. Start an LMDeploy Inference Service

    main

    Launch an LMDeploy API server to host a VLM.

    Important: You must specify the --model-name flag. This allows VLMEvalKit to select the correct prompt construction strategy (e.g., for handling specific datasets like HallusionBench for InternVL2) when communicating with the API.

    If you specify a custom --server-port, you must set the LMDEPLOY_API_BASE environment variable to point to that server address during evaluation.

    lmdeploy serve api_server OpenGVLab/InternVL2-8B --model-name InternVL2-8B

  11. Setup pre-commit hooks for VLMEvalKit

    main

    Before submitting a Pull Request to VLMEvalKit, run the pre-commit checks to ensure code quality.

    # Install pre-commit
pip install pre-commit

# Install the hook in the repository directory
pre-commit install

# Run checks on all files
pre-commit run --all-files
    pip install pre-commit
pre-commit install
pre-commit run --all-files