DOCSEARCHbyAlgolia
Submit Docs

GLM-V Documentation

repository·main·Indexed 25 days ago

https://github.com/zai-org/glm-v

Open-source repository for the GLM-4.xV series of Vision-Language Models (VLMs), featuring capabilities for multimodal reasoning, grounding, and GUI agents. Includes documentation for serving models via vLLM on AMD MI300X GPUs, integrating with Midscene.js for automation, and using the glmv_reward system (v0.1.0) for reward verification across math, science, multimodal, and agent-based tasks.

Tokens
40.1K
Snippets
69
Records
168
Agent score
81%

What's inside GLM-V

  1. Overview of GLM-4.1V-9B models

    main

    The GLM-4.1V-9B series is built on the GLM-4-9B-0414 foundation and utilizes Reinforcement Learning with Curriculum Sampling (RLCS) to enhance reasoning.

    Available versions:

    • GLM-4.1V-9B-Thinking: A reasoning-focused model that integrates a Chain-of-Thought mechanism for improved accuracy and interpretability.
    • GLM-4.1V-9B-Base: An open-source base model for research.

    Key technical specifications for the Thinking model:

    • 64k context length support.
    • Support for any aspect ratio and up to 4k image resolution.
    • Bilingual (Chinese/English) support.

  2. Overview of GLM-4.5V Models

    main

    GLM-4.5V is a versatile multimodal model based on GLM-4.5-Air. It excels in various visual reasoning tasks including:

    • Image Reasoning: Scene understanding, complex multi-image analysis, and position recognition.
    • Video Understanding: Long video shot analysis and event recognition.
    • GUI Tasks: Screen reading, icon recognition, and desktop operation assistance.
    • Document Parsing: Complex chart and long document analysis (e.g., research reports).
    • Grounding: Precise localization of visual elements.

    Thinking Mode: Users can toggle a "thinking mode" switch to choose between fast response or deep reasoning, similar to the usage pattern of the GLM-4.5 language model.

  3. Overview of GLM-4.6V models

    main

    The GLM-4.6V series provides multimodal capabilities with a focus on bridging visual perception and executable action through native function calling. It includes two primary versions:

    • GLM-4.6V (106B): A foundation model designed for cloud environments and high-performance clusters.
    • GLM-4.6V-Flash (9B): A lightweight model optimized for local deployment and low-latency applications.

    Key features include:

    • Native Multimodal Function Calling: Allows passing images, screenshots, or document pages directly as tool inputs.
    • Interleaved Image-Text Generation: Synthesizes mixed media content from multimodal contexts.
    • Multimodal Document Understanding: Supports up to 128K tokens for processing complex, image-heavy documents (charts, tables, layouts).
    • Frontend Replication: Reconstructs HTML/CSS from UI screenshots and supports natural-language-driven visual edits.

  4. Overview of GLM-V Model Series

    main

    GLM-V is a series of Vision-Language Models (VLM) designed for complex multimodal tasks, including reasoning, long-context understanding, and acting as multimodal agents. The repository provides access to the GLM-4.6V, GLM-4.5V, and GLM-4.1V series.

    Key capabilities include:

    • Visual Perception: Understanding images and videos.
    • Reasoning: Solving complex problems based on visual input.
    • Grounding: Identifying and bounding specific objects in images.
    • Function Calling: Native support in GLM-4.6V to bridge perception and executable actions.
    • UI Automation: Specialized models like UI2Code^N for UI-to-code tasks.

  5. Understand the glmv-web-replication skill

    main

    The glmv-web-replication skill is designed for frontend visual replication. It explores a target website's publicly visible pages using Playwright MCP or an agent-browser to capture screenshots, layout information, and client-side interaction patterns.

    Key Capabilities:

    • Replicates page layout and visual styling.
    • Captures navigation structure and CSS/design tokens.
    • Records client-side interaction patterns and static assets (images, fonts).

    Critical Limitations:

    • Frontend Only: It does NOT replicate backend logic, server-side behavior, databases, authentication systems, or API business logic.
    • Public Content Only: It must never scrape behind a login wall or collect credentials, secrets, or PII.
    • Legal Requirement: You must have ownership or explicit written permission to replicate a site. The agent is required to perform an authorization gate before proceeding.

  6. VLM Chat Helper project structure

    main

    The project follows an Electron-based architecture with separate processes for main logic, preload scripts, and the UI renderer:

    • src/main/: Main process code (window management, hotkeys, recording, IPC handlers, database services).
    • src/preload/: Preload scripts for secure communication between processes.
    • src/renderer/: The Vue 3 + TypeScript frontend (components, views, stores, composables).
    • build/: Build assets.
    • resources/: Application resources.
    • electron.vite.config.ts: Electron Vite configuration.
    • electron-builder.yml: Packaging configuration.

  8. Output directory convention for PDF-to-Web

    main

    All generated assets are stored in a specific directory structure under {WORKSPACE}/web/ to prevent overwriting previous conversions:

    web/
└── <pdf_stem>_<timestamp>/
    ├── outline.json        ← structured web plan (WebPlan schema)
    ├── crops/
    │   ├── fig_arch_crop.png
    │   ├── table_results_crop.png
    │   └── ...
    └── index.html          ← the website
    • <pdf_stem>: The filename of the PDF without the extension.
    • <timestamp>: Formatted as YYYYMMDD_HHMMSS.
    • Note: The index.html file references images using relative paths: crops/<name>_crop.png.

  9. The Website Replication Blueprint structure

    main

    The replication process first builds a "website replication blueprint." This is a structured directory that maps the site's information architecture using a nested folder hierarchy. Each folder represents a page, and child folders represent pages reachable from that page.

    Inside each page folder, the following files are typically generated:

    • _page.md: Page blueprint containing sections, components, and interaction summaries.
    • _full.png: A full-page screenshot.
    • _scroll_00.png ~ N.png: A sequence of screenshots captured during scrolling.
    • _interactions.md: A record of all captured interactions.
    • _interactions/: Screenshots of different interaction states.
    • _assets/: Assets specific to that page (images, videos, etc.).

    Global files include:

    • _meta.md: Site-wide metadata.
    • _sitemap.md: The site's sitemap.
    • _assets/: Global assets like fonts and favicons.
    • _navigation_graph.md: A site-wide navigation graph (rendered in Mermaid).
    blueprint/
├── _meta.md                      # Site-wide metadata
├── _sitemap.md                   # Sitemap
├── _assets/                      # Global assets (fonts, favicon, etc.)
├── home/
│   ├── _page.md                  # Page blueprint (sections, components, interaction summary)
│   ├── _full.png                 # Full-page screenshot
│   ├── _scroll_00.png ~ N.png    # Scrolling screenshot sequence
│   ├── _interactions.md          # Record of all interactions
│   ├── _interactions/
│   ├── _assets/
│   ├── products/
│   │   ├── _page.md
│   │   └── ...
│   └── ...
└── _navigation_graph.md          # Site-wide navigation graph (Mermaid)

  10. Understand the glmv-pdf-to-ppt output directory structure

    main

    The skill generates all outputs within a specific directory under {WORKSPACE}/ppt/<pdf_stem>_<timestamp>/. This ensures organized, timestamped versions of your presentations.

    Directory Layout:

    ppt/
└── <pdf_stem>_<timestamp>/
    ├── outline.json        ← structured slide plan (SlidesPlan schema)
    ├── crops/              ← locally-saved cropped images
    │   ├── slide3_method_crop.png
    │   └── slide5_results_crop.png
    ├── slide_01.html
    ├── slide_02.html
    ├── ...
    └── summary.md          ← final summary document
    • <pdf_stem>: The PDF filename without the extension.
    • <timestamp>: Formatted as YYYYMMDD_HHMMSS (e.g., 20240119_143022).
    • crops/: Contains all locally-saved cropped images used in the slides.
    • slide_XX.html: Individual HTML files for each slide. Note that these files use relative paths (e.g., crops/<name>.png) to reference images.

  11. Understand glmv-prompt-gen output modes

    main

    The skill supports three distinct output modes to tailor prompts for different AI generation tools:

    • image (default): Generates prompts specifically optimized for text-to-image tools (e.g., Midjourney, Stable Diffusion, DALL-E).
    • video: Generates prompts specifically optimized for text-to-video tools (e.g., Sora, Runway, Kling, Pika).
    • auto: Generates both text-to-image and text-to-video prompts from the provided visual input.

  12. Key Capabilities of GLM-4.6V

    main

    GLM-4.6V is designed for advanced multimodal tasks:

    • Native Multimodal Function Calling: Supports vision-based tool calls where visual data (images, screenshots, docs) serves as input without text conversion.
    • Interleaved Image-Text Generation: Generates high-quality content that mixes text and images, capable of calling search/retrieval tools to supplement visual context.
    • Multimodal Document Understanding: Supports up to 128K tokens for long documents, parsing layouts, charts, and tables directly from images.
    • Frontend Replication & Visual Editing: Can reconstruct pixel-perfect HTML/CSS from UI screenshots and perform visual edits driven by natural language instructions.