DOCSEARCHbyAlgolia
Submit Docs

comfyui-mcp

repository·main·Indexed 19 days ago

https://github.com/artokun/comfyui-mcp

A local-first, agent-native control plane for ComfyUI featuring an MCP server and autonomous sidebar agent. It enables LLMs—including Claude, ChatGPT, Gemini, and local models via Ollama—to author, edit, and manage workflows, models, and custom nodes using natural language. The package includes a Claude Code plugin and supports deployments via Docker Compose, RunPod, local, LAN, VPS, or Comfy Cloud. Version 0.49.8.

Tokens
252.1K
Snippets
497
Records
1.1K
Agent score
62%

What's inside comfyui-mcp

  1. Overview of ComfyUI MCP

    main

    ComfyUI MCP is a local-first, agent-native control plane for ComfyUI. It functions as both an MCP (Model Context Protocol) server and a Claude Code plugin. It allows AI assistants to interact with ComfyUI instances—whether they are running locally, on a remote server, or via Comfy Cloud—using natural language.

    Key capabilities include:

    • Image/Video/Audio Generation: Using tools like generate_image with support for ControlNet and IP-Adapter.
    • Workflow Management: Authoring, modifying, validating, and visualizing workflows using a compact DSL.
    • Model Management: Searching (HuggingFace/CivitAI), downloading, listing, and removing models.
    • Custom Node Management: Installing, updating, and managing node packs via ComfyUI-Manager.
    • Server Maintenance: Installing/updating ComfyUI, inspecting environments, and controlling server state (start/stop).

    It is compatible with macOS, Linux, and Windows.

  2. Overview of the ComfyUI-MCP Mobile App

    main

    The ComfyUI-MCP mobile app (available in closed beta for iOS via TestFlight and Android via Firebase App Distribution) is a Flutter-based companion that provides a mobile interface for the ComfyUI agent. It connects to the same orchestrator bridge used by the desktop Sidebar Panel, allowing you to:

    • Chat with the agent: Continue conversations, request renders, or ask for workflow fixes.
    • Monitor and manage renders: View a live queue monitor bar (sampler progress, queue depth) and use one-tap cancel to interrupt a specific prompt_id without affecting pending jobs.
    • Browse CivitAI: Full parity with desktop, including infinite scroll, creator search (search_civitai_creators), and access to your full likes collection.
    • Select Providers: Choose from Claude, ChatGPT, Gemini, Kimi K3, or local providers like Ollama, LM Studio, and llama.cpp.
    • Manage Assets: Browse installed LoRAs with detail sheets and use prompt presets.

  3. Overview of ComfyUI MCP tool categories

    main

    ComfyUI MCP tools are organized into several functional categories:

    • Image Generation: Tools for creating images.
    • Workflow Execution: Running existing workflows.
    • Workflow Authoring: Creating and editing workflows.
    • Workflow Library: Managing saved workflows.
    • Assets & Images: Handling image files and assets.
    • Models: Managing model files.
    • Custom Nodes: Installing and managing custom nodes.
    • API Nodes: Interacting with API-based nodes.
    • Install & Environment: Managing the ComfyUI installation and environment.
    • Process Control: Controlling the ComfyUI process (e.g., restarting, canceling).
    • Defaults, Stats & Skills: Accessing system statistics and specialized agent skills.

  4. Features of the ComfyUI-MCP Mobile app

    main

    The mobile app acts as a native front end to the ComfyUI agent, allowing you to perform the following tasks from your phone:

    • Agent chat: Engage in the same conversations used in the desktop Sidebar Panel.
    • Model picker: Access the live model catalog from the orchestrator and switch models mid-session.
    • Civitai browsing: Search for models and creators, and manage likes and favorites.
    • Remote control: Mirror the desktop panel tab and monitor workflow diagnostics remotely.

  5. Keep the panel node-pack in sync with the orchestrator

    main

    The orchestrator (comfyui-mcp) and the sidebar panel (comfyui-agent-panel) are separate entities. Updating the orchestrator does not automatically update the panel. To prevent version drift—which causes bridge command failures or missing UI features—use the panel-node-pack-sync skill.

    When to use this skill:

    • After updating the orchestrator (self_update, npm i -g comfyui-mcp, or environment changes).
    • When encountering errors like "panel is too old" or unimplemented graph_*/ui_* commands.
    • When a documented sidebar feature is missing.
    • When the user explicitly requests to update, pin, or unpin the panel.

  6. Explore shipped features in comfyui-mcp

    main

    The following core capabilities are currently available in the project:

    • Agent in ComfyUI: An agent in the Panel that reads graphs, edits nodes, runs workflows, and explains failures.
    • Multi-Backend LLM Support: Use Claude, ChatGPT/Codex, Gemini, Antigravity, Grok, Kimi, GLM, Ollama, LM Studio, llama.cpp, OpenRouter, or custom endpoints.
    • Local LLM Operation: Fully local workflows via Ollama, LM Studio, or llama.cpp.
    • Workflow Authoring: Build, patch, slice, validate, and lock workflows using natural language or image metadata.
    • Model & Node Management: Search, install, and repair node packs; identify missing models and determine appropriate quantizations based on VRAM.
    • Civitai Browser: Browse images, videos, and models directly in-panel and pull workflows onto the canvas.
    • Mobile Integration: Chat with the agent, monitor the queue, browse LoRAs, and mirror desktop tabs.
    • Micro-apps: Convert workflows into one-click apps with forms. Run them locally, on a pod, or via the mobile app using apps_* tools.
    • Remote & Cloud Connectivity: Connect the panel to remote ComfyUI instances, self-hosted relays, or Comfy Cloud.
    • RunPod Integration: Create, connect to, and stop RunPod pods directly from the panel.
    • Local LoRA Training: Train character LoRAs on FLUX.1-dev using train_* tools in a GPU Docker container.

  7. Overview of Z-Image model variants

    main

    Z-Image is a 6B-parameter image generation model using a Scalable Single-Stream DiT (S3-DiT) architecture and a Qwen text encoder. There are two primary variants:

    1. Z-Image Base (and RedCraft finetune): Full model supporting negative prompts, LoRA training, and ControlNet. Requires 10-30 steps.
    2. Z-Image Turbo: A DMD-distilled version optimized for speed (8-10 steps). It has no effective negative prompts because CFG is baked in.

    Important VAE Note: While Z-Image uses the Flux VAE architecture, it uses different weights. You must use z-image-ae.safetensors and not the standard Flux ae.safetensors to avoid errors.

  8. What is ERNIE-Image and when to use it

    main

    ERNIE-Image is an open-weight, single-stream Diffusion Transformer (approx. 8B parameters) released under the Apache-2.0 license, making it suitable for commercial use. It is specifically optimized for multilingual text rendering (English, Chinese, Japanese) and structured layouts.

    Key Capabilities

    • Text-to-Image: It generates images from scratch based on text descriptions. It is not an image editor (it does not follow instructions like "change the shirt color").
    • Multilingual Typography: Excellent at rendering clean glyphs for CJK and English text.
    • Layout Control: Strong at following descriptive prose for complex, multi-object scenes.

    When to choose ERNIE-Image

    • When you need commercial-safe (Apache-2.0) models.
    • When your project requires clean multilingual text (posters, signage, manga bubbles).
    • When you need high-fidelity text and layout without explicit spatial bounding boxes (unlike Ideogram 4).

  9. Overview of WAN MultiTalk (Audio-Driven Talking Avatar)

    main

    WAN MultiTalk is a workflow designed to drive a still portrait's lip-sync and head motion using an audio track. It utilizes WAN 2.1 14B Image-to-Video (I2V) via the kijai/WanVideoWrapper.

    Key Characteristics:

    • Mechanism: Uses Wav2Vec speech embeddings to condition the WAN sampler, ensuring mouth and facial expressions follow the speech.
    • Efficiency: Employs the lightx2v step-distill LoRA to allow for high-quality generation in very few sampling steps.
    • Use Cases: Talking heads, dubbing, and single-speaker avatar clips (typically ~10s at 480p).
    • Distinction: This is an audio $\rightarrow$ lip-sync process, which is different from wan-animate (which is pose/motion-driven character animation).

  10. Overview of WAN 2.2 Text-to-Video (T2V) Workflows

    main

    WAN 2.2 T2V uses a 14B parameter Mixture of Experts (MoE) architecture that splits generation across two specialized models to achieve high-quality video from text prompts:

    • HighNoise model: Handles early denoising to establish structure, motion, and composition.
    • LowNoise model: Handles late denoising to refine details and sharpen the output.

    Key Distinction: Unlike I2V (Image-to-Video) or FLF workflows, T2V does not use image conditioning nodes like CLIPVisionEncode or WanFirstLastFrameToVideo. Instead, it uses EmptyHunyuanLatentVideo for latent initialization and relies solely on text conditioning.

  11. Overview of the Agent-platform capability suite

    main

    The Agent-platform capability suite is a collection of advanced tools and services designed to extend ComfyUI's capabilities through LLM-driven automation. Note that this suite is currently PARKED and its implementation resides on the feat/agent-platform-tools branch.

    Key functional areas include:

    • LoRA Management: Cataloging and managing LoRAs, including importing sidecar metadata from willmiao ComfyUI-Lora-Manager.
    • Concept & Training: Tools for cross-provider concept images, supervising ostris/ai-toolkit training runs, and managing training packs.
    • Cloud Connectivity: Dedicated connectors for RunComfy as an alternative cloud backend to RunPod.
    • Media Curation: Integration with PhotoMapAI for searching, indexing, and curating albums, which can then be exported as training packs via the Vault service.

  12. Overview of Ideogram 4 (Ideogram Ultra) Workflows

    main

    Ideogram 4 (Ideogram Ultra) is a local open-weights pipeline for ComfyUI, not a hosted API. It is optimized for high-quality text rendering, typography, and precise graphic design layouts.

    Unlike standard text-to-image workflows, this pipeline uses a structured JSON prompt (compositional deconstruction) to provide precise control over object placement and text via bounding boxes.

    Key Architectural Concepts:

    • Dual Models: Uses two UNETs (a conditional model and an unconditional model) via a DualModelGuider node for asymmetric classifier-free guidance. There is no negative text prompt; instead, negative conditioning is handled by ConditioningZeroOut.
    • Dual Text Models:
      • qwen3vl_8b_fp8_scaled is the actual diffusion text encoder (loaded via CLIPLoader with type ideogram4).
      • gemma4_e4b_it_fp8_scaled is an optional model used only within a prompt-builder subgraph to generate the structured JSON from plain text ideas.