DOCSEARCHbyAlgolia
Submit Docs

Claude Code Video Vision

repository·main·Indexed 21 days ago

https://github.com/jordanrendric/claude-video-vision

An MCP server plugin for Claude Code that enables video perception. It extracts frames via ffmpeg and processes audio using Gemini API, OpenAI Whisper API, or local Whisper (whisper.cpp/openai-whisper) backends. The tool supports analyzing local files and YouTube URLs, featuring smart video analysis for scene change detection, silence intervals, and timestamped transcriptions.

Tokens
14.6K
Snippets
37
Records
57
Agent score
76%

What's inside claude-video-vision

  1. How the Video Session System works

    main

    The session system enables multi-pass, context-aware video analysis by persisting extracted data between tool calls.

    Identification

    Videos are identified by a SHA-256 hash of the first 64KB of the file plus the total file size. This ensures that the same video is recognized even if moved between directories.

    Persistence and Configuration

    • enable_index: true: Extracted data (manifests, analysis, and frames) is stored in ~/.claude-video-vision/sessions/{video-hash-12chars}/. This allows for efficient re-use of frames and analysis across different tool calls.
    • enable_index: false (default): Behavior remains stateless; all data is stored in /tmp/ and deleted after the tool returns.

    Lifecycle

    • Sessions are created on the first call to video_watch, video_analyze, or video_detail.
    • Sessions are automatically cleaned up after session_max_age_days (default: 7 days).
    • You can manually clear sessions using video_configure with clear_sessions: true.
    interface Config {
  enable_index: boolean;          // default: false
  session_max_age_days: number;   // default: 7
}

  2. Understand data flow and backend privacy options

    main

    The claude-video-vision plugin processes video and audio data based on the backend you select during /setup-video-vision. Data is extracted locally using ffmpeg and then handled according to your choice:

    • Local (Whisper): Runs 100% offline. No data leaves your machine. On first use, it downloads public Whisper model weights from HuggingFace.
    • Gemini API: Audio is sent to Google's Gemini API (https://generativelanguage.googleapis.com). Requires a GEMINI_API_KEY environment variable.
    • OpenAI Whisper API: Audio is sent to OpenAI's Whisper API (https://api.openai.com). Requires an OPENAI_API_KEY environment variable.

    Note for sensitive data: Use the Local (Whisper) backend if you are processing videos containing sensitive, regulated (HIPAA), or confidential material to ensure no data is transmitted to third-party APIs.

  3. The Smart Video Analysis workflow

    main

    For optimal results when using Claude with video, follow the 'analyze-first' workflow:

    1. Information Gathering: Use initial tools to understand the video context.
    2. Analyze: Run video_analyze to get a high-level overview and metadata.
    3. Plan Segments: Based on the analysis, identify specific time ranges of interest.
    4. Watch with Sampling: Use video_watch with the view_sample parameter to quickly scan segments.
    5. Drill-Down: Use video_detail to inspect specific frames or high-resolution details in the identified segments.

    Efficiency Tip: Use a binary search approach—start with narrow segments and expand only if necessary.

  4. Understanding Video Perception Results

    main

    The tools return several data types that should be combined for full understanding:

    • Manifest: (If enable_index is on) An index of all cached frames by resolution and timestamp. Use this to avoid redundant requests.
    • Frames: Images representing visual content.
    • Audio transcription: Text with timestamps.
    • Audio tags: Non-speech events like music or specific sounds.
    • Analysis data: Structural data including scene changes, silence intervals, and motion levels.

    Mental Model: Use analysis and transcription to determine WHEN things happen, and use frames to determine WHAT happens.

  5. Capabilities and limitations of Smart Video Analysis

    main

    When using the Smart Video Analysis features, be aware of the following technical constraints:

    Supported Analysis:

    • Scene change detection
    • Black interval detection
    • Silence and loudness detection
    • Freeze frame and motion detection
    • Blur, exposure, and brightness statistics
    • Transcription

    Unsupported Features (Non-Goals):

    • OCR/text detection
    • Face detection or object recognition
    • ML-based content classification
    • Remote/cloud session synchronization
    • Real-time streaming analysis

  6. Choose an audio processing backend

    main

    The plugin supports three different backends for audio transcription. All backends use ffmpeg for video frame extraction.

    BackendAudio processingCostSetup Requirements
    Gemini APINative (speech + non-speech events)Free tier: 1500 req/dayGEMINI_API_KEY env var
    Local (Whisper)whisper.cpp or Python openai-whisperFree, fully offlinebrew install whisper-cpp + auto model download
    OpenAI APIOpenAI Whisper APIPaid per usageOPENAI_API_KEY env var

  7. Requirements for Claude Video Vision

    main

    Ensure your environment meets the following requirements:

    • Node.js 20+: Required for the MCP server.
    • ffmpeg: Required for frame extraction (the setup wizard can help install this).
    • yt-dlp: Required only if you intend to process YouTube URLs (brew install yt-dlp on macOS).
    • Backend Credentials:
      • Gemini: GEMINI_API_KEY environment variable.
      • OpenAI: OPENAI_API_KEY environment variable.
      • Local: whisper.cpp installed via brew install whisper-cpp (macOS).

  8. Verify dependencies and test setup

    main

    After configuration, use the following tools to ensure the environment is ready:

    1. Verify Dependencies: Call video_setup with your configured backend and options. This checks for required binaries like ffmpeg. Note that yt-dlp is required if you intend to support YouTube URLs.
    2. Test the Pipeline: Once setup is verified, you can run a test by calling video_watch on a local video file path to see a brief summary of the results.
    video_setup(backend, options);
video_watch("path/to/video.mp4");

  9. Configure frame extraction and perception mode

    main

    Control how video frames are extracted and presented to Claude using video_configure.

    Resolution

    Set the width in pixels (height scales automatically):

    • 256px, 512px (default), 768px, or 1024px.

    FPS (Frames Per Second)

    • auto (recommended): Adapts extraction rate based on video duration.
    • Custom value: A specific numeric rate.

    Frame Mode

    Determines how Claude perceives the video:

    • Images (default): Claude receives actual image frames. Provides better visual perception but uses more tokens.
    • Descriptions: A sub-agent describes each frame as text. Uses fewer tokens but loses visual nuance. If using this mode, you can specify a frame_describer_model (Sonnet [default], Opus, or Haiku).
    video_configure(
  frame_resolution: number,
  default_fps: "auto" | number,
  frame_mode: "Images" | "Descriptions",
  frame_describer_model?: "Sonnet" | "Opus" | "Haiku"
)

  10. Standard Video Analysis Workflow

    main

    To analyze a video effectively, follow these steps in order:

    1. Get Metadata: Start with video_info to determine duration, resolution, and audio presence. For YouTube URLs, pass the URL directly as the path.
    2. Analyze Structure (Required for videos > 30s): Call video_analyze before extracting frames. Select filters based on intent:
      • General/Unclear: scene_changes, silence, transcription (always include transcription: true if audio exists).
      • Scene Transitions: scene_changes, black_intervals.
      • Motion/Action: motion.
      • Talking Heads/Stuck parts: freeze, blur.
      • Audio/Music: silence, loudness.
      • Lighting: exposure.
    3. Plan Extraction: Use analysis/transcription to set FPS. Use low FPS (0.1-0.5) for static parts and higher FPS (1-3) for scene changes or speech-referenced moments.
    4. Extract Frames:
      • Short videos (< 2 mins): Use fps: "auto" without view_sample for full coverage.
      • Long videos (> 2 mins): Use segments with variable FPS and view_sample to limit initial frame count.
    5. Drill Down: Use video_detail for specific moments. Use view_sample: 3 to preview (first, middle, last frame) before requesting specific timestamps with view.
    6. Follow-up: Consult the existing manifest in context. Do not re-extract or re-request frames already available.

  11. Configure Local Whisper settings

    main

    If the local backend is selected, you must configure the Whisper engine, model, and audio detection settings using video_configure.

    Whisper Engine

    Choose the implementation for transcription:

    • whisper.cpp: Faster, lower RAM usage, optimized for Mac/Linux.
    • openai-whisper: Python-based, more flexible and extensible.

    Whisper Model

    Select a model based on your available RAM:

    • tiny (75MB): Very fast, basic quality.
    • small (500MB): Balanced speed and quality.
    • large-v3-turbo (1.5GB): Recommended for 8GB+ RAM.
    • large-v3 (2.9GB): Maximum quality, recommended for 16GB+ RAM.
    • auto: Automatically selects based on hardware.

    Audio Tags (Whisper-AT)

    Enable whisper_at to detect non-speech audio events like coughing, music, or animal sounds. This requires Whisper-AT to be installed.

    video_configure(
  whisper_engine: "whisper.cpp" | "openai-whisper",
  whisper_model: "tiny" | "small" | "large-v3-turbo" | "large-v3" | "auto",
  whisper_at: boolean
)

  12. Install the Claude Code Video Vision plugin

    main

    To install the plugin within Claude Code, run the following commands one at a time:

    1. Add the plugin from the marketplace: /plugin marketplace add https://github.com/jordanrendric/claude-video-vision
    2. Install the plugin: /plugin install claude-video-vision

    The MCP server will automatically install via npx from npm on its first use.

    Alternative: Local Development If you are developing locally, you can clone the repository and point Claude to the plugin directory:

    git clone https://github.com/jordanrendric/claude-video-vision.git
claude --plugin-dir /path/to/claude-video-vision
    /plugin marketplace add https://github.com/jordanrendric/claude-video-vision
/plugin install claude-video-vision