DOCSEARCHbyAlgolia
Submit Docs

VidPipe

repository·main·Indexed 42 days ago

https://github.com/htekdev/vidpipe

An agentic AI video editor and content ideation engine (v1.3.0) that automates the workflow of turning raw video recordings into social-media-ready content. It uses AI agents, Whisper, and Gemini to generate transcripts, summaries, short clips, and social media posts.

Tokens
130.6K
Snippets
350
Records
521
Agent score
52%

What's inside VidPipe

  1. Overview of vidpipe outputs

    main

    After the processing pipeline completes, vidpipe generates a comprehensive content package including:

    • Transcripts: Word-level JSON with timestamps.
    • Edited Video: Versions with silence removed and captions burned in.
    • Short Clips: 15–60s highlights in portrait, square, and feed formats.
    • Medium Clips: 1–3 min standalone segments with transitions.
    • Chapters: YouTube timestamps, Markdown, JSON, and FFmpeg metadata.
    • Social Posts: Platform-tailored drafts for TikTok, YouTube, Instagram, LinkedIn, and X.
    • Blog Post: Dev.to-style articles with web-sourced links.
    • Summary: Markdown README containing key-frame screenshots.

  2. Overview of Medium-Form Video Generation

    main

    Medium-form video generation (1–3 minutes) is a pipeline stage in vidpipe designed to extract complete topics, mini-stories, or full explanations from longer recordings. This complements the existing 15–60s shorts pipeline by focusing on narrative depth rather than single punchlines or moments.

    Supported Clip Types

    • Deep Dive: A single contiguous 1–3 minute section where the speaker covers one topic.
    • Compilation: Multiple non-contiguous segments concatenated together with transitions (e.g., crossfades) to create a 'best of' or multi-topic clip.

    Implementation Details

    • Agent: Handled by MediumVideoAgent (located in src/agents/MediumVideoAgent.ts).
    • Pipeline Stage: PipelineStage.MediumClips runs after the Shorts stage.
    • Output: Clips are saved to the medium-clips/ directory, including .mp4, .ass (subtitles), and .md (metadata) files.
    • Skip Flag: You can skip this stage by setting the SKIP_MEDIUM_CLIPS environment variable (controlled via the --no-medium-clips flag).

  3. Quick Reference for Social Platform Strategies

    main

    This guide provides a high-level comparison of posting frequencies, content types, and video support across major social platforms to help plan content distribution.

    PlatformPosting FrequencyBest Content TypesVideo Support
    TikTok2–5 posts/weekShort video, medium video✅ Required
    YouTube1 video/week + 1–3 Shorts/weekLong-form video, Shorts✅ Required
    Instagram3–5 posts/weekReels, carousels, feed posts✅ Strong
    LinkedIn2–5 posts/weekText, carousels, native video⚠️ Optional
    X (Twitter)3–5 posts/dayText, short video, threads⚠️ Optional

  4. Use the ideaService for managing ideation records

    main

    The ideaService is a business logic service used to manage ideation records backed by GitHub. It handles the lifecycle of an idea (from 'ready' to 'recorded' or 'published') and can use an LLM provider to match ideas against video transcripts.

    Core Capabilities

    • Retrieve Ideas: Fetch specific ideas by their IDs or retrieve all ideas currently in the ready status.
    • Lifecycle Management:
      • Mark as Recorded: Transition an idea to status = 'recorded' and associate it with a sourceVideoSlug.
      • Mark as Published: Transition an idea to status = 'published' and append a record to its publishedContent array.
    • AI-Powered Matching: Use matchIdeasToTranscript() to automatically find the most relevant ideas for a given video transcript. This method compares the transcript summary against the summaries of all ready ideas using a configured LLM.

    Matching Behavior

    When calling matchIdeasToTranscript():

    • It only considers ideas with a status of ready.
    • It returns up to three matched ideas, ordered by relevance.
    • It returns an empty array if there are no ready ideas, if the LLM provider is unavailable, or if matching/parsing fails.

  5. Manage content ideas with ideaService

    main

    The ideaService is a GitHub-backed L3 service used to create, update, query, and manage the lifecycle of content ideas. It uses GitHub Issues as the primary storage mechanism, where idea content is stored in the issue body (using a specific markdown format), metadata is derived from labels, and history (like video links and publishing records) is reconstructed from structured comments.

    Core Capabilities:

    • Creation & Updates: Create new ideas with structured markdown and labels, or update existing ones.
    • Discovery: List, search, and find related ideas based on tags.
    • Lifecycle Management: Mark ideas as recorded (linking a video) or published (recording a publish event).
    • History Tracking: Retrieve the publishing history of an idea by parsing its comments.

  6. How the vidpipe pipeline works

    main

    Vidpipe uses a 15-stage pipeline to transform a single video input into a complete content package. The stages include:

    1. Transcribe: Converts audio to text.
    2. Silence Removal: Removes dead air.
    3. Captions: Generates caption data.
    4. Caption Burn: Embeds captions into the video.
    5. Shorts: Extracts 15-60s clips.
    6. Medium Clips: Extracts 1-3min segments.
    7. Chapters: Creates timestamped segments.
    8. Summary: Generates a Markdown summary.
    9. Social Posts: Creates short and medium platform-specific posts.
    10. Queue Build: Prepares content for publishing.
    11. Blog: Generates a long-form article.

  7. Use ideaContext builders to inject creator intent into prompts

    main

    The ideaContext module provides pure prompt-building helpers designed to inject creator intent from Idea[] records into content-generation agent system prompts. These functions transform structured idea data into natural language context for LLMs.

    Available builders include:

    • buildIdeaContext(): Generates context including the idea topic, hook, audience, key takeaway, and prioritization guidance (tailored for clip planning or social posting).
    • buildIdeaContextForPosts(): Similar to the standard builder, optimized for social posting context.
    • buildIdeaContextForSummary(): Condenses ideas into a theme list that maps each topic to its key takeaway.
    • buildIdeaContextForBlog(): Generates context including editorial angle, audience, key takeaway, and talking points for each idea.

    Note: If no ideas are provided to a builder, it will return an empty string.

    // Example conceptual usage based on specification
import { buildIdeaContext, buildIdeaContextForBlog } from './ideaContext';

const ideas = [{ topic: '...', hook: '...', ... }];

// For general clip/social planning
const generalContext = buildIdeaContext(ideas);

// For long-form blog generation
const blogContext = buildIdeaContextForBlog(ideas);

  8. Use custom vocabulary to improve Whisper transcription

    main

    To prevent OpenAI Whisper from misspelling product names, technical jargon, or proper nouns, add them to the customVocabulary array in your brand.json. These terms act as prompt hints to improve transcription accuracy for your specific domain.

    "customVocabulary": [
  "GitHub Copilot",
  "VS Code",
  "TypeScript",
  "Kubernetes",
  "PostgreSQL"
]

  9. Detect video scene and shot transitions

    main

    To segment a video into semantically meaningful scenes (e.g., for improved chapter splitting), you can use shot detection tools. For screen recordings, these tools are effective at detecting slide transitions, IDE tab switches, and application changes.

    Recommended tools:

    • PySceneDetect: A mature Python CLI/API that supports content-aware, threshold, and adaptive detectors.
    • TransNetV2: A neural network (DDCNN architecture) that is particularly effective at handling both hard cuts and gradual transitions.
    • AutoShot: A neural architecture search-based model that outperforms TransNetV2 on the SHOT dataset.
    • FFmpeg select filter: A built-in, simple method using the scenecut_threshold filter for detecting hard cuts.

  10. Understand the PipelineSpec configuration system

    main

    The PipelineSpec is a declarative configuration system used to control video processing pipelines. It determines which processing stages run, how clips are selected, and how content is distributed.

    Specs can be defined as YAML/JSON files, built-in presets, or a combination of both. The system uses a hierarchical merging strategy: partial specifications are merged with a base preset (typically the full preset) to fill in missing values, allowing for concise configuration files.

    Built-in Presets:

    • full: The merge base; all partial specs inherit from this.
    • clean: Disables shorts and visual enhancement, focusing on substance-only output.
    • minimal: Disables clips, blog, and distribution entirely, used for cleanup-only tasks.

  11. Understand the VidPipe pipeline execution model

    main

    VidPipe executes a 15-stage pipeline for each video. A key architectural feature is that each stage is wrapped in runStage(), which catches errors locally.

    Failure Behavior:

    • A stage failure does not abort the entire pipeline.
    • Subsequent stages proceed using whatever data is currently available.
    • Exception: If the Ingestion stage (Stage 1) fails, the pipeline aborts immediately because all subsequent stages depend on the metadata and file structure created during ingestion.

    Error Handling Details: runStage() performs the following:

    1. Records the current stage for cost tracking.
    2. Executes the stage function within a try/catch block.
    3. Logs success or failure along with the wall-clock duration.
    4. Pushes a StageResult record (containing success status, error message, and duration in ms).
    5. Returns undefined on failure, allowing callers to null-check and decide how to proceed.

  12. Understand vidpipe configuration priority

    main

    vidpipe resolves configuration values using a hierarchical approach. The first match found in the following order is used:

    1. CLI flags (e.g., --openai-key sk-...)
    2. Environment variables (e.g., OPENAI_API_KEY=sk-...)
    3. .env file (automatically loaded from the current working directory)
    4. Defaults (built-in fallback values)