DOCSEARCHbyAlgolia
Submit Docs

egaki

repository·main·Indexed 40 days ago

https://github.com/remorses/egaki

A TypeScript CLI tool for AI-powered image, video, and speech generation. It features an MDX-to-video framework and supports multiple providers including Google (Imagen, Gemini, Veo), OpenAI, xAI (Grok), and Kling. Version 0.10.0 includes tools for text-to-media generation, inpainting, voice cloning, and local audio analysis for BPM and loudness.

Tokens
102.9K
Snippets
385
Records
498
Agent score
49%

What's inside egaki

  1. Overview of Egaki core features and infrastructure

    main

    Egaki extends Remotion with a custom infrastructure designed for MDX-driven media composition and AI-integrated workflows. Key capabilities include:

    • MDX Pipeline: Parsing frontmatter, sections, durations, and aspect ratios.
    • AI Generation: A cachedGenerate Higher-Order Function (HOF) providing deterministic keys, deduplication, stale fallback, and progress tracking.
    • Animation Engine: Layout transitions using FLIP (ghost measurement and coordinate mapping) and composable animation primitives (Opacity, Scale, TranslateX, TranslateY, Blur).
    • Infrastructure: A Vite plugin managing client, rsc, and ssr environments via virtual modules (virtual:egaki-mdx, virtual:egaki-modules, virtual:egaki-app).
    • Dev Tools: Tweakpane integration for frame/section metadata and automatic media duration computation.
    • Export & Agent SDK: A web-renderer with HTML-in-canvas and a window.egakiSDK for programmatic control (screenshot, export, seek, and getElementPosition).
    • State Management: A Zustand vanilla store managing modules, section reports, and generation progress, accessible via the useSyncExternalStore pattern.

  2. Use built-in visual components in MDX

    main

    egaki provides a suite of visual components that are available in MDX files without requiring manual imports.

    Available Components:

    • Layout & Background: Fill, Background, LayoutTransition, AngledScreen, BasicAngledScreen
    • Shaders: BandsShader, WaveGradientShader, LiquidGradientShader, DispersionRingsShader
    • Reveals & Animations: BlurReveal, MaskedSlideReveal, StaggeredFadeUp, ShimmerSweep, Opacity, Scale, TranslateX, TranslateY, Blur
    • Media & Code: CodeBlock, Img, Audio, Video, GeneratedImage, GeneratedVideo, GeneratedSpeech

    Note: Animation primitives and Shaders have dedicated documentation pages for detailed usage.

  3. Overview of the Remotion Server-Side Renderer Protocol

    main

    The Remotion Server-Side Renderer Protocol defines how a headless Chrome renderer (used for CLI rendering and video export) communicates with a served web application. The protocol is framework-agnostic, meaning you can use any build tool (Vite, Webpack, Esbuild, etc.) as long as the served page exposes specific JavaScript functions and globals on the window object.

    Frame Rendering Flow

    1. Initialization: The renderer navigates to the serveUrl and injects globals via evaluateOnNewDocument().
    2. Ready State: The renderer polls window.remotion_renderReady to wait for the page to signal it is ready.
    3. Discovery: The renderer calls window.getStaticCompositions() to verify the project and window.remotion_calculateComposition(compId) to retrieve metadata (width, height, fps, etc.).
    4. Frame Loop: For each frame, the renderer calls window.remotion_setFrame(frame, compId, attempt).
    5. Capture: Once remotion_renderReady is true, the renderer takes a page.screenshot() to capture the frame.
    6. Stitching: The captured images are stitched into a video file (e.g., MP4) using FFmpeg.

  4. What is the Preamble in MDX Video?

    main

    The preamble is any content placed before the first # heading in your MDX file.

    Unlike sections, the preamble is rendered at the Remotion composition level, outside the <Series> that sequences sections. It persists across the entire video duration and renders in the background.

    Use the preamble for:

    • Soundtracks: <Audio src="/music.mp3" /> to play music throughout the video.
    • Backgrounds: <Video src="/bg.mp4" /> or <Background> shaders.
    • Persistent Overlays: Watermarks or logos.
    ---
fps: 30
bpm: 120
---

<Audio src="/soundtrack.mp3" />
<Video src="/ambient-bg.mp4" objectFit="cover" />

# First Section duration=5s

This content appears on top of the ambient background video.

  5. How Egaki manages image model pricing

    main

    Because the Vercel AI Gateway GET /v1/models endpoint often returns incomplete or zeroed pricing for image models, Egaki maintains a manual source of truth for billing and usage cap tracking.

    To ensure accurate billing, image model prices are stored in src/model-catalog.ts. Developers or maintainers should verify prices against official provider pages and update this file manually when new models are added or prices change.

  6. How the Ready State Protocol works

    main

    The remotion_renderReady protocol allows the page to signal to the renderer when async work (like loading assets) is complete and a frame is safe to screenshot.

    Implementation Logic

    1. Initialization: Set window.remotion_renderReady = false.
    2. Starting Async Work: Call delayRender(label). This function:
      • Generates a unique handle.
      • Pushes the handle to window.remotion_delayRenderHandles.
      • Sets window.remotion_renderReady = false.
      • (Optional) Sets a timeout based on window.remotion_puppeteerTimeout.
    3. Completing Async Work: Call continueRender(handle). This function:
      • Removes the handle from window.remotion_delayRenderHandles.
      • Clears any associated timeout.
      • If window.remotion_delayRenderHandles.length === 0, sets window.remotion_renderReady = true.
    4. Polling: The renderer continuously polls window.remotion_renderReady until it becomes true before taking a screenshot.
    // When starting async work
export const delayRender = (label?: string) => {
  const handle = Math.random()
  window.remotion_delayRenderHandles.push(handle)
  window.remotion_renderReady = false
  
  if (window.remotion_puppeteerTimeout) {
    window.remotion_delayRenderTimeouts[handle] = {
      label,
      timeout: setTimeout(() => { /* ... */ }, window.remotion_puppeteerTimeout - 2000)
    }
  }
  
  return handle
}

// When async work completes
export const continueRender = (handle: number) => {
  window.remotion_delayRenderHandles = 
    window.remotion_delayRenderHandles.filter(h => h !== handle)
  
  if (window.remotion_delayRenderTimeouts[handle]) {
    clearTimeout(window.remotion_delayRenderTimeouts[handle].timeout)
    delete window.remotion_delayRenderTimeouts[handle]
  }
  
  if (window.remotion_delayRenderHandles.length === 0) {
    window.remotion_renderReady = true
  }
}

  7. Use props as defaults for Tweakpane parameters

    main

    To ensure Tweakpane overrides live values while still respecting initial component props, do not destructure props. Instead, use the props as fallback values within the useTweakpane schema. This pattern allows the prop to serve as the initial value which Tweakpane then manages.

    const tp = useTweakpane('Card', {
  borderRadius: { value: props.borderRadius ?? 16, min: 0, max: 50 },
  opacity: { value: props.opacity ?? 1, max: 1, step: 0.01 },
})

  8. Understand the ChatGPT Codex image backend implementation

    main

    The egaki project mirrors ChatGPT's image generation by using the Codex client flow rather than the standard OpenAI API (api.openai.com/v1/images).

    Key implementation details:

    • Authentication: Uses ChatGPT OAuth to store access, refresh, and accountId tokens.
    • Endpoint: Requests are sent to https://chatgpt.com/backend-api/codex/responses.
    • Required Headers:
      • Authorization: Bearer <chatgpt access token>
      • ChatGPT-Account-ID: <workspace/account id>
    • Tool Payload: The standard tool definition used is:
      { "type": "image_generation", "output_format": "png" }
    • Image-to-Image: Achieved by including input_image content items in the user message, using base64 encoded data URLs.
    {
  "type": "input_image",
  "image_url": "data:image/png;base64,..."
}

  9. Define scene durations and transitions using headings

    main

    In Egaki MDX files, scenes are defined using Markdown headings. You can control the timing of each scene by appending duration and transition parameters to the heading line.

    • duration: The length of the scene in seconds (e.g., duration=2s).
    • transition: The length of the transition between this scene and the next (e.g., transition=0.3s).
    # Clouds duration=2s transition=0.3s

  10. Implement word-highlight captions

    main
    For caption-style text that tracks with audio, use an animated gradient background with plain text scrolling slowly upward. The currently spoken word should receive a highlight background. This pattern is designed to work with egaki/text-to-speech using word-level timestamps.

  11. Define video sections using headings

    main

    In Egaki MDX files, each # heading defines a section (or scene). These sections are rendered as sequential Remotion Series.Sequence components that play one after another. You can control the length of each section by appending a duration parameter to the heading.

    # Intro duration=3s

# Verse duration=8beats

# Bridge duration=90frames

# Outro duration=2s

  12. Select models for image and video generation

    main

    The --model or -m flag is optional for both egaki image and egaki video commands.

    • Interactive mode (TTY): If you omit the flag, egaki will show an interactive picker.
    • Non-interactive mode (Scripts/Pipes): If omitted, egaki uses a sensible default. Agents and automation should always pass the -m flag explicitly to avoid the interactive picker.

    Recommended Image Models:

    • gpt-image-1 (or gpt-image-2 / chatgpt-image-latest)
    • grok-imagine-image (xAI Grok)
    • nano-banana-pro-preview

    Recommended Video Models:

    • grok-imagine-video-1.5 (xAI Grok)
    # Example of explicit model selection for automation
egaki image --model gpt-image-1