DOCSEARCHbyAlgolia
Submit Docs

Twick SDK

repository·main·Indexed 18 days ago

https://github.com/ncounterspecialist/twick

A modular React SDK for building timeline-based video editors. Twick provides tools for timeline management, canvas-based editing via @twick/canvas, and browser-native video rendering using the WebCodecs API via @twick/browser-render. It includes @twick/ai-models for orchestrating generative AI workflows (voice, avatar, and media) through a provider-agnostic backend layer with support for multi-provider orchestration and normalization.

Tokens
222.6K
Snippets
690
Records
1K
Agent score
66%

What's inside twick

  1. Overview of @twick/ai-models

    main

    The @twick/ai-models package provides provider adapters and orchestration primitives for Twick generative AI integrations. It is designed to create a provider-agnostic backend layer that allows applications to integrate various caption, voice, avatar, and media APIs behind a single unified contract.

    Key capabilities include:

    • Multi-provider orchestration: Manage jobs with unified status tracking and automatic fallback mechanisms.
    • Standardized interfaces: Use a common ProviderAdapter interface for different generation workflows.
    • Data normalization: Use timeline injection helpers to convert provider outputs into standard patch contracts, and caption normalization for legacy payloads.
    • Job management: A job store interface (with an InMemoryJobStore implementation) to track generation tasks.

  2. Overview of Twick Architecture

    main

    Twick is a React SDK designed for building timeline-based video editing applications. It utilizes a modular monorepo architecture with a clear separation of concerns between foundation utilities, core components, and the application layer. The system relies on three primary architectural patterns:

    1. Context Pattern: Uses nested React Context providers for state management and component communication.
    2. Visitor Pattern: Handles timeline operations (like adding, removing, or splitting elements) through dedicated visitor classes.
    3. Observer Pattern: Coordinates state changes and component synchronization via callbacks and event listeners.

  4. What is @twick/ai-models?

    main

    The @twick/ai-models package provides provider adapters and orchestration primitives for Twick generative AI integrations. It acts as a provider-agnostic backend layer that allows applications to integrate various caption, voice, avatar, and media APIs behind a single unified contract.

    Key capabilities include:

    • Multi-provider orchestration: Manage jobs with unified status tracking and automatic fallback between providers.
    • Normalization: Use timeline injection helpers to convert provider outputs into standard patch contracts and normalize legacy cloud caption payloads.
    • Job Management: A job store interface (with an InMemoryJobStore implementation) to track the lifecycle of AI generation tasks.
    • Standardized Types: Provides core types like ModelInfo, AIModelProvider, and various patch/result DTOs.

  5. Overview of @twick/cloud-file-uploader

    main
    The @twick/cloud-file-uploader package is designed to generate secure, time-limited pre-signed S3 URLs. This allows clients (like web browsers) to upload files directly to an S3 bucket, bypassing your server or AWS Lambda. This approach reduces server load, avoids Lambda payload limits, and lowers data transfer costs by preventing files from flowing through your compute resources.

  6. Overview of the Twick React Video Editor SDK

    main

    Twick is a React-based video editing SDK designed to add high-level video editing capabilities to frontend applications. It provides a suite of tools for managing complex media timelines, canvas-based element manipulation, and live video playback through simple function calls.

    Core Capabilities

    • Timeline Management: Supports multi-track timelines with drag-and-drop reordering, CRUD operations via a visitor pattern, and undo/redo history management. Supported element types include Text, Video, Image, Audio, Rectangle, Circle, Icon, and Caption.
    • Video Editing: Features live video previews, timeline-based interfaces with zoom/seek controls, and customizable aspect ratios.
    • Canvas Editing: A React-based canvas library (built with Fabric.js) that allows users to move, resize, rotate, and reorder elements (via z-index) directly on the canvas.
    • Media Utilities: Provides metadata extraction (audio, video, image), dimension handling (object-fit calculations), and media type detection from URLs.
    • Live Player: A playback component that supports project data for complex configurations and manages play/pause states via React hooks.
    • Visualizer: A toolkit for 2D graphics, animation, and real-time preview of video compositions.

  7. Overview of @twick/visualizer

    main

    @twick/visualizer is a professional video visualization library designed for creating dynamic video content with animations, effects, and interactive elements. It is built on top of @twick/2d and @twick/core to provide high-performance 2D graphics.

    Developers can use it to manage various scene elements including video, audio, text, images, and shapes, applying complex animations like fades, blurs, and elastic effects to create professional-grade video presentations.

  8. Transcribe audio/video to JSON captions with @twick/cloud-transcript

    main

    The @twick/cloud-transcript package uses Google Gemini models (via Vertex AI) to transcribe audio or video files into JSON captions. It provides millisecond-level timestamps for each caption segment, making it suitable for video captioning. The service is designed to run as a serverless AWS Lambda container image (Linux/AMD64).

    {
  "captions": [
    {
      "t": "Example phrase 1",
      "s": 0,
      "e": 1500
    }
  ],
  "rawText": "Full raw response text from the model..."
}

  9. Twick Feature Overview

    main

    Twick is a React Video Editor SDK designed for building custom video applications. Key features include:

    • AI Caption Generation: Powered by Google Vertex AI (Gemini).
    • Timeline Editing: Timeline-based video editing capabilities.
    • Canvas Tools: Tools for video manipulation on a canvas.
    • MP4 Export: Serverless MP4 export via AWS Lambda.
    • Real-time Preview: Real-time video previewing.
    • Transcription: Video transcription API.
    • React Integration: A suite of React components for video editing.

  10. Explore @twick/media-utils exports

    main

    The @twick/media-utils package exports several classes, interfaces, and utility functions for media manipulation.

    Classes

    • VideoFrameExtractor: Used for extracting frames from video files.

    Interfaces

    • AudioSegment: Represents a segment of audio.
    • VideoFrameExtractorOptions: Configuration options for the VideoFrameExtractor.

    Key Functions

    • File & URL Utilities: blobUrlToFile, detectMediaTypeFromUrl, downloadFile, loadFile, saveAsFile.
    • Video & Audio Processing: extractAudio, getAudioDuration, getDefaultVideoFrameExtractor, getThumbnail, getThumbnailCached, getVideoMeta, hasAudio, stitchAudio.
    • Dimension & Scaling Utilities: getImageDimensions, getObjectFitSize, getScaledDimensions.
    • Other: limit.

  11. How the Visitor Pattern handles timeline operations

    main

    Timeline operations in Twick are implemented using the Visitor Pattern. Instead of direct manipulation, dedicated visitor classes are used to perform specific actions on TrackElement objects. This pattern allows for clean separation of logic for different operation types.

    Key Visitors

    • ElementAdder: Adds media or text elements to a track.
    • ElementRemover: Removes elements from the timeline.
    • ElementUpdater: Modifies element properties.
    • ElementSplitter: Splits audio/video at a specific timestamp.
    • ElementCloner: Clones existing elements.
    • ElementSerializer/ElementDeserializer: Handles storage operations.
    • ElementValidator: Validates element data.

    Implementation Example

    Visitors often use a "friend" class pattern (e.g., track.createFriend()) to access protected methods of the track or element they are operating on.

    // Example of an ElementSplitter visitor
class ElementSplitter implements ElementVisitor {
  constructor(private splitTime: number) {}
  
  visit(element: TrackElement): SplitResult {
    // Split element at specific time logic
    return element.split(splitTime);
  }
}

// Example of an ElementAdder visitor
class ElementAdder extends ElementVisitor {
  constructor(private track: Track) {
    this.track = track;
    this.trackFriend = track.createFriend();
  }
  
  async visit(element: TrackElement): Promise<boolean> {
    return trackFriend.addElement(element);
  }
}

  12. How the Observer Pattern coordinates components

    main

    The Observer Pattern is used to coordinate state changes between different parts of the system using callbacks and event listeners.

    Editor-Level Coordination

    When initializing a TimelineEditor, you provide callbacks to handle undo/redo, history resets, and timeline actions:

    const editor = new TimelineEditor({
  contextId,
  setTotalDuration,
  setPresent: undoRedoContext.setPresent,
  handleUndo: undoRedoContext.undo,
  handleRedo: undoRedoContext.redo,
  handleResetHistory: undoRedoContext.resetHistory,
  updateChangeLog: updateChangeLog,
  setTimelineAction: (action: string, payload?: unknown) => {
    setTimelineActionState({ type: action, payload });
  },
});

    Canvas Event Observation

    Canvas operations (like selecting or updating an item) are observed to propagate changes back to the timeline or editor state:

    const handleCanvasOperation = (operation, data) => {
  switch (operation) {
    case CANVAS_OPERATIONS.ITEM_SELECTED:
      setSelectedItem(data); // Propagate selection
      break;
    case CANVAS_OPERATIONS.ITEM_UPDATED:
      setTimelineAction(TIMELINE_ACTION.UPDATE_ELEMENT, {
        updates: data
      });
      break;
  }
};