Diffusion Studio Core Documentation

A Composition object is the central state manager for a video project in Diffusion Studio. It manages tracks, clips, and the overall timeline.

To create a new composition, import Composition from @diffusionstudio/core. You can optionally provide configuration for the canvas size and background color.

Default Configuration:

• height: 1080
• width: 1920
• background: '#000000'

Note that width and height define the canvas size for editing/visualization, but the final output resolution is determined by the Encoder during export.

import * as core from '@diffusionstudio/core';

const composition = new core.Composition();

A mask defines a specific region within a composition where content is visible. It is used to crop, hide, or emphasize specific parts of a video clip. While this guide focuses on RectangleMask, the same principles apply to other mask types like CircleMask.

import * as core from '@diffusionstudio/core';

const composition = new core.Composition();
const mask = new core.RectangleMask({
  x: 480,
  y: 0,
  width: composition.width - 960,
  height: composition.height,
  radius: 100,
});

await composition.add(
  new core.VideoClip(new File([], 'sample.mp4'), {
    mask,
  })
);

A Track acts as a container that automatically bubbles up events from its constituent clips. Instead of attaching listeners to every individual clip, you can listen for clip events directly on the Track.

Additionally, Tracks emit lifecycle events when they are added to or removed from a Composition:

• attach: Emitted when the track is added to a composition.
• detach: Emitted when the track is removed from a composition.

track.on('attach', console.log);
track.on('detach', console.log);

Tracks are an abstraction layer that provide two primary functions:

1. Efficient Clip Rendering: Tracks determine which clips are rendered. Only visible clips within a track are processed, allowing the engine to bypass unnecessary iterations over non-visible clips.
2. Layering Control: Tracks define the visual stacking order. The track at index 0 is rendered last, meaning it appears on top of all other tracks.

To include a track in a video, you must add it to a Composition using composition.shiftTrack(track) or create it via composition.createTrack(type).

When using a Layer with mode: 'SEQUENTIAL', clips added to the layer are automatically aligned. If you split a clip or change its range, the subsequent clips in the layer will have their delay automatically adjusted to follow the previous clip.

const layer = await composition.add(
  new core.Layer({ mode: 'SEQUENTIAL' })
);

await layer.add(new core.AudioClip(source));
await layer.clips[0].split(8);

// Delays for subsequent clips are automatically adjusted
layer.clips[0].range = [0.5, 4];
layer.clips[1].range = [14, 16];

Instead of manually managing offsets for split clips, you can use a stacked track. In a stacked track, offsets are handled automatically when clips are added or split.

1. Create a track using composition.appendTrack(core.AudioTrack).
2. Convert it to a stacked track using .stacked().
3. Append clips to the track using track.appendClip().

Once clips are in a stacked track, you can access them via the track.clips array.

// Setup a stacked track
const track = composition.appendTrack(core.AudioTrack).stacked();

// Append a clip to the track
await track.appendClip(
  new core.AudioClip(
    await core.AudioSource.from('https://diffusion-studio-public.s3.eu-central-1.amazonaws.com/audio/piano.mp3')
  )
);

// Manipulate clips via the track reference
await track.clips[0].split(240);
track.clips[0].subclip(15, 80);
track.clips[1].subclip(420);

Checkpoints allow you to save and restore the complete state of a composition, including all layers, clips, and their configurations. This is useful for persisting project state, implementing undo/redo functionality, or saving/loading projects.

A checkpoint is a JSON-serializable object. However, because assets (sources) can be large or complex, they are managed separately from the composition state. To successfully restore a project, you must save both the checkpoint object and the serialized assets.

// The checkpoint captures the composition structure
const checkpoint = await composition.createCheckpoint();

// The assets capture the source data
const assets = core.serializeSources(sources);

To work effectively with the Diffusion Studio API, familiarize yourself with these core abstractions:

• Asset: A serializable object representing a file, containing its location and MIME type.
• Source: A class used to prepare an asset for rendering. It handles data caching and allows sharing resources between multiple clips.
• Composition: The top-level root object that holds all elements to be rendered in a single video.
• Layer: A chronological sequence of clips of a specific type (e.g., a layer dedicated to video clips).
• Clip: An object defining specific rendering parameters, such as its position and duration of visibility within the video.
• Framerate: The display rate in frames per second (FPS). The default is 30 FPS if not explicitly specified.
• Encoder: The tool responsible for compressing frames using codecs like H.264/AVC1 or H.265/HEVC.

Tracks are an abstraction layer used to manage the video timeline. They provide three primary functions:

1. Efficient Clip Rendering: Only visible clips within a track are processed, allowing the engine to bypass unnecessary iterations over non-visible clips.
2. Layering Control: Tracks define the rendering order. The track at index 0 is rendered last, meaning it appears on top of all other tracks.
3. Clip Management: Tracks handle the lifecycle of clips, including initialization, addition, removal, and updates.

Tracks are typed; for example, a TextTrack is specifically designed to hold TextClip instances or classes derived from them (like RichTextClip).

To work effectively with the API, familiarize yourself with these core concepts:

• Composition: The root object containing all assets to be rendered in a single video.
• Track: A chronological sequence of clips of the same type (e.g., video clips).
• Clip: An object containing specific rendering information, such as position and duration of visibility.
• Framerate: The rate at which frames are displayed (default is 30 FPS).
• Encoder: The tool that compresses frames using codecs like H.264/AVC1 or H.265/HEVC.
• Renderer: The abstraction managing the rendering context (WebGL or Canvas) to draw the composition state.
• Ticker: The tool ensuring the composition state updates at the correct framerate and renders to the screen.

The engine can be used for free, but rendered videos will include a "Made with Diffusion Studio" watermark. To remove the watermark, you must purchase a license key.

Key Details:

• One-time purchase: Keys do not expire.
• Offline verification: Keys are signed payloads verified locally using a public key; no internet connection is required for verification.
• Usage: Keys can be used across as many of your own apps or domains as needed, but cannot be shared with other organizations.