VideoKit for Unity

Use the VideoKitRecorder component to record different media types, including:

• MP4, HEVC, and WEBM videos
• Animated GIF images
• JPEG image sequences
• WAV audio files

VideoKit requires an access key to function.

1. Log in to videokit.ai and generate an access key.
2. In the Unity Editor, navigate to Project Settings > VideoKit and enter your key.

CAUTION

If your project is open-source, add ProjectSettings/VideoKit.asset to your .gitignore to prevent leaking your access key.

To record microphone audio and play it back in Unity, use the VideoKitRecorder component to capture the audio. Once recording is finished, you can access the resulting MediaAsset via a handler method (e.g., OnRecordingCompleted). Use the MediaAsset.ToAudioClip() method to convert the recorded media into a Unity AudioClip, which can then be played using an AudioSource component.

// Conceptual workflow:
// 1. Use VideoKitRecorder to record audio.
// 2. Receive the MediaAsset in a handler.
// 3. Convert to AudioClip and play.

public void OnRecordingCompleted(MediaAsset recordedAsset) 
{
    AudioClip clip = recordedAsset.ToAudioClip();
    audioSource.clip = clip;
    audioSource.Play();
}

To record ultra-high frame rate video (e.g., 120FPS or 240FPS), do not use the standard VideoKitRecorder component. The VideoKitRecorder relies on the VideoKitCameraView component, which is limited by the application's frame rate rather than the camera's actual preview frame rate.

Instead, follow this pattern:

1. Register a listener to the VideoKitCameraManager.OnPixelBuffer event to capture raw camera device pixel buffers at the hardware's high frame rate.
2. Convert the captured buffers to RGBA8888 format using the PixelBuffer.CopyTo method.
3. Append these converted buffers directly to your recorder.

To install VideoKit, add the required scoped registries and the ai.videokit.videokit dependency to your Unity project's Packages/manifest.json file.

{
  "scopedRegistries": [
    {
      "name": "VideoKit",
      "url": "https://registry.npmjs.com",
      "scopes": ["ai.videokit"]
    },
    {
      "name": "Muna",
      "url": "https://registry.npmjs.com",
      "scopes": ["ai.muna"]
    }
  ],
  "dependencies": {
    "ai.videokit.videokit": "1.0.13"
  }
}

Use the AudioAsset.Caption method to convert an AudioClip into text captions.

AudioClip clip = ...;
var asset = await MediaAsset.FromAudioClip(clip);
var caption = await asset.Caption();
Debug.Log(caption);

Use the MediaAsset.Share method to open the native device share sheet for images, audio, or video files. You can convert a Texture2D into an ImageAsset first.

Texture2D image = ...
ImageAsset asset = await MediaAsset.FromTexture(image);
string receiverAppId = await asset.Share();

Use the TextAsset.To<T> method to parse a natural language prompt into a custom C# struct. This is useful for implementing voice commands or text-based user interfaces. You can use [Description] attributes on your struct fields to help guide the parsing logic.

using System.ComponentModel; // for `DescriptionAttribute"
using VideoKit.Assets;

struct Command {
    [Description(@"The user's name")]
    public string name;

    [Description(@"The user's age")]
    public int age;
}

async void ParseCommand() {
    var prompt = "My name is Jake and I'm thirteen years old.";
    var asset = await MediaAsset.FromText(prompt);
    var command = await asset.To<Command>();
    // command = { "name": "Jake", "age": 13 }
}

Use the MediaAsset.SaveToCameraRoll method to save images or videos directly to the user's device camera roll.

Texture2D image = ...
ImageAsset asset = await MediaAsset.FromTexture(image);
bool saved = await asset.SaveToCameraRoll();

Use MediaAsset.FromCameraRoll<T> to present the native system gallery UI and allow the user to select an asset. You can then convert the selected asset back into a Texture2D.

// This will present the native gallery UI
var asset = await MediaAsset.FromCameraRoll<ImageAsset>() as ImageAsset;
Texture2D image = await asset.ToTexture();
// Do stuff with `image`...

Requirements

• Unity 2022.3+

Supported Platforms

• Android: API Level 24+
• iOS: 14+
• macOS: 12+ (Apple Silicon and Intel)
• Windows: 10+ (64-bit only)
• WebGL: Requires Unity 6 (Chrome 91+, Firefox 90+, Safari 16.4+)