NodeAV

NodeAV supports all hardware acceleration methods available in FFmpeg. You can use HardwareContext.auto() to automatically detect the best available hardware, or manually specify a hardware type using HardwareContext.create() with constants like AV_HWDEVICE_TYPE_CUDA or AV_HWDEVICE_TYPE_VAAPI.

import { HardwareContext } from 'node-av/api';
import { FF_ENCODER_LIBX264 } from 'node-av/constants';

// Automatically detect best available hardware
const hw = HardwareContext.auto();
console.log(`Using hardware: ${hw.deviceTypeName}`);

// Use with decoder
const decoder = await Decoder.create(stream, {
  hardware: hw
});

// Use with encoder (use hardware-specific codec)
const encoderCodec = hw?.getEncoderCodec('h264') ?? FF_ENCODER_LIBX264;
const encoder = await Encoder.create(encoderCodec, {
  decoder,
});

import { AV_HWDEVICE_TYPE_CUDA, AV_HWDEVICE_TYPE_VAAPI } from 'node-av/constants';

// Use specific hardware type
const cuda = HardwareContext.create(AV_HWDEVICE_TYPE_CUDA);
const vaapi = HardwareContext.create(AV_HWDEVICE_TYPE_VAAPI, '/dev/dri/renderD128');

Use DeviceAPI to capture video, audio, or screen content directly from system devices. The API handles platform-specific backends automatically:

• macOS: AVFoundation (Video/Audio/Screen)
• Linux: V4L2 (Video), ALSA (Audio), x11grab (Screen)
• Windows: DirectShow (Video/Audio), GDIGrab (Screen)

import { DeviceAPI } from 'node-av/api';

// List devices and query capabilities
const devices = await DeviceAPI.list();
const modes = await DeviceAPI.modes(devices.find(d => d.type === 'video')!.name);

// Camera
await using camera = await DeviceAPI.openCamera({ videoDevice: 0, width: 1280, height: 720, frameRate: 30 });

// Microphone
await using mic = await DeviceAPI.openMicrophone({ audioDevice: 0, sampleRate: 48000, channels: 2 });

// Combined video + audio (macOS/Windows)
await using device = await DeviceAPI.openDevice({ videoDevice: 0, audioDevice: 0, width: 1280, height: 720, frameRate: 30 });

// Screen capture
await using screen = await DeviceAPI.openScreen({ frameRate: 30, drawMouse: true });

// Screen capture with system audio (macOS 13.0+)
await using screen2 = await DeviceAPI.openScreen({
  frameRate: 30,
  avfoundation: { captureSystemAudio: true, audioSampleRate: 48000, audioChannels: 2 },
});

The Low-Level API provides direct access to FFmpeg's C APIs with minimal abstractions. It is ideal when you need granular control over FFmpeg functionality, such as manually managing FormatContext, CodecContext, Packet, and Frame lifecycles. Note that this API requires careful resource management, often using the using keyword (Disposable pattern) to ensure memory is freed correctly.

import { AVERROR_EOF, AVMEDIA_TYPE_VIDEO } from 'node-av/constants';
import { Codec, CodecContext, FFmpegError, FormatContext, Frame, Packet, Rational } from 'node-av/lib';

// Open input file
await using ifmtCtx = new FormatContext();

let ret = await ifmtCtx.openInput('input.mp4');
FFmpegError.throwIfError(ret, 'Could not open input file');

ret = await ifmtCtx.findStreamInfo();
FFmpegError.throwIfError(ret, 'Could not find stream info');

// Find video stream
const videoStreamIndex = ifmtCtx.findBestStream(AVMEDIA_TYPE_VIDEO);
const videoStream = ifmtCtx.streams?.[videoStreamIndex];

if (!videoStream) {
  throw new Error('No video stream found');
}

// Create codec
const codec = Codec.findDecoder(videoStream.codecpar.codecId);
if (!codec) {
  throw new Error('Codec not found');
}

// Allocate codec context for the decoder
using decoderCtx = new CodecContext();
decoderCtx.allocContext3(codec);

ret = decoderCtx.parametersToContext(videoStream.codecpar);
FFmpegError.throwIfError(ret, 'Could not copy codec parameters to decoder context');

// Open decoder context
ret = await decoderCtx.open2(codec, null);
FFmpegError.throwIfError(ret, 'Could not open codec');

// Process packets
using packet = new Packet();
packet.alloc();

using frame = new Frame();
frame.alloc();

while (true) {
  let ret = await ifmtCtx.readFrame(packet);
  if (ret < 0) {
    break;
  }

  if (packet.streamIndex === videoStreamIndex) {
    // Send packet to decoder
    ret = await decoderCtx.sendPacket(packet);
    if (ret < 0 && ret !== AVERROR_EOF) {
      FFmpegError.throwIfError(ret, 'Error sending packet to decoder');
    }

    // Receive decoded frames
    while (true) {
      const ret = await decoderCtx.receiveFrame(frame);
      if (ret === AVERROR_EOF || ret < 0) {
        break;
      }

      console.log(`Decoded frame ${frame.pts}, size: ${frame.width}x${frame.height}`);

      // Process frame data...

      frame.unref();
    }
  }

  packet.unref();
}

// Flush the decoder
ret = await decoderCtx.sendPacket(null);
if (ret < 0 && ret !== AVERROR_EOF) {
  FFmpegError.throwIfError(ret, 'Failed to flush decoder');
}

while (true) {
  const ret = await decoderCtx.receiveFrame(frame);
  if (ret === AVERROR_EOF || ret < 0) {
    break;
  }

  console.log(`Decoded frame ${frame.pts}, size: ${frame.width}x${frame.height}`);

  // Process frame data...

  frame.unref();
}

NodeAV supports multiple input methods via the Demuxer API:

• Files or Network: Pass a file path or a network URL (e.g., rtsp://...).
• Buffers: Pass a Node.js Buffer directly to Demuxer.open().
• Custom I/O Callbacks: Implement IOInputCallbacks to provide custom read and seek logic.
• Raw Media: Open raw video (e.g., YUV) or raw audio (e.g., PCM) by specifying the type, dimensions, pixel format, or sample rate.

// From Files or Network
const media = await Demuxer.open('input.mp4');
// or
const media = await Demuxer.open('rtsp://example.com/stream');

// From Buffers
import { readFile } from 'fs/promises';
const buffer = await readFile('input.mp4');
const media = await Demuxer.open(buffer);

// Custom I/O Callbacks
import type { IOInputCallbacks, IOOutputCallbacks } from 'node-av/api';

const inputCallbacks: IOInputCallbacks = {
  read: (size: number) => {
    return buffer; // or null for EOF
  },
  seek: (offset: bigint, whence: number) => {
    return newPosition;
  }
};

await using input = await Demuxer.open(inputCallbacks, { format: 'mp4' });

// Raw video input
const rawVideo = await Demuxer.open({
  type: 'video',
  input: 'input.yuv',
  width: 1280,
  height: 720,
  pixelFormat: AV_PIX_FMT_YUV420P,
  frameRate: { num: 30, den: 1 }
});

// Raw audio input
const rawAudio = await Demuxer.open({
  type: 'audio',
  input: 'input.pcm',
  sampleRate: 48000,
  channels: 2,
  sampleFormat: AV_SAMPLE_FMT_S16
}, { format: 's16le' });