DOCSEARCHbyAlgolia
Submit Docs

ffmpeg-sidecar

repository·main·Indexed 19 days ago

https://github.com/nathanbabcock/ffmpeg-sidecar

A Rust crate that wraps the FFmpeg CLI to provide an intuitive Iterator interface for interacting with video as a sequence of raw RGB frames. It includes a builder API (FfmpegCommand) for configuring FFmpeg arguments, support for automatically downloading platform-specific binaries, and utilities for handling raw video frames and execution events.

Tokens
8.6K
Snippets
34
Records
41
Agent score
68%

What's inside ffmpeg-sidecar

  1. How ffmpeg-sidecar works: The Iterator abstraction

    main

    The core concept of ffmpeg-sidecar is to wrap the FFmpeg CLI and expose its output as a high-level Rust Iterator.

    Instead of manually parsing CLI arguments or stderr logs, you use a builder API (similar to std::process::Command) to configure FFmpeg. When you call .spawn(), it returns a process handle that can be converted into an iterator via .iter(). This iterator allows you to loop over decoded video frames as raw RGB data, abstracting away the complexity of video decoding, container formats, and stream mappings.

    let iter = FfmpegCommand::new()
  .testsrc()
  .rawvideo()
  .spawn()?
  .iter()?;

for frame in iter.filter_frames() {
  // frame.data contains raw RGB pixels
}

  2. Realtime transcription from microphone (Windows)

    main

    To transcribe live audio from a microphone on Windows, follow these steps:

    1. Find your audio device name: Run ffmpeg -list_devices true -f dshow -i dummy to identify the exact name of your microphone (e.g., "Microphone (Realtek(R) Audio)").

    2. Capture and transcribe: Use the dshow input format with the whisper filter. Use -f null - to prevent FFmpeg from trying to write an output media file, as the transcription is being sent to stdout instead.

    Example Command:

    # 1. List devices
ffmpeg -list_devices true -f dshow -i dummy

# 2. Transcribe
ffmpeg -hide_banner -loglevel error -f dshow -i audio="Microphone (Realtek(R) Audio)" -af "whisper=model=./whisper.cpp/models/ggml-base.en.bin:destination=-:queue=2" -f null -

  3. Run project examples

    main

    The repository includes several examples demonstrating different use cases. You can run them using cargo run --example <name>:

    • Hello World: cargo run --example hello_world (Read raw video frames)
    • H265 Transcoding: cargo run --example h265_transcode (Decode H265, modify frames, and re-encode)
    • FFplay Preview: cargo run --example ffplay_preview (Pipe FFmpeg to FFplay for debugging)
    • Named Pipes: cargo run --example named_pipes --features named_pipes (Pipe multiple outputs into a Rust program)

  4. Automatically download and install FFmpeg

    main

    You can automatically download and install a platform-specific FFmpeg binary (Windows, MacOS, or Linux) by calling ffmpeg_sidecar::download::auto_download(). This can be used for development setup or included in your application to handle runtime installation.

    To skip installing ffplay and ffprobe and only install the core ffmpeg binary, set the KEEP_ONLY_FFMPEG environment variable to 1 or true before calling the download function.

    ffmpeg_sidecar::download::auto_download().unwrap();

  6. Download Whisper ggml models

    main

    The whisper filter requires a local model in ggml format. You can obtain these by using the whisper.cpp repository's download script.

    1. Clone the repository: git clone https://github.com/ggml-org/whisper.cpp.git
    2. Navigate to the directory: cd whisper.cpp
    3. Run the download script for a specific model (e.g., base.en): sh ./models/download-ggml-model.sh base.en
    git clone https://github.com/ggml-org/whisper.cpp.git
cd whisper.cpp
sh ./models/download-ggml-model.sh base.en

  8. Use FfmpegCommand to build FFmpeg commands

    main

    The FfmpegCommand struct is a builder interface that wraps std::process::Command to provide convenient, type-safe aliases for common FFmpeg arguments. It is designed to simplify the construction of complex FFmpeg CLI strings while ensuring compatibility with the project's log parser.

    Key features include:

    • Automatic Configuration: New instances automatically set up piped stdin, stdout, and stderr, configure a specific log level (-loglevel level+info), and disable console window creation on Windows.
    • Safety: The spawn() method automatically prevents interactive overwrite prompts by ensuring -n (no overwrite) is present if -y or -n are not explicitly set.
    • Presets: Includes built-in presets for common tasks like generating test video or emitting raw video to stdout.
    • Escape Hatches: You can access the underlying std::process::Command using as_inner() or as_inner_mut() for any functionality not covered by the builder.
    use ffmpeg_sidecar::FfmpegCommand;

let mut cmd = FfmpegCommand::new();
cmd.input("input.mp4")
   .codec_video("libx264")
   .output("output.mp4");

let mut child = cmd.spawn().expect("failed to spawn");
// ... use child

  9. Read raw video frames using FfmpegCommand

    main

    Use the FfmpegCommand builder to construct an FFmpeg command and iterate over decoded frames. The API provides discoverable aliases for common FFmpeg arguments and argument presets for common tasks like generating test sources or outputting raw video.

    use ffmpeg_sidecar::command::FfmpegCommand;

fn main() -> anyhow::Result<()> {
  // Run an FFmpeg command that generates a test video
  let iter = FfmpegCommand::new() // <- Builder API like `std::process::Command`
    .testsrc()  // <- Discoverable aliases for FFmpeg args
    .rawvideo() // <- Convenient argument presets
    .spawn()?   // <- Ordinary `std::process::Child`
    .iter()?;   // <- Blocking iterator over logs and output

  // Use a regular "for" loop to read decoded video data
  for frame in iter.filter_frames() {
    println!("frame: {}x{}", frame.width, frame.height);
    let _pixels: Vec<u8> = frame.data; // <- raw RGB pixels! 🎨
  }

  Ok(())
}

  10. Use the FFmpeg `whisper` filter

    main

    The whisper audio filter (-af) allows for realtime transcription.

    Key Parameters:

    • model: The file path to the downloaded .bin ggml model.
    • destination=-: Using - outputs the transcription text to stdout via FFmpeg AVIO syntax.
    • queue: The number of seconds to buffer before processing. Increasing this can improve accuracy at the cost of higher latency.

    Minimal Example (Dummy Audio):

    ffmpeg -f lavfi -i "sine=frequency=1000:duration=5" -af "whisper=model=./whisper.cpp/models/ggml-base.en.bin:destination=-:queue=2" -f null -

  11. Configure FFmpeg installation via environment variables

    main

    You can control the contents of the FFmpeg installation using the KEEP_ONLY_FFMPEG environment variable.

    VariableValueEffect
    KEEP_ONLY_FFMPEG1 or true (case-insensitive)Installs only the ffmpeg binary. Skips ffplay and ffprobe.
    KEEP_ONLY_FFMPEG(not set) or any other valueInstalls ffmpeg, ffplay, and ffprobe (where available).

  12. Use the FfmpegCommand builder to stream video frames

    main

    FFmpeg Sidecar provides a builder API similar to std::process::Command to wrap FFmpeg binaries. You can use discoverable aliases for common FFmpeg arguments (like .testsrc() or .rawvideo()) to configure the command. Once spawned, you can call .iter() to get a blocking iterator over the output, and use .filter_frames() to iterate specifically over decoded video frames containing raw pixel data.

    use ffmpeg_sidecar::command::FfmpegCommand;

fn main() -> anyhow::Result<()> {
  // Run an FFmpeg command that generates a test video
  let iter = FfmpegCommand::new() // <- Builder API like `std::process::Command`
    .testsrc()  // <- Discoverable aliases for FFmpeg args
    .rawvideo() // <- Convenient argument presets
    .spawn()?   // <- Ordinary `std::process::Child`
    .iter()?;   // <- Blocking iterator over logs and output

  // Use a regular "for" loop to read decoded video data
  for frame in iter.filter_frames() {
    println!("frame: {}x{}", frame.width, frame.height);
    let _pixels: Vec<u8> = frame.data; // <- raw RGB pixels! 🎨
  }

  Ok(())
}