ffmpeg-over-ip

When fallbackToLocal is enabled, the client attempts to run the host's local ffmpeg (or ffprobe) if the initial TCP connection to the server fails. This ensures transcoding continues even if the remote GPU machine is offline.

Key Behaviors

• Trigger: Only triggered on initial TCP connection failure. Mid-stream errors are fatal and do not trigger a local restart.
• Binary Selection: The client selects ffmpeg or ffprobe based on its own argv[0] basename. A binary named ffprobe (or containing it) looks for ffprobe on $PATH; otherwise, it looks for ffmpeg.
• Environment Security: FFMPEG_OVER_IP_* environment variables are stripped from the local process to prevent the authSecret from leaking into /proc/<pid>/environ.
• Path Searching: The client searches $PATH for the local binary. On Windows, it tries the bare name first, then follows %PATHEXT% order.

Security Considerations

• Path Hijacking: Only enable this on hosts where you trust the $PATH. A user with write access to a directory earlier in $PATH than the real binary could hijack transcodes.
• Windows %PATHEXT%: A malicious ffmpeg.com could be picked up before ffmpeg.exe depending on the declared order.
• Root Users: Audit $PATH directory permissions if running the client as root to prevent non-root users from hijacking transcodes.

Rewrites allow the server (via rewrites) or the client (via fallbackRewrites) to substitute elements in the argv array before execution. This is useful for mapping requested codecs to available hardware (e.g., requesting h264_nvenc but using h264_qsv on the server).

Rules

• Exact Matches: Each [find, replace] pair matches whole argv elements, not substrings. h264_nvenc will match, but h264_nvenc_extra will not.
• Whitespace Handling: find and replace strings are split on whitespace. This allows a single rewrite to match a run of elements and replace them with a run of a different length.
• Order of Operations: Rewrites are applied in the order they are declared. A later rewrite can match tokens produced by an earlier one.
• Global Application: The same rewrite is applied to every matching run in the argv array.

Examples

{
  "rewrites": [
    // Swap a two-element run for a different two-element run
    ["-hwaccel qsv", "-hwaccel cuda"],

    // Expand a two-element run into a four-element run
    ["-hwaccel qsv", "-hwaccel cuda -hwaccel_output_format cuda"],

    // Remove a one-element argv entry entirely
    ["-nostdin", ""],

    // Remove a two-element run
    ["-preset veryfast", ""]
  ]
}

{
  "rewrites": [
    // Swap a two-element run (-hwaccel qsv) for a different two-element run.
    ["-hwaccel qsv", "-hwaccel cuda"],

    // Expand a two-element run into a four-element run.
    ["-hwaccel qsv", "-hwaccel cuda -hwaccel_output_format cuda"],

    // Remove a one-element argv entry entirely (empty replacement).
    ["-nostdin", ""],

    // Remove a two-element run.
    ["-preset veryfast", ""],
  ],
}

ffmpeg-over-ip enables GPU-accelerated transcoding on a remote machine without requiring GPU passthrough, shared filesystems (NFS/SMB), or complex driver alignment. It uses a client-server model where the client acts as a drop-in replacement for the ffmpeg binary.

The Workflow:

1. Invocation: Your media server calls the ffmpeg-over-ip-client using standard ffmpeg arguments.
2. Connection: The client establishes a TCP connection to the server and sends the command using HMAC authentication.
3. Execution: The server launches a patched ffmpeg process. This patched version tunnels all file I/O (reads and writes) back through the TCP connection to the client's local filesystem.
4. Streaming: stdout and stderr are forwarded in real-time. When the server-side ffmpeg exits, the client exits with the same exit code.

Key Benefits:

• No GPU Passthrough: The server handles the hardware acceleration (NVENC, QSV, VAAPI, AMF, VideoToolbox, etc.).
• No Shared Filesystems: Files stay on the client; the server only receives the I/O stream.
• Concurrency: Multiple clients can connect to a single server simultaneously, with each session spawning its own ffmpeg process.

You can configure the client or server entirely via environment variables without a config file, provided you set both the ADDRESS and AUTH_SECRET variables. Note that rewrites (server) and fallbackRewrites (client) cannot be configured via environment variables; they require a JSONC config file.

Client Environment Variables

Server Environment Variables

Docker Example

docker run \
  -e FFMPEG_OVER_IP_CLIENT_ADDRESS=192.168.1.100:5050 \
  -e FFMPEG_OVER_IP_CLIENT_AUTH_SECRET=my-secret \
  -v ./ffmpeg-over-ip-client:/usr/bin/ffmpeg \
  your-image