DOCSEARCHbyAlgolia
Submit Docs

Video Duplicate Finder

repository·master·Indexed 25 days ago

https://github.com/0x90d/videoduplicatefinder

A cross-platform tool for finding duplicate video and image files based on visual and audio similarity. It features a Desktop GUI, a CLI for automation, and a Web UI deployable via Docker. Key capabilities include partial clip detection using audio fingerprinting and AI-powered matching via DINOv2 (ONNX Runtime) to identify transformed copies, such as cropped, mirrored, or zoomed videos. Requires FFmpeg and FFprobe for operation.

Tokens
4.8K
Snippets
7
Records
42
Agent score
86%

What's inside Video Duplicate Finder

  1. Install and Setup Desktop GUI

    master

    Video Duplicate Finder requires FFmpeg and FFprobe. On first launch, VDF attempts to download them automatically.

    Windows

    1. Download the latest FFmpeg GPL shared package from https://ffmpeg.org/download.html.
    2. Extract ffmpeg.exe and ffprobe.exe into the same folder as VDF.GUI.exe, a subfolder named bin, or ensure they are on your PATH.

    Linux

    1. Install FFmpeg:
    sudo apt-get update && sudo apt-get install ffmpeg
    1. Run the application:
    chmod +x VDF.GUI
./VDF.GUI

    Optional: Add to application menu

    sed -i "s|/opt/videoduplicatefinder|$(pwd)|g" videoduplicatefinder.desktop

mkdir -p ~/.local/share/applications
cp videoduplicatefinder.desktop ~/.local/share/applications/

    macOS

    1. Install FFmpeg via Homebrew:
    brew install ffmpeg
    1. Launch the .app bundle. If blocked by security, right-click and choose Open.
    2. If blocked by system policy (macOS 14+), clear quarantine and re-sign:
    xattr -cr "Video Duplicate Finder.app"
codesign --force --deep --sign - "Video Duplicate Finder.app"

  2. Enable AI Matching

    master

    AI Matching uses neural image embeddings (DINOv2 via ONNX Runtime) to find transformed copies (cropped, mirrored, zoomed, color-graded) and visual partial clips (works on silent/muted videos). This pass is additive and does not hide classic results.

    To enable in the GUI, configure the following:

    SettingWhereDefaultDescription
    AI matching (additional pass)Settings → MatchingoffEnables embedding comparison on top of classic mode.
    AI similarity threshold (%)Settings → Matching94Similarity required for embeddings. Lower to ~92 for more aggressive matching.
    Detect partial duplicates visually (AI)Settings → Partial Clip DetectionoffDense keyframe matching for trimmed clips (no audio needed).
    AI frame hit threshold (%)Settings → Partial Clip Detection89Per-keyframe similarity needed; requires at least 4 hits to pair.

  3. Enable Partial Clip Detection

    master

    Partial clip detection finds when a shorter video is a partial clip of a longer one using audio fingerprinting. It runs as an optional second phase after the normal visual scan. Matched pairs show a Clip Offset column indicating the start position in the source. Note: This requires audio tracks in both files.

    To enable in the GUI, go to Settings → Partial Clip Detection and configure the following:

    SettingDefaultDescription
    Min clip / source ratio (%)10Minimum clip duration as a percentage of the source duration.
    Min audio similarity (%)80Minimum average Hamming similarity for the fingerprint match.
    Require visual confirmationonRejects audio matches if frames at the offset don't look similar.
    Min visual similarity (%)85Minimum frame similarity for the visual confirmation step.

  4. Deploy Web UI using Docker Compose

    master

    For permanent installations, use docker-compose.yml.

    1. Download the docker-compose.yml from the repository.
    2. Edit the file to add your media volume mounts and an optional VDF_WEB_PASSWORD.
    3. Run docker compose up -d to start the service.
    4. Access via http://localhost:8080.
    5. To update, run docker compose pull && docker compose up -d.
    # Example docker-compose.yml snippet
environment:
  - VDF_WEB_PASSWORD=mysecretpassword
volumes:
  - /mnt/nas/movies:/mnt/nas/movies:ro
  - /mnt/nas/series:/mnt/nas/series:ro

  5. Run the Web UI from a self-contained archive

    master

    The Web UI runs as a local web server for headless machines, NAS devices, or remote management.

    Setup Steps:

    1. Download Web-<platform> from the releases page and extract it.
    2. Linux/macOS: Run chmod +x VDF.Web followed by ./VDF.Web.
    3. Windows: Run VDF.Web.exe.
    4. Open http://localhost:5000 in your browser.
    5. Enter the password printed to your console during launch.

    Requirements:

    • FFmpeg and FFprobe must be installed and available on your PATH. If running outside Docker, the application will attempt to download them automatically on first launch.
    # Linux/macOS
chmod +x VDF.Web
./VDF.Web

# Windows
VDF.Web.exe

  6. Use the Video Duplicate Finder CLI

    master

    The CLI is designed for scripting and automation. It requires FFmpeg and FFprobe to be on your PATH or in the same directory as the vdf-cli binary.

    Installation (Linux/macOS)

    chmod +x vdf-cli

    Basic Usage

    Scan and compare a directory:

    vdf-cli scan-and-compare --include /path/to/media

    Scan multiple directories and save to JSON:

    vdf-cli scan-and-compare \
  --include /mnt/movies \
  --include /mnt/series \
  --exclude /mnt/movies/extras \
  --format json \
  --output results.json

  7. Run the Web UI via Docker

    master

    Docker is the recommended method for NAS or Linux servers as it includes FFmpeg automatically.

    Quick Start: Run the following command to start the container. Note that you must mount your media directory to a path (e.g., /media) and then add that path inside the Web UI settings.

    Note on Persistence: To ensure settings and scan databases persist across container updates, you must mount volumes to /root/.config/VDF and /root/.local/state/VDF.

    # Basic run with media mount
docker run -d \
  --name vdf-web \
  -p 8080:8080 \
  -v vdf-db:/root/.config/VDF \
  -v vdf-state:/root/.local/state/VDF \
  -v /path/to/your/media:/media:ro \
  ghcr.io/0x90d/vdf-web:latest

# Run with a custom password
docker run -d \
  --name vdf-web \
  -p 8080:8080 \
  -e VDF_WEB_PASSWORD=mysecretpassword \
  -v vdf-db:/root/.config/VDF \
  -v vdf-state:/root/.local/state/VDF \
  -v /path/to/your/media:/media:ro \
  ghcr.io/0x90d/vdf-web:latest

  8. Configure Web UI authentication and port

    master

    You can control the Web UI behavior using environment variables:

    • VDF_WEB_PASSWORD: Set a custom password (replaces the auto-generated one).
    • VDF_WEB_AUTH: Set to false to disable authentication entirely.
    • ASPNETCORE_URLS: Change the listening port (e.g., ASPNETCORE_URLS=http://+:8080 ./VDF.Web).

  9. Mount media directories in VDF Web Docker container

    master

    To allow the VDF Web service to scan your media, mount your host directories into the container using the host_path:container_path:ro format. The :ro flag is recommended to ensure the container has read-only access to your media.

    Example:

        volumes:
      - /mnt/nas/movies:/mnt/nas/movies:ro
      - "/mnt/path with spaces:/mnt/media:ro"

  10. Configure VDF Web via Docker Compose

    master

    The vdf-web service can be deployed using Docker. It exposes the web interface on port 8080. You can configure authentication, proxy trust, and media volume mounts using environment variables and volumes.

    services:
  vdf-web:
    image: ghcr.io/0x90d/vdf-web:latest
    ports:
      - "8080:8080"
    volumes:
      - vdf-db:/root/.config/VDF
      - vdf-state:/root/.local/state/VDF

volumes:
  vdf-db:
    driver: local
  vdf-state:
    driver: local

  11. Reference: Web UI File Storage Paths

    master

    The Web UI saves settings and scan databases in the following locations depending on the OS:

    Web Settings and Credentials:

    • Windows: %APPDATA%\VDF%
    • Linux: ~/.config/VDF/ (or $XDG_CONFIG_HOME/VDF/)
    • macOS: ~/Library/Preferences/VDF/

    Scan Database (ScannedFiles.db): Stored next to the executable if writable, otherwise:

    • Windows: %LOCALAPPDATA%\VDF%
    • Linux: ~/.local/state/VDF/ (or $XDG_STATE_HOME/VDF/)
    • macOS: ~/Library/Application Support/VDF/

  12. Reference: Web UI Docker Volumes

    master

    When using Docker, use these volume mappings to manage data persistence:

    | Volume | Purpose |
|--------|---------|
| `/root/.config/VDF` | Settings (`web-settings.json`) and login credentials — mount a named volume here so configuration persists across container updates |
| `/root/.local/state/VDF` | Scan database (`ScannedFiles.db`) — mount a named volume here so hashed-file data persists across container updates |
| Your media paths | Mount each media directory you want to scan. Read-only (`:ro`) is recommended. |