DOCSEARCHbyAlgolia
Submit Docs

Dud

repository·main·Indexed 42 days ago

https://github.com/kevin-hanselman/dud

A lightweight, high-performance tool for versioning large binary data alongside source code and building data pipelines using YAML-based stages. It provides a minimalist alternative to DVC, focusing on data versioning and reproduction without experiment tracking. Dud utilizes symbolic link checkouts for speed and delegates remote cache management to Rclone, supporting over 28,000 cloud storage providers.

Tokens
13K
Snippets
64
Records
80
Agent score
49%

What's inside Dud

  1. Overview of Dud

    main

    Dud is a lightweight tool designed for versioning large binary data alongside source code and building data pipelines. It extends source control benefits to large files by using small YAML files called stages (or recipes) to describe how data is retrieved or generated.

    Key capabilities include:

    • Versioning: Commit, checkout, fetch, and push large files and directories.
    • Pipelines: Chain stages together to create data pipelines, similar to how Make works.
    • Integration: Link data to code by storing stage files in your source control system.

  2. Compare Dud performance with DVC benchmarks

    main

    Dud provides performance benchmarks comparing its execution speed against DVC across different data profiles. The benchmarks evaluate core commands: checkout, commit, fetch, push, and status.

    Performance Profiles

    1. Few large files: Tests performance using a dataset of four 1 GB files in a single directory. Dud typically shows significant advantages in checkout, commit, and status operations.
    2. Many small files: Tests performance using a dataset of twenty thousand 100 KB files in a single directory. Dud demonstrates substantial speed improvements over DVC in commit, fetch, and push operations.

  3. How Dud differs from DVC

    main

    Dud is designed as a minimalist, high-performance alternative to DVC, following the UNIX philosophy of composing small, focused tools.

    Key Architectural Differences:

    • Scope: Unlike DVC, Dud does not manage experiments or metrics. It focuses strictly on data versioning and reproduction. You can pair Dud with tools like MLflow or Aim for experiment tracking.
    • Explicit Commits: Commits to the cache are never side effects. They only occur when you explicitly run dud commit. This prevents accidental cache bloat during rapid iteration.
    • Symlink Checkouts: By default, dud checkout uses symbolic links. This makes checkouts nearly instantaneous, prevents data duplication between the workspace and cache, and makes dud status extremely fast.
    • Non-destructive Pipelines: Running a pipeline in Dud never implicitly modifies or removes existing workspace files. The user has full control over whether to clear outputs or reuse them by coding that logic into the stage's command.
    • Remote Storage via Rclone: Dud delegates all remote cache management (dud fetch, dud push) to Rclone. This provides access to over 28,000 cloud storage providers (S3, GCS, Dropbox, etc.) and leverages Rclone's robust CLI for configuration.

  4. Understand Dud core concepts: Artifacts, Stages, Index, and Cache

    main

    Dud operates using four primary abstractions to manage data pipelines:

    • Artifact: The fundamental unit of tracking. An Artifact is any file or directory tracked by Dud. While typically stored in the Cache, they can exist elsewhere.
    • Stage: A logical grouping of Artifacts or a specific operation that consumes and/or produces them. Stages are defined by the user in YAML files, which should be committed to source control. For details on the YAML schema, run dud stage --help.
    • Index: The project-wide collection of all Stages. Stored as a plain text file at .dud/index, the Index represents a dependency graph of Stages, allowing you to construct complex data pipelines.
    • Cache: A local, content-addressed directory used to store and version the contents of Artifacts. Because it is content-addressed, it manages multiple versions of Artifacts without duplication or conflicts.

  5. Install Dud from a release

    main

    To install Dud using a pre-compiled release:

    1. Go to the Dud GitHub releases page.
    2. Download the tarball that matches your operating system and CPU architecture.
    3. Extract the contents and move the dud executable to a directory in your $PATH.

    You can use the following command to download a release and install it directly to your Go binary path (~/go/bin):

    curl -fL 'https://github.com/kevin-hanselman/dud/releases/COPY_URL_FROM_GITHUB.tar.gz' \
    | tar -C ~/go/bin -zxvf - dud

  6. Create a data pipeline stage with inputs and commands

    main

    Dud stages can define a command that generates outputs based on specific inputs. This allows for reproducible data pipelines. When generating a pipeline stage, use:

    • -i <input>: Declares an input file/directory that the stage depends on.
    • -o <output>: Declares the output file/directory.
    • -- <command>: The shell command used to generate the output. The -- separator tells Dud to treat everything following it as the command.

    If an input file changes, Dud will know the stage needs to be re-run.

    # Example: Extracting a tarball into a directory
$ dud stage gen \
    -i input.tar.gz \
    -o output_dir/ \
    -- tar -xvf input.tar.gz \
    | tee extract_stage.yaml

  7. Share data using Dud and Rclone

    main

    Dud uses rclone to synchronize the local Dud cache (.dud/cache) with a remote storage provider. To share data, you must configure a remote in a project-specific rclone configuration file and then point Dud to that remote.

    1. Configure Rclone

    When you run dud init, Dud creates .dud/rclone.conf. You can use the rclone config command with the --config flag to set up a remote interactively, or manually edit the file.

    Example of a local filesystem remote in .dud/rclone.conf:

    [fake_remote]
type = local

    2. Configure Dud Remote

    Use the dud config set command to tell Dud which rclone remote to use. The value must follow the rclone remote path syntax (e.g., remote_name:path/to/cache).

    dud config set remote fake_remote:/tmp/dud/cache

    3. Push and Fetch Data

    • dud push: Iterates through all stages, looks up committed artifacts by checksum, and uses rclone to upload them to the remote cache.
    • dud fetch: The inverse of push; it looks up artifacts from stage files and copies them from the remote cache to your local .dud/cache.
    # Set the remote
dud config set remote fake_remote:/tmp/dud/cache

# Upload data to remote
dud push

# Download data from remote
dud fetch

  8. Register and commit stages to the Dud cache

    main

    After generating a stage file, you must register it with Dud's index and then commit the files to the cache to enable versioning and content-addressed storage.

    1. Add the stage: Use dud stage add <stage_file.yaml> to add the definition to the project index.
    2. Commit the stage: Use dud commit to copy the files/directories defined in the stage to the Dud cache. Once committed, files in your working directory are replaced by read-only links to the cache to prevent accidental modification.
    # 1. Add the stage to the index
$ dud stage add my-data.yaml

# 2. Commit the files to the cache
$ dud commit

  10. Share data with Dud and Rclone

    main

    To share data using Dud and Rclone, you must configure a remote Dud cache.

    1. Initialize the rclone configuration within your Dud project directory by running the rclone config command pointing to the Dud-managed config file: rclone --config .dud/rclone.conf config.
    2. Once the remote is set up, update your Dud configuration file at .dud/config.yaml by setting the remote key to the valid rclone remote path (e.g., myremote:path/to/data).

    Refer to the rclone documentation for the correct syntax of remote paths.

    rclone --config .dud/rclone.conf config

  11. System requirements for Dud

    main

    Dud is designed for UNIX-based operating systems. While it is explicitly tested on Linux with a 64-bit x86 CPU, it is expected to work on:

    • macOS
    • Windows Subsystem for Linux (WSL)
    • 32-bit x86 architectures
    • ARM architectures

    Dependency: Dud requires rclone to interact with remote storage. You must have rclone installed to use the push and fetch commands.

  12. Version Dud pipeline code with Git

    main

    To version your data pipeline, use a Source Control Management (SCM) tool like Git. You should version your Dud stage files (e.g., cifar.yaml, extract_cifar.yaml) and configuration files, but do not version the large binary data files or the Dud cache.

    Best Practices

    1. Ignore Data/Cache: Use a .gitignore file to exclude large binary files (e.g., *.tar.*, *.png) and local data directories.
    2. Dud Auto-Ignore: dud init automatically creates a .dud/.gitignore file to prevent the .dud/cache directory from being accidentally committed to Git.
    3. Track Stage Files: Ensure your .yaml stage definitions are tracked so collaborators can reconstruct the pipeline.
    # Example .gitignore
/cifar-10-batches-py
*.tar.*
*.png