DVC Documentation

Data Version Control (DVC) is an open-source tool designed for data management, ML pipeline automation, and experiment management. It enables data science and machine learning teams to manage large datasets, ensure project reproducibility, and improve collaboration by leveraging existing software engineering tools like Git, IDEs, CI/CD, and cloud storage.

Core Principles

• Codification: ML project aspects (data/model versions, pipelines, experiments) are defined in human-readable metafiles.
• Versioning: Uses Git (or other SCMs) to version the entire project. DVC metafiles act as placeholders in Git that point to the actual data stored in a cache.
• Secure Collaboration: Allows controlled access to project components and sharing across teams.

DVC remotes are distributed storage locations (like Amazon S3, Google Drive, SSH, HTTP, or local file systems) used to share or back up your datasets and ML models. The dvc remote command suite allows you to configure and manage these storage locations by reading or modifying your DVC configuration files (typically .dvc/config).

dvc remote {add,default,remove,modify,list,rename}

The dvc plots command suite allows you to visualize and compare data series or images from ML projects. It supports JSON, YAML 1.2, CSV, and TSV data files, as well as JPEG, GIF, PNG, and SVG images. These files are typically artifacts from an ML pipeline or performance logs produced by [DVCLive].

You can use the following subcommands to manage your visualizations:

• show: Display plots.
• diff: Compare plots between different stages or experiments.
• templates: Work with plot templates.

For detailed configuration, refer to the [Visualizing Plots] guide and the [plots schema] in your dvc.yaml files.

dvc plots {show,diff,templates}

DVCLive provides integrations for a variety of Machine Learning frameworks to track experiments. The currently supported frameworks include:

• Fast.ai
• Hugging Face
• Keras
• LightGBM
• MMCV
• Optuna
• PyTorch
• PyTorch Lightning
• Scikit-learn
• TensorFlow
• XGBoost
• Ultralytics YOLO v8

If you need support for a framework not listed here, you can propose an integration by checking existing issues or creating a new issue on the DVCLive GitHub repository under the integrations label.

DVC (Data Version Control) is designed for data science workflows where you need to manage data files, datasets, and machine learning models alongside your code. Use DVC if you need to:

• Track and save data/models: Capture data and ML models using the same versioning logic used for code.
• Version data and models: Easily create and switch between different versions of your datasets and ML artifacts.
• Understand lineage: Maintain visibility into how specific datasets and ML artifacts were originally constructed.
• Compare experiments: Evaluate and compare model metrics across different experimental runs.
• Apply engineering best practices: Bring software engineering rigor (version control, reproducibility) to data science projects.

Instead of tracking entire files as dependencies, you can track specific hyperparameters or configuration values using the params field in dvc.yaml. This allows for more granular invalidation: DVC only reruns a stage if the specific key/value pair defined in params changes, even if other parts of the parameters file remain the same.

By default, DVC tracks keys in a params.yaml file.

To compare parameters across different project versions, use dvc params diff.

stages:
  train:
    cmd: ...
    deps: ...
    params: # from params.yaml
      - learning_rate
      - nn.epochs
      - nn.batch_size
    outs: ...

A DVC pipeline is a sequence of repeatable actions used to get or update project results, such as data science workflows. Instead of manual steps, you define these stages in a standard YAML format using .dvc and dvc.yaml files. This makes your workflow more manageable and ensures it is consistent and reproducible.

A typical pipeline might include:

1. Gathering data for training and validation.
2. Extracting features from a dataset.
3. (Re)training an ML model.
4. Evaluating results against a validation set.

Use the --no-scm flag to initialize DVC in a directory that is not part of a Git repository or to force DVC to ignore an existing Git repository. This is useful for simple projects, using other version control systems, or deployment automation (e.g., cron jobs).

Limitations:

• No Versioning Features: You cannot use features that require Git history, such as dvc diff, dvc metrics diff, or the automatic creation/updating of .gitignore files during dvc add or dvc stage add.
• Configuration: DVC sets the core.no_scm config option to true. Even if Git is initialized in the directory later, DVC will remain detached from Git.

For machine learning workflows, you can track hyperparameters using the -p or --params option. This allows you to treat specific key/value pairs in a structured file as dependencies.

• Default file: ./params.yaml
• Usage: Provide a comma-separated list of parameters: -p learning_rate,epochs.
• Custom files: Use a prefix with a colon: -p myparams.toml:threshold or -p myparams.toml: to include all parameters in that file.

# Example using a custom params file
dvc stage add -n train -p params.json:learning_rate,batch_size ./train.py

DVC is often compared to systems like Airflow or Luigi that manage data pipelines and dependency graphs.

• Focus: DVC is lightweight and optimized for data science and modeling. It is easy to create and modify pipelines compared to heavy-duty workflow engines.
• Execution: DVC is a command-line tool that does not run daemons or servers. It lacks advanced execution features like continuous monitoring, complex error handling, or automatic recovery.
• Visualization: While it lacks a GUI, DVC can generate images to visualize pipeline and experiment workflows.
• Complementary Tools: For advanced orchestration and CI/CD integration, DVC is often used alongside CML (Continuous Machine Learning).