DOCSEARCHbyAlgolia
Submit Docs

DVC (Data Version Control)

website·Indexed 40 days ago

https://doc.dvc.org/

DVC is a data science and machine learning tool for data versioning and pipeline management. It provides a CLI and a Python API (including dvc.api and DVCFileSystem) to track data files, directories, metrics, and parameters, and integrates with Git to manage versions of data and models stored in remote storage.

Tokens
103.5K
Snippets
604
Records
767
Agent score
68%

What's inside DVC

  1. Overview of DVC remote storage management

    DVC remotes are distributed storage locations used to share or back up cached assets (datasets and ML models), similar to how Git remotes work for code. Supported storage types include Amazon S3, Google Drive, SSH, HTTP, and local file systems. These commands modify the DVC configuration files (typically .dvc/config).

  2. Overview of Machine Learning Experiment Tracking with DVC

    DVC (Data Version Control) provides a local-first experiment management system that integrates directly with Git to ensure reproducibility. Unlike tools that rely on separate servers or custom logging APIs, DVC captures changesets (input data, source code, hyperparameters, artifacts, and metrics) automatically by working on top of Git commits, branches, and tags. This approach allows developers to use standard Git workflows, such as pull requests and distributed collaboration, to manage ML experiments. DVC is language-agnostic and supports any environment, including Jupyter notebooks, Scala, CSVs, or HDFS.

  3. Overview of DVC metrics commands

    DVC provides a set of commands to track, display, and compare machine learning experiment performance metrics (such as AUC, ROC, or false positives). Metrics are stored as structured key/value pairs in hierarchical files. The primary subcommands are show to display current values and diff to report numeric differences between experiments (e.g., comparing the current workspace against the HEAD commit).
    dvc metrics {show,diff}

  4. Overview of DVC Pipelines

    DVC (Data Version Control) allows you to define a sequence of repeatable actions—such as data gathering, feature extraction, model training, and evaluation—as a pipeline. These pipelines are defined in a standard YAML format using .dvc and dvc.yaml files, ensuring that data science workflows are manageable and consistently reproducible.

  5. Overview of DVC Remote Storage

    DVC remotes provide access to external storage locations used to track and share data artifacts and ML models. They function similarly to Git remotes but for cached data instead of code. Remote storage is primarily used to synchronize large files, centralize data for collaboration, and back up different versions of datasets to save local disk space.

  6. Overview of Data Version Control (DVC)

    Data Version Control (DVC) is a free, open-source tool designed for data management, ML pipeline automation, and experiment management. It enables data science and ML teams to manage large datasets, ensure project reproducibility, and improve collaboration by leveraging existing software engineering tools like Git, IDEs, and CI/CD pipelines.

    DVC operates on three core principles:

    1. Codification: ML project aspects (data/model versions, pipelines, experiments) are defined in human-readable metafiles.
    2. Versioning: Git (or any SCM) is used to version the project by committing DVC metafiles, which act as placeholders for the actual data assets.
    3. Secure Collaboration: Access to project components can be controlled and shared selectively.

  7. Overview of DVC Data Pipelines

    DVC (Data Version Control) allows you to define data pipelines to describe how models and other data artifacts are built. This provides an efficient, reproducible way to manage ML projects, functioning similarly to 'Makefiles' but specifically designed for data and machine learning workflows.

  8. Overview of DVC plot visualization

    DVC can generate and render plots based on project data as static HTML webpages, which can be viewed in a web browser or via the DVC VS Code Extension Plots Dashboard. The typical workflow involves saving plot data, defining the visualization in dvc.yaml, running experiments, and comparing the resulting plots.

  9. Implement a shared data caching hub with DVC

    DVC's built-in caching allows teams to implement a global storage layer (via a shared DVC cache) on a network drive or external storage. This architecture de-duplicates files across datasets and prevents repetitive downloads from cloud object stores. By linking working files to the central cache instead of copying them, teams can switch data inputs quickly and ensure data security policies are maintained as data remains in central storage.

  10. Prerequisites for using DVC

    DVC does not replace or include Git. You must have git installed in your system to enable core features such as data versioning and quick experimentation.

  11. Supported ML Frameworks for DVCLive

    DVCLive provides built-in support for integrating with a variety of machine learning frameworks to track metrics and experiments. Supported frameworks include:

    • Fast.ai
    • Hugging Face
    • Keras
    • LightGBM
    • MMCV
    • Optuna
    • PyTorch
    • PyTorch Lightning
    • Scikit-learn
    • TensorFlow
    • XGBoost
    • Ultralytics YOLO v8

  12. DVC macOS installation compatibility matrix

    Depending on your macOS hardware architecture, only certain installation methods are supported:

    • ARM (Apple Silicon): Use Homebrew or pip.
    • x86 (Intel): Use Homebrew, pip, conda, or the PKG binary.