DOCSEARCHbyAlgolia
Submit Docs

DVCLive

repository·main·Indexed 42 days ago

https://github.com/treeverse/dvclive

A Python library for logging machine learning metrics and metadata in simple file formats. Fully compatible with DVC, it allows for experiment tracking and comparison without external servers by storing data as plain text files versioned by Git or DVC. It includes integrations for PyTorch Lightning, Lightning Fabric, and Hugging Face via the DVCLiveCallback and DVCLiveLogger.

Tokens
14K
Snippets
58
Records
70
Agent score
51%

What's inside DVCLive

  1. What is DVCLive and how does it differ from MLFlow or W&B?

    main

    DVCLive is an ML Logger that stores metrics, parameters, and plots as plain text files.

    Unlike MLFlow, Weights & Biases, or Neptune, DVCLive:

    • Does not require additional services or servers to run.
    • Uses files that can be versioned by Git or tracked via DVC.
    • Can save experiments as hidden Git commits.

  2. Compare experiments in VS Code or DVC Studio

    main

    DVCLive outputs can be visualized through several interfaces:

    1. DVC Extension for VS Code: Use the Experiments and Plots views to compare results. Live updates are displayed while experiments are running.
    2. DVC Studio: Push results to DVC Studio to compare experiments against your entire repository history. You can enable Studio Live Experiments to see live updates during execution.

  4. Log Scikit-learn plots with DVCLive

    main

    DVCLive provides specialized plot types for Scikit-learn metrics. These plot classes expect a specific input format: a tuple containing (y_true, y_score) or (y_true, y_pred).

    When using these classes, the dump method internally calls the corresponding sklearn.metrics or sklearn.calibration function, passing any additional **kwargs provided to the dump call directly to the Scikit-learn function.

  5. Parameter sanitization in DVCLiveLogger

    main

    When using DVCLiveLogger.log_hyperparams, the logger performs automatic sanitization to prevent errors when logging complex objects.

    Specifically:

    1. Multi-dimensional arrays: Any object with an ndim attribute where ndim > 1 is converted to its string representation (str(v)).
    2. argparse.Namespace: Any argparse.Namespace objects are converted to strings.

    This ensures that only supported scalar-like values are passed to the underlying Live experiment.

  6. Visualize and inspect DVCLive experiment results

    main

    Once experiments are logged, you can explore them using the following methods:

    1. CLI Plots: Use dvc plots show to visualize logged metrics.
    2. CLI Experiment Table: Use dvc exp show to see a summary table of all experiments.
    3. Python API: Use dvc.api.exp_show() to retrieve experiment data as a pandas DataFrame for programmatic analysis.
    4. Notebook HTML: If using report="notebook", you can display the generated plots directly in a notebook using IPython.display.HTML.
    # Show plots via CLI
dvc plots show

# Show experiment table via CLI
dvc exp show
    import dvc.api
import pandas as pd

# View experiments as a DataFrame
pd.DataFrame(dvc.api.exp_show())

  10. Integrate DVCLive with a Lightning Fabric training loop

    main

    When combining DVCLive with Lightning Fabric, follow this pattern to ensure distributed compatibility:

    1. Initialize Logger: Create DVCLiveLogger.
    2. Setup Fabric: Initialize Fabric().
    3. Data Loading: Use fabric.rank_zero_first to ensure only the primary rank downloads data, and use fabric.setup_dataloaders to prepare loaders for distributed training.
    4. Model/Optimizer Setup: Use fabric.setup(model, optimizer) to automatically move components to the correct device.
    5. Training: Use fabric.backward(loss) instead of loss.backward().
    6. Metrics: Use fabric.all_gather to aggregate metrics across processes before logging.
    7. Saving: Use fabric.save for checkpoints and logger.experiment.log_artifact for final model files (ensuring rank 0 only).
    from lightning.fabric import Fabric
from dvclive.fabric import DVCLiveLogger

logger = DVCLiveLogger(report="notebook")
fabric = Fabric()

# Setup data
with fabric.rank_zero_first(local=False):
    train_dataset = MNIST(DATASETS_PATH, download=fabric.is_global_zero, train=True)

train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=64)
train_loader, test_loader = fabric.setup_dataloaders(train_loader, test_loader)

# Setup model
model = Net()
optimizer = torch.optim.Adadelta(model.parameters(), lr=1.0)
model, optimizer = fabric.setup(model, optimizer)

# Training loop
for batch in train_loader:
    optimizer.zero_grad()
    output = model(batch[0])
    loss = F.nll_loss(output, batch[1])
    fabric.backward(loss)
    optimizer.step()
    logger.log_metrics({"loss": loss.item()})

# Finalize
logger.finalize("success")

  11. Initialize a DVC Repository for Experiment Tracking

    main

    Before running experiments, initialize a Git repository and a DVC repository to ensure DVCLive can track and version your experiments effectively.

    !git init -q
!git config --local user.email "you@example.com"
!git config --local user.name "Your Name"
!dvc init -q
!git commit -m "DVC init"

  12. Initialize a DVC Repository

    main

    DVCLive works best when integrated with a DVC (Data Version Control) repository. To set up a local environment for experiment tracking, initialize a git repository and then initialize DVC.

    !git init -q
!git config --local user.email "you@example.com"
!git config --local user.name "Your Name"
!dvc init -q
!git commit -m "DVC init"