Continuous Machine Learning (CML)

To get started with CML on GitHub, follow these steps:

1. Clone a fork: Clone your fork of an example repository to your local machine.
2. Create a workflow file: Create .github/workflows/cml.yaml in your repository.
3. Configure the workflow: Use actions/setup-python@v4 and iterative/setup-cml@v1 to prepare the environment. Ensure you pass the REPO_TOKEN environment variable using ${{ secrets.GITHUB_TOKEN }}.
4. Generate and post a report: In your workflow's run step, train your model, append metrics and images to a markdown file (e.g., report.md), and call cml comment create report.md.

Example workflow structure:

name: model-training
on: [push]
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
      - uses: iterative/setup-cml@v1
      - name: Train model
        env:
          REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          pip install -r requirements.txt
          python train.py

          cat metrics.txt >> report.md
          echo "![](./plot.png)" >> report.md
          cml comment create report.md

CML integrates with DVC to handle large datasets and visualize metric/plot differences between Git commits.

To use them together:

1. Use a CML Docker image: Use container: ghcr.io/iterative/cml:0-dvc2-base1 in your GitHub Action to have NodeJS, Python 3, DVC, and CML pre-installed.
2. Pull data: Use dvc pull data --run-cache to retrieve datasets from remote storage.
3. Compare metrics: Use dvc metrics diff main --show-md >> report.md to append a markdown comparison of metrics against the main branch to your report.
4. Compare plots: Use dvc plots diff to generate visualizations (like Vega JSON) and convert them to images (e.g., using vl2png) to include in your report.

Example workflow snippet:

jobs:
  run:
    runs-on: ubuntu-latest
    container: ghcr.io/iterative/cml:0-dvc2-base1
    steps:
      - uses: actions/checkout@v3
      - name: Train model
        env:
          REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: |
          pip install -r requirements.txt
          dvc pull data --run-cache
          dvc repro

          echo "## Metrics" >> report.md
          git fetch --prune
          dvc metrics diff main --show-md >> report.md

          cml comment create report.md

name: model-training
on: [push]
jobs:
  run:
    runs-on: ubuntu-latest
    container: ghcr.io/iterative/cml:0-dvc2-base1
    steps:
      - uses: actions/checkout@v3
      - name: Train model
        env:
          REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: |
          # Install requirements
          pip install -r requirements.txt

          # Pull data & run-cache from S3 and reproduce pipeline
          dvc pull data --run-cache
          dvc repro

          # Report metrics
          echo "## Metrics" >> report.md
          git fetch --prune
          dvc metrics diff main --show-md >> report.md

          # ... (plot diff logic) ...

          cml comment create report.md

To use CML with GitHub, you must create a workflow file at .github/workflows/cml.yaml. The workflow typically involves checking out the code, setting up the CML environment using iterative/setup-cml@v1, running your training script, and then using the cml comment create command to post results as a comment in the Pull Request.

For self-hosted runners, you may need to manually set up Node.js and Python. You can also optionally use a pre-configured container image that includes Ubuntu LTS, DVC, and CML.

name: your-workflow-name
on: [push]
jobs:
  run:
    runs-on: ubuntu-latest
    # optionally use a convenient Ubuntu LTS + DVC + CML image
    # container: ghcr.io/iterative/cml:0-dvc2-base1
    steps:
      - uses: actions/checkout@v3
      # may need to setup NodeJS & Python3 on e.g. self-hosted
      # - uses: actions/setup-node@v3
      #   with:
      #     node-version: '16'
      # - uses: actions/setup-python@v4
      #   with:
      #     python-version: '3.x'
      - uses: iterative/setup-cml@v1
      - name: Train model
        run: |
          # Your ML workflow goes here
          pip install -r requirements.txt
          python train.py
      - name: Write CML report
        env:
          REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          # Post reports as comments in GitHub PRs
          cat results.txt >> report.md
          cml comment create report.md

CML can automatically allocate cloud instances (AWS, Azure, GCP, or Kubernetes) to run computationally intensive jobs, such as those requiring GPUs. When a job finishes, the instance is automatically shut down. cml runner also provides automatic job restarts in case of GitHub Actions timeouts or cloud spot instance interruptions.

To use cloud allocation, you must provide cloud service credentials (e.g., AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) as environment variables in your workflow and a REPO_TOKEN (Personal Access Token) with repository read/write access.

name: Train-in-the-cloud
on: [push]
jobs:
  deploy-runner:
    runs-on: ubuntu-latest
    steps:
      - uses: iterative/setup-cml@v1
      - uses: actions/checkout@v3
      - name: Deploy runner on EC2
        env:
          REPO_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: |
          cml runner launch \
            --cloud=aws \
            --cloud-region=us-west \
            --cloud-type=g4dn.xlarge \
            --labels=cml-gpu
  train-model:
    needs: deploy-runner
    runs-on: [self-hosted, cml-gpu]
    # ... rest of job

CML (Continuous Machine Learning) is designed to run in CI/CD environments like GitHub Actions, GitLab CI, and Bitbucket Pipelines to automate machine learning workflows, such as training models and reporting results.

Supported Platforms

• GitHub Actions
• GitLab CI
• Bitbucket Pipelines

Environment Setup

To use CML, you typically need a runtime environment with both Python 3 and Node.js installed. If you are using self-hosted runners, you may need to install these manually.

Recommended Container Image

For a convenient setup on Ubuntu LTS, you can use the official CML Docker image which comes pre-configured with DVC and CML: ghcr.io/iterative/cml:0-dvc2-base1

Example: GitHub Actions Setup

When using GitHub Actions, you can use standard setup actions to configure your environment:

- uses: actions/setup-node@v3
  with:
    node-version: '16'

- uses: actions/setup-python@v4

# Example GitHub Actions configuration
- uses: actions/setup-node@v3
  with:
    node-version: '16'

- uses: actions/setup-python@v4