DOCSEARCHbyAlgolia
Submit Docs

CML Documentation

website·Indexed 34 days ago

https://cml.dev/doc

Documentation for CML, including guides for getting started on GitHub, GitLab, and Bitbucket. It covers usage with DVC, configuring self-hosted runners (on-premise or cloud), and a command reference for tools such as publish, runner, workflow, comment, pr, ci, tensorboard, and check.

Tokens
15.1K
Snippets
80
Records
108
Agent score
70%

What's inside CML

  1. Overview of CML (Continuous Machine Learning)

    CML is a tool designed to automate the training and evaluation of machine learning models directly within pull requests or merge requests. It enables the embedding of reports containing metrics and plots based on the results of these automated runs, facilitating a continuous integration (CI) workflow for ML projects.

  2. Overview of Continuous Machine Learning (CML)

    Continuous Machine Learning (CML) is an open-source tool designed to implement continuous integration and delivery (CI/CD) specifically for machine learning projects. It allows developers to automate development workflows, including model training and evaluation, monitoring changing datasets, and comparing ML experiments across project history.

  3. CML Command Reference Overview

    CML provides several CLI commands to manage cloud compute and reporting:

    • cml runner: Launch a runner on cloud providers (AWS, Azure, GCP, Kubernetes) or locally.
    • cml pr: Commit files to a new branch and create a pull request.
    • cml comment: Post a Markdown report as a commit comment.
    • cml check: Post a Markdown report as a GitHub check.
    • cml tensorboard: Generate a link to a tensorboard.dev page.

  4. Set up a CML development environment

    To develop CML locally, you need Node.js 16.x installed. Clone your fork of the repository and install the dependencies to set up coding style pre-commit hooks.
    $ git clone git@github.com:<your-username>/cml.git
$ cd cml
$ npm install

  5. Authenticate CML on GitHub self-hosted runners

    CML requires authentication to create reports and comments. You can use either a Personal Access Token (PAT) or a GitHub App. It is recommended to use a dedicated 'bot account' rather than a personal account to limit security exposure.
    # Example using a GitHub App to generate a token on-the-fly
steps:
  - uses: navikt/github-app-token-generator@v1
    id: get-token
    with:
      private-key: ${{ secrets.CML_GITHUB_APP_PEM }}
      app-id: ${{ secrets.CML_GITHUB_APP_ID }}
  - uses: actions/checkout@v3
      token: ${{ steps.get-token.outputs.token }}
  - name: Train model
    env:
      REPO_TOKEN: ${{ steps.get-token.outputs.token }}
    run: |
      ...
      cml comment create report.md

  6. Create a CML report in a GitHub Action using Docker

    You can use a CML-enabled Docker container to run machine learning experiments and post results (metrics and plots) as comments on a GitHub Pull Request. The cml comment create command is used to publish the generated markdown report.
    name: CML
on: [push]
jobs:
  train-and-report:
    runs-on: ubuntu-latest
    container: docker://ghcr.io/iterative/cml:0-dvc2-base1
    steps:
      - uses: actions/checkout@v3
      - name: Train model
        env:
          REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          pip install -r requirements.txt
          python train.py  # generate plot.png
          # Create CML report
          cat metrics.txt >> report.md
          echo '![](./plot.png "Confusion Matrix")' >> report.md
          cml comment create report.md

  7. Install CML using the setup-cml GitHub Action

    The iterative/setup-cml action allows you to use CML commands (like cml comment create and cml runner) in your GitHub Actions workflow without requiring a specific Docker container. This is useful for mixing CML with other data science tools. Note that this action does not include DVC; use the Setup DVC Action for DVC dependencies.
    steps:
  - uses: iterative/setup-cml@v1

  8. Integrate DVC with CML in GitLab CI

    Use the iterativeai/cml:0-dvc2-base1 Docker image in GitLab CI to have Python, DVC, and CML pre-installed. This simplifies the pipeline by removing the need for separate installation steps.
    train-and-report:
  image: iterativeai/cml:0-dvc2-base1
  script:
    - pip install -r requirements.txt
    - dvc pull data --run-cache
    - dvc repro
    - echo "## Metrics: workflow vs. main" >> report.md
    - git fetch --depth=1 origin main:main
    - dvc metrics diff --show-md main >> report.md
    - dvc plots diff --target loss.csv --show-vega main > vega.json
    - vl2png vega.json > plot.png
    - echo '![](./plot.png "Training Loss")' >> report.md
    - cml comment create report.md

  9. Create a CML report in Bitbucket Pipelines

    Use the iterativeai/cml:0-dvc2-base1 image in bitbucket-pipelines.yml to run training scripts and post a markdown report as a comment using cml comment create.
    image: iterativeai/cml:0-dvc2-base1
pipelines:
  default:
    - step:
        name: Train and Report
        script:
          - pip install -r requirements.txt
          - python train.py
          - cat metrics.txt >> report.md
          - echo '![](./plot.png "Confusion Matrix")' >> report.md
          - cml comment create report.md

  10. Authenticate CML with Bitbucket Cloud

    Bitbucket Cloud uses a Base64 encoded string of the username and password for the REPO_TOKEN variable. Use either a personal account with an App Password (Read: Account; Write: Pull requests, Pipelines, Runners) or a dedicated bot account.
    # Generate the token
echo -n $USERNAME:$PASSWORD | base64
    1. Copy the resulting Base64 string.
    2. Add to Bitbucket: Repository Settings → Repository Variables.
      • Name: REPO_TOKEN
      • Value: [Base64 String]
      • Secured: Enabled

  11. Authenticate CML with GitHub Apps

    Alternatively to PATs, a GitHub App can be used to generate tokens on-the-fly. This requires the CML_GITHUB_APP_ID and CML_GITHUB_APP_PEM (private key) secrets. The App must have the following write permissions:

    • Repository: Administration (for cml runner), Checks (for cml check), and Pull requests (for cml pr and cml comment).
    • Organization: Self-hosted runners (for cml runner).
    steps:
  - uses: navikt/github-app-token-generator@v1
    id: get-token
    with:
      private-key: ${{ secrets.CML_GITHUB_APP_PEM }}
      app-id: ${{ secrets.CML_GITHUB_APP_ID }}
  - uses: actions/checkout@v3
      token: ${{ steps.get-token.outputs.token }}
  - name: Train model
    env:
      REPO_TOKEN: ${{ steps.get-token.outputs.token }}
    run: |
      ...
      cml comment create report.md

  12. Implement a complete CML workflow with setup-cml

    This workflow demonstrates using the setup-cml action to run a training script and post a markdown report containing a visualization to a GitHub Pull Request.
    name: CML
on: [push]
jobs:
  train-and-report:
    runs-on: ubuntu-latest
    steps:
      - uses: iterative/setup-cml@v1
      - uses: actions/checkout@v3
      - name: Train model
        env:
          REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          pip install -r requirements.txt
          python train.py  # generate plot.png

          echo "# CML report" >> report.md
          echo '![](./plot.png "Confusion Matrix")' >> report.md
          cml comment create report.md