DOCSEARCHbyAlgolia
Submit Docs

Made With ML

repository·main·Indexed 13 days ago

https://github.com/gokumohandas/made-with-ml

A course and repository focused on combining machine learning with software engineering to design, develop, deploy, and iterate on production-grade ML applications. It covers the full ML lifecycle, including training with train.py, hyperparameter tuning with tune.py, model evaluation, and serving via serve.py. The project integrates with MLflow for experiment tracking, Anyscale for workspace and cluster management, and GitHub Actions for CI/CD automation.

Tokens
6.4K
Snippets
25
Records
25
Agent score
50%

What's inside Made With ML

  1. Set up an Anyscale Workspace

    main

    To run the course using Anyscale, create a workspace via the web UI or the CLI. Using the web UI, use the following configuration:

    • Workspace name: madewithml
    • Project: madewithml
    • Cluster environment name: madewithml-cluster-env
    • Compute config: madewithml-cluster-compute-g5.4xlarge (Ensure you toggle Select from saved configurations to find this).

    Alternatively, use the Anyscale CLI to create the workspace.

    anyscale workspace create ...

  2. Authenticate with Anyscale CLI

    main

    If you are running outside of Anyscale Workspaces (e.g., locally or on a self-managed cluster), you must explicitly set your Anyscale credentials using environment variables. You can retrieve your CLI token from the Anyscale credentials page.

    export ANYSCALE_HOST=https://console.anyscale.com
export ANYSCALE_CLI_TOKEN=$YOUR_CLI_TOKEN

  3. Run the Jupyter Notebook

    main

    The core machine learning workloads are explored in the notebooks/madewithml.ipynb notebook.

    Local Setup: Run Jupyter Lab from your terminal:

    # Start notebook
jupyter lab notebooks/madewithml.ipynb

    Anyscale Setup:

    1. Click the Jupyter icon at the top right of your Anyscale Workspace page.
    2. Navigate to the notebooks directory.
    3. Open madewithml.ipynb.

  4. Set up CI/CD with GitHub Actions

    main

    Automate deployment using GitHub Actions. The workflow follows these steps:

    1. Branching: Create a dev branch for changes.
    2. Secrets: Add ANYSCALE_HOST and ANYSCALE_CLI_TOKEN to your GitHub repository secrets (/settings/secrets/actions).
    3. Pushing: Commit and push to the dev branch. You will need to use a GitHub Personal Access Token (with repo and workflow scopes) as your password.
    4. Validation: Open a PR to main. This triggers the workloads.yaml workflow, which runs Anyscale Jobs and posts training/evaluation results as PR comments.
    5. Deployment: Merge the PR into main to trigger the serve.yaml workflow, which rolls out the new service to production.
    # Create dev branch
git remote set-url origin https://github.com/$GITHUB_USERNAME/Made-With-ML.git
git checkout -b dev

# Configure git user
git config --global user.name $GITHUB_USERNAME
git config --global user.email you@example.com

# Commit and push
git add .
git commit -m "your message"
git push origin dev

  5. Run tests for code, data, and models

    main

    The project uses pytest for three types of testing:

    1. Code Tests: python3 -m pytest tests/code
    2. Data Tests: Requires DATASET_LOC environment variable. pytest --dataset-loc=$DATASET_LOC tests/data
    3. Model Tests: Requires RUN_ID (retrieved via predict.py). pytest --run-id=$RUN_ID tests/model

    To generate coverage reports, use the --cov flag with pytest.

    # Code coverage with HTML report
python3 -m pytest tests/code --cov madewithml --cov-report html --disable-warnings

  6. Submit Anyscale jobs for ML workloads

    main

    To execute ML workloads (training, evaluation, etc.), submit a job using a YAML configuration file.

    Important: Before submitting, you must update the $GITHUB_USERNAME placeholders in your job configuration (e.g., deploy/jobs/workloads.yaml) within the runtime_env section. This ensures your code is correctly uploaded to S3 and environment variables are set.

    Example runtime_env configuration:

    runtime_env:
  working_dir: .
  upload_path: s3://madewithml/$GITHUB_USERNAME/jobs
  env_vars:
    GITHUB_USERNAME: $GITHUB_USERNAME
    anyscale job submit deploy/jobs/workloads.yaml

  7. Set up a local development environment

    main

    To run the course on your local laptop, follow these steps to set up a Python virtual environment and install dependencies. It is highly recommended to use Python 3.10 via pyenv (macOS) or pyenv-win (Windows).

    1. Set the PYTHONPATH to include the current directory.
    2. Create and activate a virtual environment.
    3. Upgrade core pip tools.
    4. Install requirements and configure pre-commit hooks.
    export PYTHONPATH=$PYTHONPATH:$PWD
python3 -m venv venv  # recommend using Python 3.10
source venv/bin/activate  # on Windows: venv\Scripts\activate
python3 -m pip install --upgrade pip setuptools wheel
python3 -m pip install -r requirements.txt
pre-commit install
pre-commit autoupdate

  8. Deploy and manage Anyscale services

    main

    Once workloads are complete, you can serve your model using Anyscale Services.

    1. Configure: Update $GITHUB_USERNAME in your service config (e.g., deploy/services/serve_model.yaml).
    2. Rollout: Deploy the service.
    3. Query: Interact with the service endpoint via HTTP.
    4. Manage: Rollback to previous versions or terminate the service as needed.

    Note: Ensure the ray_serve_config.import_path correctly points to your entrypoint.

    # Rollout service
anyscale service rollout -f deploy/services/serve_model.yaml

# Query
curl -X POST -H "Content-Type: application/json" -H "Authorization: Bearer $SECRET_TOKEN" -d '{
  "title": "Transfer learning with transformers",
  "description": "Using transformers for transfer learning on text classification tasks."
}' $SERVICE_ENDPOINT/predict/

# Rollback
anyscale service rollback -f $SERVICE_CONFIG --name $SERVICE_NAME

# Terminate
anyscale service terminate --name $SERVICE_NAME

  9. Serve models with `serve.py`

    main

    Deploy a model as a service using madewithml/serve.py. This typically requires a Ray cluster.

    Local Setup:

    1. Start Ray: ray start --head
    2. Run the server: python madewithml/serve.py --run_id $RUN_ID

    Inference via Python: Once running, you can send POST requests to http://127.0.0.1:8000/predict with a JSON payload containing title and description.

    import json
import requests
title = "Transfer learning with transformers"
description = "Using transformers for transfer learning on text classification tasks."
json_data = json.dumps({"title": title, "description": description})
requests.post("http://127.0.0.1:8000/predict", data=json_data).json()

  10. View experiments with MLflow Tracking UI

    main

    The project uses MLflow to track experiments. To view the dashboard locally, start an MLflow server pointing to the project's model registry.

    To find the registry path, use the config module from madewithml.

    export MODEL_REGISTRY=$(python -c "from madewithml import config; print(config.MODEL_REGISTRY)")
mlflow server -h 0.0.0.0 -p 8080 --backend-store-uri $MODEL_REGISTRY

  11. Configure Git and Credentials

    main

    First, create a new repository on GitHub named Made-With-ML. Important: Toggle Add a README file during creation to ensure a main branch is initialized. Then, clone the repository:

    git clone https://github.com/GokuMohandas/Made-With-ML.git .

    Next, set up your credentials by creating a .env file and defining your GitHub username:

    touch .env

    Inside .env:

    GITHUB_USERNAME="CHANGE_THIS_TO_YOUR_USERNAME"

    Then, source the file to load the environment variables:

    source .env

  12. Compare Zero-shot vs Few-shot learning

    main

    The notebook demonstrates two prompting strategies for benchmarking LLMs:

    1. Zero-shot learning: Provide only the system_content (instructions) without any examples of the desired input-output mapping.
    2. Few-shot learning: Augment the assistant_content with a small number of examples (e.g., 2 samples per class) from the training data. This helps the model generalize by seeing the expected format and label mapping.

    Implementation Pattern: To implement few-shot, extract samples from the training set for each unique tag and format them into a string for the assistant_content parameter.

    # Few-shot context construction example
num_samples = 2
additional_context = []
for tag in tags:
    samples = train_df[["title", "description", "tag"]][train_df.tag == tag][:num_samples].to_dict(orient="records")
    additional_context.extend(samples)

assistant_content = f"Here are some examples with the correct labels: {additional_context}"