DOCSEARCHbyAlgolia
Submit Docs

Arcus - Azure Machine Learning

website·Indexed 32 days ago

https://azure-ml.arcus-azure.net/

A library that wraps the Azure ML SDK to simplify interactive experimentations, asynchronous trainings, and logging experiments. It includes data collection modules for downloading data from various sources, such as Kaggle datasets, into Azure ML environments. Available in version v1.1.0 and Next.

Tokens
5.8K
Snippets
31
Records
34
Agent score
47%

What's inside Arcus

  1. Overview of Arcus - Azure Machine Learning capabilities

    Arcus is an open-source library that reduces the overhead of the AzureML SDK. Key capabilities include:

    • Workspace Connectivity: Simplified connection to Azure ML workspaces.
    • Hybrid Execution: Run experiments in local development environments (Jupyter notebooks or .py scripts) while automatically tracking runs, parameters, and results in the Azure ML workspace.
    • Cloud Transition: Execute the same training logic and dependency files both locally and in the cloud.
    • Hyperparameter Tuning: Tracking and tracing for Grid Searches.
    • Data Management: Methods for data collection and importing data into Azure Machine Learning.
    • Experiment Logging: Logging of metrics and results during the training phase.

  2. Overview of Arcus - Azure Machine Learning capabilities

    Arcus is an open-source library that streamlines Azure ML development by reducing the overhead of the AzureML SDK. Key capabilities include:

    • Workspace Connectivity: Simplified connection to Azure ML workspaces.
    • Hybrid Execution: Run experiments locally (Jupyter notebooks or .py scripts) while automatically tracking runs, parameters, and results in the Azure ML workspace.
    • Cloud Scaling: Execute training using the same logic and dependency files both locally and in the cloud.
    • Hyperparameter Tuning: Tracking and tracing for Grid Searches.
    • Experiment Management: Logging metrics, data collection, and importing data into Azure ML.

  3. Mount or download FileDataSets for Azure ML training

    When using an Estimator for training, you can make FileDataSet objects available on the training cluster using two different methods:

    1. Mounting: The dataset is referenced on the cluster. Files are accessed or downloaded on-demand only when the script performs an action (e.g., listing files or reading a specific file). This is generally faster for startup.
    2. Downloading: All files are fully downloaded onto the training cluster's training folder before the script starts. This may increase the initial startup time but ensures all data is local.

    Regardless of the method used, files are made available in a directory named exactly after the dataset, allowing the training script to access them using the same logic.

    mount_dataset_name = 'ds_to_mount'
download_dataset_name = 'ds_to_download'

# Start training with one dataset mounted and another downloaded
trainer.start_training(
    training_name,
    environment_type='tensorflow',
    input_datasets=[mount_dataset_name],
    input_datasets_to_download=[download_dataset_name],
    compute_target='gp-cluster',
    gpu_compute=True,
    script_parameters=arguments
)

  4. Access datasets and log metrics during training

    Within the training script, file datasets are available in a relative directory named after the dataset. Tabular datasets are accessed via AzureMLEnvironment. Use the AzureMLTrainer class to evaluate classifiers or log custom metrics to the AzureML Run. To persist the model, save it to the outputs folder, which is automatically uploaded to the AzureML workspace.
    # Load tabular dataset
df = work_env.load_tabular_dataset('datasetname')

# Evaluate classifier (uploads confusion matrix, ROC curve, and metrics)
trainer.evaluate_classifier(logreg, X_test, y_test, show_roc=True, upload_model=True)

# Log a custom metric (e.g., dice coefficient)
trainer._log_metrics('dice_coef_loss', list(fitted_model.history.history['dice_coef_loss'])[-1], description='')

# Persist model to AzureML workspace
fitted_model.save('outputs/model')

  5. Install Arcus - Azure Machine Learning

    Arcus - Azure Machine Learning is available via PyPi and can be installed using pip. To install the stable version, use the standard install command. To upgrade to the latest version, use the --upgrade flag. For development or alpha releases (pushed during CI builds), use the --pre flag.
    # Install stable version
pip3 install arcus-azureml

# Upgrade to latest stable version
pip3 install --upgrade arcus-azureml

# Install or upgrade to the latest alpha/dev release
pip3 install --upgrade --pre arcus-azureml

  6. Download Kaggle datasets to AzureML

    Use the KaggleDataCollector class to download datasets from Kaggle into an AzureML workspace (DataStore & Dataset). The copy_to_azureml method handles the transfer. For security, it is recommended to store Kaggle credentials in an Azure KeyVault; if user_name and user_key are omitted, the collector defaults to retrieving KAGGLE-USERNAME and KAGGLE-KEY secrets from the KeyVault.
    # Creating the AzureML environment
aml_environment = AzureMLEnvironment.CreateFromContext()

# Create the KaggleDataCollector class
collector = KaggleDataCollector()

# Copy the dataset to Azure ML
collector.copy_to_azureml(aml_environment, 'dataset-name', local_path='kaggle_data')

  7. Install Arcus - Azure Machine Learning

    Arcus is available via PyPi and can be installed using pip. You can install the stable version, upgrade an existing installation, or install pre-release/dev versions using the --pre flag.
    # Install stable version
pip3 install arcus-azureml

# Upgrade to latest stable version
pip3 install --upgrade arcus-azureml

# Install or upgrade to the latest dev/alpha release
pip3 install --upgrade --pre arcus-azureml

  8. Create a new training package with arcus-azureml

    Use the AzureMLEnvironment class to automatically create a training directory based on an experiment name. This process generates two essential files: train.py (a standard training script template) and requirements.txt (default packages for the Docker image). If requirements.txt is modified, a new Docker image will be built during the next training run, which may increase training time.
    from arcus.azureml.environment.aml_environment import AzureMLEnvironment

work_env = AzureMLEnvironment.Create()
training_name = 'my-training'
trainer = work_env.start_experiment(training_name)
trainer.setup_training(training_name, overwrite=False)

  9. Access data and log metrics in AzureML training

    Within the training script:

    • File datasets: Available in a relative directory named after the dataset (hyphens are not supported).
    • Tabular datasets: Loaded via AzureMLEnvironment.load_tabular_dataset.
    • Metrics: Logged using trainer._log_metrics.
    • Model Persistence: Save models to the outputs folder to ensure they are persisted to the AzureML workspace.
    # Load tabular dataset
df = aml_environment.load_tabular_dataset('datasetname')

# Log a custom metric (e.g., dice coefficient)
trainer._log_metrics('dice_coef_loss', list(fitted_model.history.history['dice_coef_loss'])[-1], description='')

# Evaluate a classifier and upload results/model
trainer.evaluate_classifier(logreg, X_test, y_test, show_roc=True, upload_model=True)

# Persist model to AzureML workspace
fitted_model.save('outputs/model')

  10. Launch training on local or AzureML compute

    Use trainer.start_training to execute the package. You must specify an environment_type to determine the base Docker image:

    • tensorflow: TensorFlow/Keras (CPU/GPU)
    • sklearn: Scikit-learn
    • pytorch: PyTorch
    • None: Empty image (all dependencies must be in requirements.txt)

    Set compute_target='local' for local Docker execution or provide the name of an AzureML compute target (e.g., 'gpu-cluster') for cloud execution.

    arguments = {
    '--epochs': 5,
    '--filter_count': 5,
    '--max_images': 20
}

# Local execution (requires Docker running locally)
trainer.start_training(
    training_name, 
    environment_type='tensorflow', 
    compute_target='local', 
    gpu_compute=True, 
    script_parameters=arguments
)

# AzureML Compute execution
trainer.start_training(
    training_name, 
    environment_type='tensorflow', 
    compute_target='gpu-cluster', 
    gpu_compute=True, 
    script_parameters=arguments
)

  11. Create a new training package with arcus-azureml

    Use AzureMLEnvironment to initialize a training package. This process creates a directory named after the experiment containing two essential files: train.py (a standard training script template) and requirements.txt (defines Python packages for the Docker image).
    from arcus.azureml.environment.aml_environment import AzureMLEnvironment

work_env = AzureMLEnvironment.Create()
training_name = 'my-training'
trainer = work_env.start_experiment(training_name)
trainer.setup_training(training_name, overwrite=False)

  12. Connect to an AzureML workspace using AzureMLEnvironment

    The AzureMLEnvironment.Create method allows you to connect to an AzureML workspace using three different strategies depending on your environment and authentication needs.
    from arcus.azureml.environment.aml_environment import AzureMLEnvironment

# Option 1: Using a local configuration file (e.g., .azureml/config.json)
# Config file format: {"subscription_id": "", "resource_group": "", "workspace_name": ""}
work_env = AzureMLEnvironment.Create(config_file='.azureml/config.json')

# Option 2: Passing parameters directly in the constructor
# write_config=True saves these details to the config file for future use
work_env = AzureMLEnvironment.Create(subscription_id='id', resource_group='group', workspace_name='name', write_config=True)

# Option 3: Connect to the current running context (ideal for Estimators or training scripts)
work_env = AzureMLEnvironment.CreateFromContext()