Aim Documentation

Aim is an open-source, self-hosted ML experiment tracking tool designed to manage tens of thousands of training runs. It provides a UI for exploring and comparing training runs, and an SDK for programmatic access to metadata, which is useful for automation and Jupyter Notebook analysis.

Key capabilities include:

• Metadata Logging: Track ML experiments and any metadata across your pipeline with integrations for popular ML frameworks.
• Visualization: Use Aim Explorers to visualize and compare metadata, including grouping and aggregation.
• Querying: Query metadata using Python expressions.
• Training Monitoring: Track system information, resource usage, and set up real-time alerting/notifications on training progress.
• Organization: Organize runs using tags and experiments for easy debugging and centralized dashboarding.

The Aim CLI provides a command-line interface to organize, record, search, and compare AI experiments. Supported commands include:

• init: Initialize an Aim repository.
• version: Display the installed Aim CLI version.
• up: Launch the Aim web UI.
• reindex: Process 'in progress' runs and optimize finished runs.
• server: Run a remote tracking server for RPC requests.
• runs: Manage run data (list, remove, copy, move, upload).
• convert: Convert 3rd party data (Tensorboard, MLFlow) to Aim format.
• storage: Perform repository maintenance (upgrade, restore, reindex, prune).

Aim is a tool designed for experiment tracking with a focus on three core criteria:

• Run data isolation: Each training run process is isolated in terms of data and does not require additional services to run.
• Scalability: The Aim web app can handle thousands of training runs. For distributed multi-host environments, Aim provides a Remote Tracking server (available from v3.4.0) to run multiple parallel experiments.
• Flexibility: The Aim UI and query language allow for arbitrary selection, grouping, and filtering of tracked data.

Aim UI is a powerful interface for interacting with and comparing ML training runs tracked by Aim. It is primarily built around Explorers, which allow users to query and compare various types of tracked data such as metrics, images, parameters, and scatterplots. Key capabilities include:

• Querying and Comparing: Use Explorers to manipulate and compare results across large numbers of experiments.
• Deep-Dive Analysis: Navigate from grouped views to a Single Run Page to observe all tracked information (params, metrics, images, distributions, etc.) for a specific run.
• Runs Management: Search through runs, view differences (diffs), and inspect last metric values.
• Organization: Tag runs, delete or archive them, and save Explorer states to share reproducible experiment analyses with teams.

Aim's data model is built around two primary abstractions:

• Aim Run: Represented by the aim.Run SDK class, a Run is an abstraction for the tracked data of a single experiment. It is used in training scripts to trace metrics, objects, and hyperparameters. Runs are queryable and can be explored via the UI.
• Aim Repo: Represented by the aim.Repo SDK class, a Repo is a centralized directory (acting like a database) that stores data for multiple training experiments (multiple Runs). It is responsible for managing repository resources and querying/iterating over stored data.

The Params explorer provides a parallel coordinates view for metrics and parameters, which is specifically designed for hyperparameter search analysis. Features include:

• Querying any metrics and parameters.
• Grouping runs or dividing them into subplots.
• Applying chart indicators to identify correlations.

A Run Sequence is a set of homogeneous, ordered objects bound to a Run. When tracking values, they are appended to an existing or newly created Sequence.

Key characteristics:

• Sequences are agnostic to the element type they hold.
• They can be queried using Aim QL and sliced.
• UI representation and SDK methods depend on the element type (e.g., a Metric sequence of scalars can be converted to a numpy.ndarray).

The Runs explorer provides a holistic view of all logged machine learning runs. Use it to view the last tracked values for each metric and inspect tracked hyperparameters. You can search for specific runs using the following criteria:

• Date
• Experiment
• Hash
• Tag
• Parameters
• Run or Experiment name

The Aim Home Page serves as a high-level dashboard for monitoring training activities. It is organized into three primary functional areas:

1. Explore Aim: Provides quick access to frequently used features, recent searches, experiment data, tags, and bookmarks.
2. Overview: Displays high-level training statistics, active runs, a contributions heatmap, and an activity feed.
3. Latest Changes: Contains version updates, the Aim changelog, and documentation links.

AimQL is a powerful query language used to filter stored metadata. It uses python expressions evaluated in a Boolean context. When iterating over entities (like run or metric), the expression is evaluated; if it is "truthy", the entity is yielded.

Key Concepts:

• Entities: Data is organized into entities such as run and metric.
• Filtering Only: AimQL is currently used exclusively for filtering data, not for sorting or aggregating.
• Security: Expressions are evaluated using RestrictedPython to prevent unsafe function execution.

The Single run page provides a detailed view of all metadata associated with a specific run. It is accessible via tables and tooltips throughout the UI. On this page, you can:

• View all logged parameters for the run.
• View all tracked metrics, including system metrics.