Ax Adaptive Experimentation Platform

Ax provides several advanced features for real-world optimization tasks:

• Expressive API: Handles complex search spaces, multiple objectives, constraints on parameters and outcomes, and noisy observations. It supports parallel design suggestions (synchronous and asynchronous) and early-stopping of evaluations.
• State-of-the-art Methods: Leverages Bayesian optimization algorithms implemented in BoTorch.
• Flexibility: Highly configurable, allowing users to plug in novel optimization algorithms, models, and experimentation flows.
• Production Readiness: Includes automation, orchestration features, and robust error handling for large-scale deployment.

Ax uses Bayesian Optimization (BO) as its core optimization engine. BO is an adaptive experimentation method that balances exploration (learning about unknown parameter regions) and exploitation (refining known good parameter regions).

Ax implements BO by:

1. Building a surrogate model (typically a Gaussian Process) to predict outcomes and uncertainty at unobserved points.
2. Using an acquisition function to identify the next best candidate parameterization to evaluate.
3. Iteratively updating the surrogate model with new observations.

This approach is particularly effective for expensive-to-evaluate objective functions where minimizing the number of evaluations is critical.

Ax's Analysis module provides a framework for producing plots, tables, and messages to help understand experiments. Analyses implement a compute method that consumes an Experiment, GenerationStrategy, and/or Adapter to output a collection of AnalysisCard objects.

There are three base classes provided by Ax:

1. Analysis: For creating tables.
2. PlotlyAnalysis: For producing plots using the Plotly library.
3. MarkdownAnalysis: For producing messages.

Analysis cards can be saved to the database using save_analysis_cards, allowing for pre-computation and later display.

A GenerationStrategy in Ax acts as a finite state machine that defines the optimization methodology for an experiment. It pre-specifies optimization algorithms and the dynamic conditions (transitions) for moving between them.

A strategy consists of:

• Generation Nodes: Represent a specific 'generation purpose' (e.g., Quasi-random initialization or Bayesian optimization) that produces new trials.
• Node Transitions: Edges between nodes defined by TransitionCriterion. A transition occurs when all criteria on an edge are met.

Note: Not all nodes in a strategy must be traversed during an experiment.

Ax is a machine learning platform designed to automate and guide the experimentation process for tuning configurations. It is most effective for problems that are expensive to evaluate or where the number of evaluations must be limited.

Key use cases include:

• Machine Learning: Tuning hyperparameters like learning rates.
• Infrastructure & Compilers: Optimizing 'magic numbers' or compiler flags.
• Engineering: Tuning design parameters in physical engineering tasks.
• A/B Testing: Optimizing discrete configurations (e.g., different variants of a test).
• Simulations: Managing costly simulations using adaptive experimentation techniques.

Ax supports continuous (integer or floating point), discrete, and mixed-valued search spaces using Bayesian optimization.

Ax uses an iterative 'ask-tell' process to find optimal points (called Arms) by balancing exploration and exploitation. The optimization process is managed through three high-order components:

• Experiment: Tracks the entire optimization process, including state, Trials, SearchSpace, and OptimizationConfig.
• GenerationStrategy: Defines the methodology used to produce the next Arms to try.
• Orchestrator (optional): Automates the full experiment, including trial deployment and data fetching.

Note: It is recommended to interact with the optimization process via the Client API (using methods like Client.get_next_trials and Client.complete_trial) rather than interacting with the Experiment object directly.

To perform multi-objective optimization (MOO), use the configure_optimization method on an Ax Client instance. You specify multiple objectives by providing a comma-separated string of metric names.

By default, objectives are assumed to be maximized. To minimize an objective, prepend the metric name with a minus sign (-).

Example objective strings:

• "-cost, utility": Minimizes cost and maximizes utility.
• "metric_a, metric_b": Maximizes both metric_a and metric_b.

client.configure_optimization(objectives="-cost, utility")

The Orchestrator is a closed-loop manager class designed to automate the entire experiment lifecycle. It asynchronously deploys trial runs to external systems (like training job queues, simulators, or A/B test managers), polls for status, fetches results, and uses those results to generate new trials.

Key capabilities include:

• Closed-loop execution: Run an entire experiment with minimal code.
• Concurrency management: Maintain user-defined limits for parallel trials.
• Failure tolerance: Manage and track tolerated levels of failed trial runs.
• Persistence: Supports SQL storage for easy resumption of experiments.

To save experiment data to a SQL database, initialize a Client using a StorageConfig object. The StorageConfig requires a database URL (e.g., for SQLite, MySQL, or PostgreSQL). The Ax Client automatically persists experiment data, including configuration and metric setup, to the specified database at every stage.

client = Client()

url = "sqlite:///path/to/database.db"
storage_config = StorageConfig(url = url)

client.configure_experiment(...)
client.configure_optimization(...)

To run the Ax website locally, you need Node.js >= 18.x and Yarn.

1. Navigate to the website directory.
2. Install dependencies with yarn install.
3. Start the server with yarn start.
4. Access the site at http://localhost:3000.

Note: You may need to switch to the "Next" version of the website documentation to see your latest changes.

cd website
yarn install
yarn start