DOCSEARCHbyAlgolia
Submit Docs

Ambrosia Python Library

repository·main·Indexed 42 days ago

https://github.com/mtswebservices/ambrosia

A Python library for the full A/B testing pipeline, including pilot design, multi-group splitting, and effect measurement. Version 0.5.3.post1 supports pandas and Spark data formats. It features tools for variance reduction via Cuped, MultiCuped, and MLVarianceReducer, as well as a Designer for calculating effect uplift and group size, and a Tester for assessing statistical significance with multiple testing corrections.

Tokens
21.4K
Snippets
65
Records
86
Agent score
53%

What's inside Ambrosia

  1. Use the Designer for experiment design

    main

    The ambrosia.designer module provides tools to calculate A/B test parameters, including effect uplift, group size, and statistical power, using historical metric values.

    There are two primary approaches to designing experiments:

    1. Theoretical approach: Faster calculation of parameters.
    2. Empirical approach: Uses historical data to inform the design.

    Key components include:

    • Designer: The main class used to execute design runs via the .run() method.
    • design(): Function for standard experiment design.
    • design_binary(): Function specifically for binary (e.g., conversion rate) experiment design.
    • load_from_config(): Utility to initialize a designer from a configuration object.

  2. Available data preprocessing tools

    main

    Ambrosia provides several specialized modules for experimental data processing:

    • aggregation: Tools for combining or summarizing data points.
    • robust: Robust statistical methods for handling outliers or non-standard distributions.
    • simple_transformation: Basic data transformations.
    • advanced_transformations: Complex or specialized mathematical transformations.
    • processor: The core orchestration component for managing transformation pipelines.

  3. Perform Sample Ratio Mismatch (SRM) checks

    main

    The Tester can verify if observed group sizes match expected ratios using a chi-square test (at the strict 0.0005 level). A detected mismatch (SRM) typically indicates a broken assignment procedure, making results unreliable.

    How to enable SRM checks:

    1. Equal splits: Pass check_srm=True to the run method.
    2. Unequal splits: Provide the srm_expected_ratios parameter (e.g., {"A": 0.9, "B": 0.1}). Providing these ratios enables the check automatically.

    Requirements and Performance:

    • Data Granularity: The check assumes one row per randomization unit (e.g., one user). If using event- or session-level data, you must aggregate to the unit level first.
    • Spark Performance: For Spark data, the check incurs the cost of one count job per group.
    • Standalone Diagnostic: You can use the standalone function ambrosia.tools.srm.check_srm to run this diagnostic on any pandas or Spark dataframe.

  6. The Ambrosia A/B testing workflow

    main

    Ambrosia automates the routine calculation steps required for running A/B tests by providing tools for four distinct stages of an experiment lifecycle:

    1. Process: Handles raw data aggregation, outlier removal, and metric transformation. It supports storable, reusable data processing pipelines and methods for experiment acceleration.
    2. Design: Uses historical metric data to design experiment parameters, including effect uplift, group size, and statistical power, using either theoretical or empirical approaches.
    3. Split: Provides group split methods for creating control and test groups. It supports various strategies and multi-group splits. Note that currently, only batch data splitting methods are supported.
    4. Test: Performs statistical inference to calculate relative and absolute effects and construct confidence intervals for continuous and binary variables. Supported tests include t-tests, non-parametric tests, bootstrap, and others.

  8. Install Ambrosia

    main

    Ambrosia requires Python 3.9 or higher. You can install the core library using pip. If you need to process PySpark data, you must install the [spark] extra.

    To install the standard version:

    pip install ambrosia

    To install with PySpark support:

    pip install ambrosia[spark]
    pip install ambrosia[spark]

  9. Apply Multiple Hypothesis Correction (MHC) with Tester

    main

    When evaluating multiple metrics or multiple group combinations, Tester can automatically apply Multiple Hypothesis Correction (MHC) to adjust p-values and confidence intervals.

    The total number of hypotheses is calculated as: (number of group combinations) * (number of metrics).

    By default, the Bonferroni correction is applied. To disable this behavior and use uncorrected p-values, pass correction_method=None to the .run() method.

    # With default Bonferroni correction
multi_tester.run(method='theory')

# Without correction
multi_tester.run(method='theory', correction_method=None)

  10. Choose the right multiple testing correction method

    main

    The choice of correction_method depends on whether you want to control the Family-Wise Error Rate (FWER) or the False Discovery Rate (FDR):

    1. Control FWER (Probability of at least one false positive)

    Use these when you cannot afford even a single false discovery among your metrics. These methods are more conservative.

    • "bonferroni": Simplest and most conservative.
    • "holm": A step-down method that is more powerful than Bonferroni.
    • "holm-sidak"
    • "sidak"
    • "hommel"
    • "simes-hochberg"

    2. Control FDR (Expected proportion of false positives)

    Use these when you have many metrics (e.g., on a dashboard) and want a balance between discovering real effects and controlling the rate of false discoveries.

    • "fdr_bh" (Benjamini–Hochberg)
    • "fdr_by" (Benjamini–Yekutieli)

    Summary Table

    SituationRecommended correction_method
    One metric, two groupsNone
    Few key metrics, zero false positives allowed"holm" or "bonferroni"
    Many metrics on a dashboard"fdr_bh"
    Need adjusted confidence intervals"bonferroni" or "sidak"

  11. Compare Theoretical vs Empirical design approaches

    main

    Ambrosia provides two distinct methods for calculating experiment parameters:

    1. Theoretical Approach (method='theory')

    • Mechanism: Uses analytical formulas for the difference of normally distributed quantities.
    • Pros: Extremely fast; only requires mean and variance of the metric.
    • Cons: Assumes normality (though CLT helps for large groups); does not account for specific distribution features or custom statistical criteria.
    • Best for: Initial exploration and quick estimates.

    2. Empirical Approach (method='empiric')

    • Mechanism: Repeatedly samples groups from historical data, models the effect, and applies a statistical test (e.g., T-test) to estimate power.
    • Pros: Accounts for specific metric distributions and custom statistical criteria.
    • Cons: Computationally expensive (high CPU/Memory usage); results can be noisy if bs_samples is too low.
    • Best for: Finalizing designs where metric distributions are non-standard or specific criteria are required.

    Note: For binary metrics, do not use the empirical approach. Instead, use the specialized binary method described in separate documentation.

  12. Common interface for Ambrosia preprocessing classes

    main

    All preprocessing classes in Ambrosia share a consistent interface, making them easy to use and swap. The standard methods available are:

    • fit(dataframe, ...): Learns necessary parameters from the provided data.
    • transform(dataframe): Applies the learned transformation to the data.
    • fit_transform(dataframe, ...): Performs both fitting and transformation in a single step.
    • get_params_dict(): Returns a dictionary of the currently fitted parameters.
    • load_params(path): Loads pre-fitted parameters from a file (e.g., JSON).
    • store_params(path): Saves the currently fitted parameters to a file.

    Additionally, most transformation methods support an inplace=True parameter to modify the input DataFrame directly.