tea-tasting

The multiple hypothesis testing problem occurs when an A/B test involves multiple success metrics or multiple treatment variants, increasing the risk of false positives.

tea-tasting provides two main categories of correction methods:

1. False Discovery Rate (FDR) controlling procedures: Controls the expected proportion of false discoveries among all rejections.

   • Benjamini-Hochberg: Default; assumes non-negative correlation between hypotheses.
   • Benjamini-Yekutieli: Use by setting arbitrary_dependence=True; assumes arbitrary dependence.

2. Family-wise Error Rate (FWER) controlling procedures: Controls the probability of making at least one type I error.

   • Hochberg's step-up: Default; assumes non-negative correlation.
   • Holm's step-down: Use by setting arbitrary_dependence=True; assumes arbitrary dependence.

When using FWER with arbitrary dependence, it is recommended to use the method="bonferroni" parameter.

When an experiment has more than two variants, tea-tasting performs pairwise comparisons using two-sample statistical tests.

Controlling Comparisons

• Specified Control: If you pass a control ID to analyze(..., control=ID), that variant is compared against every other variant.
• Automatic Control: If control=None (default), tea-tasting compares all possible pairs. In each pair, the variant with the lowest ID is treated as the control.

To enable pairwise comparisons for all variants, set all_variants=True in the analyze method.

Accessing Results

The result of an analysis with multiple variants is a mapping where keys are tuples representing the pair (control, treatment). You can access a specific pair's results using this tuple as a key.

>>> # Compare all variants pairwise
>>> results = experiment.analyze(data_three_variants, all_variants=True)

>>> # Access results for a specific pair (e.g., variant 0 vs variant 1)
>>> pair_results = results[0, 1]

You can reduce variance in your A/B test results using CUPED or CUPAC within the Mean and RatioOfMeans metric classes. This requires providing pre-experimental data as covariates.

1. Prepare Data: Ensure your dataset contains covariate columns (e.g., sessions_covariate, orders_covariate, revenue_covariate). If using tt.make_users_data, set covariates=True.
2. Configure Metrics:
   • For Mean: Use the covariate parameter to specify the column name.
   • For RatioOfMeans: Use numer_covariate and denom_covariate to specify covariates for the numerator and denominator respectively.

>>> data_cuped = tt.make_users_data(rng=42, covariates=True)
>>> experiment_cuped = tt.Experiment(
...     sessions_per_user=tt.Mean("sessions", "sessions_covariate"),
...     orders_per_session=tt.RatioOfMeans(
...         numer="orders",
...         denom="sessions",
...         numer_covariate="orders_covariate",
...         denom_covariate="sessions_covariate",
...     ),
...     orders_per_user=tt.Mean("orders", "orders_covariate"),
...     revenue_per_user=tt.Mean("revenue", "revenue_covariate"),
... )
>>> result_cuped = experiment_cuped.analyze(data_cuped)

To speed up simulations, use the map_ parameter to provide an alternative mapping function (such as those from concurrent.futures or multiprocessing).

Implementation Details

• map_ parameter: Accepts a mapping function like executor.map.
• batch_size parameter: Useful when running in parallel to group tasks.

Platform & Environment Constraints

• WASM/Online Playgrounds: Parallel execution using multiprocessing is not supported in marimo online playgrounds (WASM-based) because they run entirely in the browser.
• Windows vs. POSIX:
  • On POSIX systems, you can use the "fork" multiprocessing start method via mp.get_context("fork"), which allows worker processes to inherit functions defined in the current session.
  • On Windows, you must use the "spawn" method. To ensure compatibility, define your treatment functions and data generators at the top level of a Python module and wrap your execution in an if __name__ == "__main__": block.

Alternative Parallelism Options

• concurrent.futures.ThreadPoolExecutor: Avoids multiprocessing import issues as threads run in the same process (speedup depends on GIL release).
• joblib: Good for interactive workflows.
• Distributed Engines: Use Dask or Ray for simulations spanning multiple machines.

>>> import concurrent.futures
>>> import multiprocessing as mp
>>> import tqdm

>>> ctx = mp.get_context("fork")
>>> with concurrent.futures.ProcessPoolExecutor(mp_context=ctx) as executor:
...     results_parallel = experiment.simulate(
...         data,
...         100,
...         rng=42,
...         treat=treat,
...         batch_size=10,
...         map_=executor.map,
...         progress=tqdm.tqdm,
...     )