DOCSEARCHbyAlgolia
Submit Docs

MABWiser

repository·master·Indexed 43 days ago

https://github.com/fidelity/mabwiser

A Python research library for rapid prototyping of parallelizable contextual multi-armed bandit algorithms, designed for combinatorial optimization and personalized recommendation systems.

Tokens
38K
Snippets
96
Records
117
Agent score
54%

What's inside MABWiser

  1. Explore the MABWiser Public API modules

    master

    The MABWiser library is organized into several core modules that provide the functionality for Multi-Armed Bandit (MAB) algorithms, simulation environments, and utility functions:

    • mabwiser.base_mab: Contains the base classes and interfaces for bandit implementations.
    • mabwiser.mab: The primary module for accessing specific bandit algorithms and policies.
    • mabwiser.simulator: Provides tools to simulate bandit environments and evaluate policy performance.
    • mabwiser.utils: Contains helper functions and utilities for data processing and common tasks.

  2. Implement a new bandit algorithm by inheriting from BaseMAB

    master

    To implement a new bandit algorithm in MABWiser, your new bandit class should inherit from the BaseMAB abstract meta class. BaseMAB defines the public interface and dictates the required function signatures for core bandit operations.

    Key methods you must implement for training include:

    • fit(): Used for training the model.
    • partial_fit(): Used for online/incremental training.
    • fit_arm(): Used for training specific arms.

  3. What are Multi-Armed Bandits (MAB)?

    master

    Multi-armed bandit (MAB) algorithms are designed for sequential, online decision-making problems under uncertainty. They are used in scenarios where you must choose between multiple options (arms) and learn the best course of action by testing them sequentially.

    Core concepts include:

    • Exploitation: Sticking to known options to receive an expected reward.
    • Exploration: Choosing less-known options to potentially learn more about their performance.
    • Reward: The outcome received after pulling an arm.
    • Contextual Multi-Armed Bandits: A variant where the agent also observes side information (called context) at each time step. The optimal arm to pull may change depending on the current context. The ultimate goal is to maximize the cumulative expected reward over time.

  4. Core MAB Workflow: MAB class

    master

    The MAB class follows a scikit-learn style interface for rapid prototyping of multi-armed bandit algorithms. It supports context-free, parametric, and non-parametric contextual bandit models.

    Key methods:

    • fit(decisions, rewards): Trains the model on a sequence of decisions and their corresponding rewards.
    • predict(): Uses the trained model to predict the best arm(s).

    Initialization: mab = MAB(arms, learning_policy, neighborhood_policy=None)

  5. Understand the synthetic data generation for LinTS reproducibility

    master

    The synthetic dataset used for LinTS reproducibility experiments simulates a 4-armed bandit scenario where a single item is recommended per decision based on context.

    Data Generation Details:

    • Method: Uses Scikit-learn's make_classification to generate binary classification observations (context features and class 0/1).
    • Context Uniqueness: To ensure separate contexts for each arm, 100 binary classification observations are generated 4 times, using the arm index as the seed.
    • Mathematical Property: The generation ensures that $A^{-1}$ for each arm contains duplicate singular values.
    • Data Split: Follows a 70-30 train-test split regime.
    • Output: The resulting data is stored in simulated_data.csv.

  6. Avoid non-deterministic behavior in LinTS using Cholesky decomposition

    master
    The LinTS algorithm can suffer from non-deterministic behavior due to the way multivariate sampling is handled (often relying on SVD in older NumPy versions). To ensure consistent and reproducible results across different environments and operating systems, use Cholesky decomposition for multi-variate sampling instead of SVD-based methods.

  7. Core MABWiser Workflow: Fit, Predict, and Online Learning

    master

    MABWiser follows a standard machine learning workflow for Multi-Armed Bandits. You train a model using historical decisions and rewards, then use it to make future decisions.

    Key Methods:

    • fit(decisions, rewards, contexts=None): Trains the model on historical data. decisions and rewards can be lists, 1D numpy arrays, or pandas Series. contexts can be 2D lists, 2D numpy arrays, pandas Series, or DataFrames.
    • predict(contexts=None): Predicts the next best decision (arm). For contextual bandits, provide the context.
    • predict_expectations(contexts=None): Returns the expected reward for each arm.
    • partial_fit(decisions, rewards, contexts=None): Enables online learning by updating the model with new, incremental data.
    • add_arm(arm): Adds a new arm to the bandit.
    • warm_start(arm_to_features, distance_quantile=...): Initializes a new arm using feature information to avoid starting from scratch.
    from mabwiser.mab import MAB, LearningPolicy

# Initialize
model = MAB(arms=[1, 2], learning_policy=LearningPolicy.EpsilonGreedy(epsilon=0.1))

# Train
model.fit(decisions=[1, 2, 1], rewards=[10, 20, 15])

# Predict
prediction = model.predict()
expectations = model.predict_expectations()

# Online update
model.partial_fit(decisions=[2], rewards=[25])

# Add and warm start new arm
model.add_arm(3)
model.warm_start({3: [0.5, 0.5]}, distance_quantile=0.5)

  8. Set up the MacOS Big Sur MKL environment

    master

    To run specific notebooks (like MacOSBigSur_MKL.ipynb) that require a MacOS Big Sur environment with a NumPy MKL backend, follow these installation steps using Anaconda.

    Requirements

    • OS: MacOS Big Sur (specifically macOS-10.16-x86_64-i386-64bi for exact reproducibility).
    • Package Manager: Anaconda.

    Installation Steps

    Run the following commands from your Anaconda prompt:

    conda create -n mac_bs_mkl python=3.7.10
conda activate mac_bs_mkl
conda install numpy==1.18.5
conda install scikit-learn==0.24.1
conda install jupyter
pip install tensorflow==1.15.5
pip install pandas
pip install mabwiser

    Verify NumPy MKL Backend

    To ensure NumPy is correctly using the MKL backend, run the following in a Python environment:

    import numpy as np
np.show_config()

    Look for entries related to MKL in the output.

  9. Build MABWiser from source

    master

    If you need to build a wheel package for your specific platform from scratch, follow these steps:

    1. Clone the repository.
    2. Install setuptools and wheel if they are not already present.
    3. Build the source distribution and wheel.
    4. Install the generated .whl file from the dist/ directory.
    git clone https://github.com/fidelity/mabwiser.git
cd mabwiser
pip install setuptools wheel
python setup.py sdist bdist_wheel
pip install dist/mabwiser-X.X.X-py3-none-any.whl

  10. Reproduce LinTS results on MovieLens data

    master

    To reproduce the results shown in Tables 2 and 3 regarding LinTS reproducibility and the impact of Cholesky decomposition, you must run specific notebooks within matching computational environments to ensure deterministic numerical output.

    Numerical reproducibility in these tables depends on the underlying linear algebra libraries (OpenBLAS, LAPACK, or MKL). You must select the notebook that corresponds to your specific operating system and math library environment:

    • Linux RedHat OpenBLAS: Use LinuxRedHat_OpenBLAS.ipynb in the ../environments/LinuxRedHat_OpenBLAS environment.
    • Linux Sagemaker LAPACK: Use LinuxSagemaker_LAPACK.ipynb in the ../environments/LinuxSagemaker_LAPACK environment.
    • MacOS Darwin MKL: Use MacOSDarwin_MKL.ipynb in the ../environments/MacOSDarwin_MKL environment.
    • MacOS Darwin OpenBLAS: Use MacOSDarwin_OpenBLAS.ipynb in the ../environments/MacOSDarwin_OpenBLAS environment.
    • Windows OpenBLAS: Use Windows_OpenBLAS.ipynb in the ../environments/Windows_OpenBLAS environment.

    After running the environment-specific notebook, you can run the Analysis.ipynb notebook in any environment to generate the final numbers for the tables.

  11. How to add a new bandit algorithm to MABWiser

    master

    Adding a new bandit algorithm to MABWiser is a three-step process involving the public API, the core implementation, and testing. This process is designed to follow the library's architecture where users interact with immutable namedtuple objects via the LearningPolicy or NeighborhoodPolicy classes, while the actual logic resides in private implementor classes (prefixed with _) that inherit from BaseMAB.

    High-Level Workflow

    1. Expose the Public API: Register your policy in mab.py so users can instantiate it via LearningPolicy.YourPolicyName(...).
    2. Implement the Algorithm: Create a private class (e.g., _YourPolicyName) inheriting from BaseMAB in a new file under the /mabwiser directory.
    3. Test the Behavior: Ensure the policy works as expected through doctests and integration tests.
    # Example of how a user would interact with a newly added policy
from mabwiser.mab import MAB, LearningPolicy

# Data
arms = ['Arm1', 'Arm2']
decisions = ['Arm1', 'Arm1', 'Arm2', 'Arm1']
rewards = [20, 17, 25, 9]

# Model instantiated with the new policy
mab = MAB(arms, LearningPolicy.MyCoolPolicy(my_parameter=42))

# Training and Prediction
mab.fit(decisions, rewards)
mab.predict()