contextualbandits

This package implements solutions for the contextual bandits problem (associative reinforcement learning).

Core Constraints:

• Discrete Rewards: This package deals only with discrete rewards in the set {0, 1}. It is not intended for continuous rewards.
• Shared Covariates: It assumes all arms see the same covariates (features).
• Iterative Process: At each round, an agent receives covariates, chooses an arm, and receives a reward for that arm only.

Key Problem Areas Covered:

• Online Learning (contextualbandits.online): Managing exploration/exploitation in real-time.
• Off-policy Learning (contextualbandits.offpolicy): Learning from data collected by a different policy (Counter-factual Risk Minimization).
• Policy Evaluation (contextualbandits.evaluation): Evaluating strategies based on partially-labeled data.

The package includes non-stochastic linear regression procedures with exact partial_fit solutions. These are recommended to be used alongside online policies to achieve better incremental updates.

# Available classes in contextualbandits.linreg
# - LinearRegression
# - ElasticNet

The package is available on PyPI. Use the standard pip command to install the latest version. If you encounter errors related to C code compilation, you can install an earlier pure-Python version instead.

# Standard installation
pip install contextualbandits

# Pure-Python fallback if C compilation fails
pip install contextualbandits==0.1.8.5

Detailed usage examples with public datasets are available in the following IPython notebooks:

• Online Contextual Bandits
• Off-policy Learning in Contextual Bandits
• Policy Evaluation in Contextual Bandits

The setup script attempts to add the -march=native flag to tune the package for the local CPU. This may make the resulting binary incompatible with other machines.

To ensure maximum compatibility (at the cost of speed) when building wheels or Docker images, use one of the following methods:

1. Disable native tuning: Set the DONT_SET_MARCH environment variable to 1.
2. Manually specify architecture: Set CFLAGS and CXXFLAGS to a specific architecture (e.g., -march=x86-64).

# Option 1: Disable native tuning
export DONT_SET_MARCH=1
pip install contextualbandits

# Option 2: Specify architecture
export CFLAGS="-march=x86-64"
export CXXFLAGS="-march=x86-64"
pip install contextualbandits

The contextualbandits.online module provides various policy classes for online learning. If you are unsure which method to choose, BootstrappedUCB is recommended as a safe starting point.

Policies are categorized by their exploration strategy:

• Randomized: AdaptiveGreedy, SoftmaxExplorer, EpsilonGreedy, ExploreFirst.
• Active choices: ActiveExplorer, AdaptiveGreedy (with active_choice != None), ExploreFirst (with prob_active_choice > 0).
• Thompson sampling: BootstrappedTS, PartitionedTS, ParametricTS, LogisticTS, LinTS.
• Upper confidence bound (UCB): BootstrappedUCB, PartitionedUCB, LogisticUCB, LinUCB.
• Naive: SeparateClassifiers.

Standard Python pickle is likely to fail when serializing objects from this library. Instead, use cloudpickle or dill, which provide the same syntax as pickle but handle the library's complex objects correctly.

import cloudpickle
from sklearn.linear_model import SGDClassifier
from contextualbandits.online import BootstrappedUCB

# Initialize model
m = BootstrappedUCB(SGDClassifier(loss="log_loss"), nchoices = 5, batch_train = True)

# Serialize
cloudpickle.dump(m, open("saved_ucb_model.pkl", "wb"))

# Deserialize
m = cloudpickle.load(open("saved_ucb_model.pkl", "rb"))

The package can be installed using pip. Note that as of version 0.2.0, the package contains Cython code and requires a C/C++ compiler configured for Python.

Standard installation:

pip install contextualbandits

If the standard installation fails, try:

pip install --no-use-pep517 contextualbandits

To install an older, pure-python version (no C compiler required):

pip install contextualbandits==0.1.8.5