The ValueIterationGS class implements the Gauss-Seidel variant of the Value Iteration algorithm. This variant typically converges faster by using updated values immediately within the same iteration.

Parameters:

• transitions: Transition probability matrices.
• reward: Reward matrices or vectors.
• discount: Discount factor (float).
• epsilon (float, optional): Stopping criterion. Default is 0.01.
• max_iter (int, optional): Maximum number of iterations. Default is 10.
• initial_value (array, optional): The starting value function. Default is a vector of zeros.
• skip_check (bool, optional): If True, skips validation of transitions and reward.

Data Attributes:

• policy: The epsilon-optimal policy.
• iter: Number of completed iterations.
• time: CPU time used.

import mdptoolbox
import numpy as np

P, R = mdptoolbox.example.forest()
vigs = mdptoolbox.mdp.ValueIterationGS(P, R, 0.9)
vigs.run()

print(vigs.V)
print(vigs.policy)

The QLearning class implements the Q-learning algorithm to solve a discounted Markov Decision Process. It learns the Q-matrix (state-action values) through iterative simulation of trajectories.

Parameters:

• transitions: Transition probability matrices.
• reward: Reward matrices or vectors.
• discount: Discount factor (float).
• n_iter (int, optional): Number of iterations to execute. Must be an integer greater than 10,000. Default is 10,000.
• skip_check (bool, optional): If True, skips the validation check on transitions and reward arguments.

Data Attributes:

• Q: The learned Q matrix of shape (S, A).
• V: The learned value function of shape (S).
• policy: The learned optimal policy of shape (S).
• mean_discrepancy: A vector of the mean V discrepancy over 100 iterations.

import numpy as np
import mdptoolbox

# Example setup
P = np.array([[[0.5, 0.5],[0.8, 0.2]],[[0, 1],[0.1, 0.9]]])
R = np.array([[5, 10], [-1, 2]])
np.random.seed(0)

# Initialize and run Q-Learning
ql = mdptoolbox.mdp.QLearning(P, R, 0.9)
ql.run()

print(ql.Q)
print(ql.policy)

The MDP class is the base class for all Markov Decision Process solvers in mdptoolbox. To instantiate it, you must provide transition probabilities, rewards, a discount factor, a stopping criterion (epsilon), and a maximum number of iterations (max_iter).

Parameters

• transitions: Transition probability matrices. Can be a numpy array of shape (A, S, S) or a list/tuple of length A containing (S, S) arrays (useful for sparse matrices).
• reward: Reward matrices or vectors. Supported shapes include (S, A), (S,), or (A, S, S). Can also be a list of length A containing (S,), (S, 1), (1, S), or (S, S) arrays. Sparse scipy.sparse.csr_matrix objects are supported.
• discount: Float ($0 < \text{discount} \le 1$). If set to 1, convergence cannot be assumed.
• epsilon: Float ($> 0$). The stopping criterion for the value function convergence.
• max_iter: Integer ($> 0$). Maximum number of iterations allowed.
• skip_check: Boolean. If True, skips the validation of transitions and reward arrays.

Attributes

• P: The processed transition probability matrices.
• R: The processed reward vectors/matrices.
• V: The optimal value function.
• policy: The optimal policy.
• time: The CPU time used to converge.

Note: MDP.run() is an abstract method and must be implemented by a subclass.

from mdptoolbox.mdp import MDP
import numpy as np

# Example setup (conceptual, as MDP.run() must be overridden)
# transitions = np.random.rand(A, S, S)
# rewards = np.random.rand(S, A)
# mdp = MDP(transitions, rewards, discount=0.9, epsilon=0.01, max_iter=100)

The ValueIteration class solves a discounted MDP by iteratively applying the Bellman operator until an epsilon-optimal policy is found or the maximum number of iterations is reached.

Parameters:

• transitions: Transition probability matrices.
• reward: Reward matrices or vectors.
• discount: Discount factor (float).
• epsilon (float, optional): Stopping criterion. Default is 0.01.
• max_iter (int, optional): Maximum number of iterations. If discount < 1, a bound for max_iter is automatically computed. Default is 1000.
• initial_value (array, optional): The starting value function. Default is a vector of zeros.
• skip_check (bool, optional): If True, skips validation of transitions and reward.

Data Attributes:

• V: The optimal value function.
• policy: The optimal policy function (tuple of integers representing the best action for each state).
• iter: Number of iterations completed.
• time: CPU time used.

Methods:

• run(): Executes the algorithm.
• setSilent(): Disables verbose output.
• setVerbose(): Enables verbose output (displays variation of V per iteration).

import mdptoolbox
import numpy as np

P = np.array([[[0.5, 0.5],[0.8, 0.2]],[[0, 1],[0.1, 0.9]]])
R = np.array([[5, 10], [-1, 2]])

vi = mdptoolbox.mdp.ValueIteration(P, R, 0.9)
vi.run()

print(vi.V)
print(vi.policy)

When working with Markov Decision Processes in this toolbox, you may encounter specific exceptions related to invalid MDP definitions or malformed transition matrices. You can catch these exceptions to handle errors gracefully during your simulations or computations.

Available Exception Classes

All specific error classes (except Error) allow for an optional custom error message. If no message is provided, they use a default error string specific to the error type.

from mdptoolbox.error import InvalidError, StochasticError

try:
    # Example: code that might raise a StochasticError
    run_mdp_simulation()
except StochasticError as e:
    print(f"MDP Error: {e}")
except InvalidError as e:
    print(f"Definition Error: {e}")