POMDPs.jl

The POMDPs.jl ecosystem consists of several specialized packages that implement the core interface. Depending on your needs, you can choose from tools for belief filtering, pre-implemented models, MDP/POMDP solvers, or Reinforcement Learning algorithms.

Tools

Used for implementing commonly-used components like filters:

• POMDPTools: Core interface components.
• ParticleFilters: Particle filter implementations.
• GaussianFilters: Gaussian filter implementations.

Implemented Models

Commonly used POMDP models implemented via the interface:

• POMDPModels, LaserTag, RockSample, TagPOMDPProblem, DroneSurveillance, ContinuumWorld, VDPTag2, RoombaPOMDPs.

Solvers

MDP Solvers

• DiscreteValueIteration: Offline, discrete states/actions (High reliability).
• LocalApproximationValueIteration: Offline, continuous states, discrete actions.
• GlobalApproximationValueIteration: Offline, continuous states, discrete actions.
• MCTS: Online, continuous states/actions (Monte Carlo Tree Search).

POMDP Solvers

• QMDP: Offline, discrete (Suboptimal).
• AdaOPS: Online, continuous states/observations.
• ARDESPOT: Online, continuous states.
• BasicPOMCP: Online, continuous states.
• CompressedBeliefMDPs: Offline, continuous states/actions/observations.
• NativeSARSOP: Offline, discrete.
• SARSOP: Offline, discrete.
• POMCPOW: Online, continuous states/actions.
• ParticleFilterTrees: Online, continuous states/actions.

Reinforcement Learning

• TabularTDLearning: Discrete states/actions.
• DeepQLearning: Continuous states (uses observations as input).
• Crux: Continuous states and actions.

The distinction between these functions depends on whether you are implementing a problem (the model) or a solver/simulator.

For Problem Implementers

• transition: Implement this to define the state transition distribution. You can provide an explicit distribution or use an ImplicitDistribution if you only have a sampler.
• gen: Implement this only if your simulator outputs multiple values (next state, observation, and reward) simultaneously (e.g., when rewards are tied to the movement between states).
• @gen: Do not implement or modify this. It is a tool used by solvers and simulators.

For Solver/Simulator Implementers

• @gen: Call this to sample the next state, observation, and/or reward. It automatically orchestrates calls to rand, transition, observation, reward, and gen without overhead.
• transition: Call this only when you need the explicit transition probability distribution.
• gen: Do not call this directly. It is intended for problem implementers to use when defining joint distributions.

A distribution object represents a probability distribution used within a POMDP (e.g., for transition or observation models). To ensure compatibility with solvers, your distribution object should implement the following functions:

• rand([rng,] d): Samples from the distribution. Note: It is highly recommended to implement Base.rand(rng::AbstractRNG, s::Random.SamplerTrivial{<:YourDistribution}) to support both rand(d) and rand(rng, d) via the standard Julia rand interface.
• support(d): Returns the set of values for which the distribution is defined.
• pdf(d, x): Returns the probability density/mass function at x.
• mode(d): Returns the mode of the distribution.
• mean(d): Returns the mean of the distribution.

For pre-made distributions, you can use Distributions.jl or POMDPTools.

To implement a new solver, you must provide four distinct components that work together to bridge the gap between problem definition and action selection:

1. A Solver type: A subtype of Solver that stores the solver's configuration parameters and options.
2. A Policy type: A subtype of Policy that holds the data/structures required to select actions (e.g., a value function for offline solvers or a search tree for online solvers).
3. A solve method: A function with the signature solve(solver, problem) that performs the offline computation and returns the Policy object.
4. An action method: A function with the signature action(policy, state_or_belief) that returns an action based on the current state or belief.

Tip: You can often reuse existing Policy types from the POMDPTools package to satisfy requirements 2 and 4, allowing you to focus on the solve logic.

# Conceptual structure of a solver
# 1. Solver type (configuration)
# 2. Policy type (data for action selection)
# 3. solve(solver, problem) -> policy (offline work)
# 4. action(policy, state_or_belief) -> action (online work)

POMDPTools provides interface extensions that act as shortcuts for common tasks. While these have default implementations based on the core POMDPs.jl interface, you should use them whenever possible because they allow for optimized implementations if available.

Key extensions include:

• Weighted Iteration: Use weighted_iterator to iterate through value => probability pairs of a distribution. This is more efficient than looking up probability mass for every value when the distribution support is large.
• Observation Weight: Use obs_weight to get the relative likelihood of an observation. This is useful for algorithms like particle filtering where you need the likelihood without implementing a full custom observation distribution.
• Ordered Spaces: Since POMDPs.jl does not guarantee that spaces are ordered consistently with their index function, use ordered_actions, ordered_states, or ordered_observations to get lists that are guaranteed to match the indexing order.

In POMDPs.jl, states, actions, and observations can be any Julia object (integers, strings, vectors, etc.).

Best Practices

• Immutability: Use immutable objects (like Int, Float64, or StaticArrays) for states, actions, and observations. They are often used as dictionary keys or in histories. If you must use mutable objects, ensure hash and == are correctly implemented (e.g., using AutoHashEquals.jl).
• Explicit vs. Implicit Spaces: You can define spaces using the states, actions, or observations keyword arguments with a Vector. However, for continuous or hybrid spaces, you often only need to specify the type using statetype or obstype (e.g., obstype = Float64). Many solvers do not require an explicit enumeration of the space.

State- or Belief-dependent Action Spaces

If the available actions depend on the current state (MDP) or belief (POMDP), pass a function to the actions argument instead of a vector.

For MDPs (state-dependent):

actions = function (s = nothing) 
    if s == 1
        return [1]
    else
        return [1,2,3]
    end
end

For POMDPs (belief-dependent):

actions = function (b)
    if pdf(b, 1) > 0.0
        return [1,2,3]
    else
        return [2,3]
    end
end