DOCSEARCHbyAlgolia
Submit Docs

ArviZ Documentation

repository·main·Indexed 23 days ago

https://github.com/arviz-devs/arviz

ArviZ is a Python package for the exploratory analysis of Bayesian models, providing tools for posterior analysis, data storage, model checking, comparison, and diagnostics. It is a backend-agnostic library that integrates with probabilistic programming libraries including PyMC, Stan, MCX, Pyro, NumPyro, PyJAGS, and TensorFlow Probability. The library is composed of arviz-base, arviz-stats, and arviz-plots, all exposed through the top-level arviz namespace.

Tokens
7.6K
Snippets
13
Records
38
Agent score
83%

What's inside ArviZ

  1. Overview of ArviZ features

    main

    ArviZ is a modular and flexible Python library designed for the exploratory analysis of Bayesian models. It provides robust, interpretable diagnostics and visualizations for the Bayesian workflow.

    Key capabilities include:

    • Interoperability: Integrates with major probabilistic programming libraries such as PyMC, CmdStanPy, Pyro, NumPyro, and emcee.
    • Visualizations: A large suite of over 30 plotting functions for visualizing distributions, MCMC diagnostics, model checking, and model comparison.
    • Diagnostics: Implementation of modern, theory-grounded statistical diagnostics.
    • Model Comparison: Functions for comparing models using both fast approximate cross-validation and brute force methods.
    • Collaboration: Supports cross-language serialization using netCDF or Zarr formats, allowing data to be shared with the Julia version of ArviZ.
    • Labeled Data: Built on top of xarray to support labeled dimensions and coordinates.

  2. Participate in the ArviZ community

    main

    The ArviZ community consists of probabilistic programming practitioners contributing to technical and social infrastructure for open and reproducible research. You can participate by:

    • Using, citing, and sharing ArviZ use cases.
    • Writing posts or giving talks about ArviZ.
    • Participating in GitHub issues to help define the roadmap.
    • Answering questions in affine forums.

    ArviZ aims to act as a bridge between different programming languages and encourages collaboration across the Bayesian ecosystem.

  3. Access ArviZ APIs via the top-level namespace

    main
    ArviZ is composed of three separate libraries: arviz-base, arviz-stats, and arviz-plots. While they are structured as separate packages, they are designed to be used together. For convenience, all functions and classes available in these three libraries are also exposed through the top-level arviz namespace. You can access most objects using the arviz.<object_name> syntax.

  4. Understand the InferenceData schema for ArviZ

    main

    ArviZ organizes Bayesian inference outputs using DataTree objects from the xarray library. This structured, labeled format ensures usefulness in analysis, reproducibility, and interoperability between different inference backends and languages.

    Core Structure

    • Groups: Nodes in the tree representing conceptual quantities (e.g., posterior, observed_data). A group contains one or several variables.
    • Variables: Multidimensional labeled arrays (NetCDF-like) representing specific quantities. Variables and their dimensions must be named.
    • Dimensions: Named axes of a variable (e.g., [chain, draw, dim0]).
    • Coordinates: Named arrays that label a dimension (e.g., a coordinate named chain with values [0, 1, 2, 3]).
    • Attributes: Ordered dictionaries used to store arbitrary metadata at the DataTree, group, or variable level.

  5. Understand the ArviZ modular architecture

    main

    ArviZ has been redesigned from a single package into a modular ecosystem of independent installable packages. This allows users to customize their installation by only including the components they need.

    Key components include:

    • Data Handling: Uses the DataTree class from xarray (replacing the old InferenceData) for flexible, nested, and structured storage of Bayesian inference outputs.
    • Statistical Interfaces:
      • Low-level array interface: Uses numpy and scipy only. Intended for advanced users and library developers.
      • High-level xarray interface: Designed for end users; automates metadata handling and common tasks.
    • Plotting Layers:
      • Batteries-included plots: High-level functions with sensible defaults for MCMC diagnostics, predictive checks, and model comparison.
      • Intermediate API: Uses the PlotCollection class to allow developers to focus on plotting logic while delegating faceting and aesthetics.
      • Low-level engine: Separates computation from plotting to support multiple backends (currently matplotlib, Bokeh, and plotly).

  6. Understand the Bayesian Python ecosystem and ArviZ integrations

    main

    ArviZ serves as the standard library for visualization and diagnostics in Python, acting as a backend-agnostic tool for any Probabilistic Programming Library (PPL).

    PPLs that integrate with ArviZ:

    • PyMC
    • Stan
    • MCX
    • Pyro and NumPyro
    • PyJAGS
    • TensorFlow Probability

    Other compatible libraries:

    Several libraries use ArviZ for visualization/diagnostics or are compatible with InferenceData objects, including:

    • Bambi
    • corner.py

  7. Understand relations between InferenceData groups

    main

    ArviZ uses DataTree objects to store different groups of Bayesian inference data. While a DataTree can contain any subset of these groups, related groups must follow specific relational rules regarding variable names, dimensions, and sample matching.

    Key Group Relationships

    • posterior vs unconstrained_posterior: unconstrained_posterior contains samples in the transformed space. All variables in unconstrained_posterior should have a counterpart in posterior with the same name (though shapes/dimensions may differ). If unconstrained_posterior is present, use it for transformed variables; otherwise, fall back to posterior.
    • posterior vs log_likelihood: log_likelihood samples must match posterior samples. Its variables should match observed_data variables (though names may differ).
    • posterior vs log_prior: log_prior samples must match posterior samples. Its variables should be a subset of or match posterior variables.
    • posterior vs posterior_predictive: posterior_predictive samples must match posterior samples. Its variables should match observed_data variables (though names may differ).
    • observed_data vs posterior_predictive: Every variable in observed_data should have a counterpart in posterior_predictive (names may differ).
    • prior vs posterior: prior samples do not need to match posterior samples, but they must follow the same chain and draw dimension conventions and have matching variable names.
    • prior vs prior_predictive: prior_predictive samples should match prior samples. Each variable should have a counterpart in observed_data or posterior_predictive.
    • predictions vs posterior_predictive: predictions (out-of-sample) samples must match posterior samples. Their variables should have counterparts in posterior_predictive, though coordinate values may differ.

    Prefix Conventions for Specialized Groups

    To support evolving models and samplers, ArviZ uses prefixes to denote specialized data:

    • sample_stats_ prefix: Used for diagnostics related to any group other than the posterior. For example, diagnostics for prior samples should be in sample_stats_prior.
    • warmup_ prefix: Used for samples generated during adaptation/warmup phases. This prefix is applied first (e.g., warmup_posterior or warmup_sample_stats_prior).
    • unconstrained_ prefix: Used for samples in unconstrained space (e.g., unconstrained_prior linked to prior).

  8. Rules for creating compliant InferenceData

    main

    When constructing a DataTree for ArviZ, follow these rules:

    1. Naming: Every variable must be named, and variables must not share names with dimensions.
    2. Dimension Identification: Identify dimensions by name only; the order of dimensions does not matter.
    3. Sample Dimensions:
      • For observed_data or constant_data groups, sample dimensions can be omitted.
      • For prior, posterior, or posterior_predictive groups, you must include either the sample dimension OR both chain and draw dimensions.
    4. Coordinates: Dimensions must share a name with a coordinate that specifies the index values. Coordinate values can be repeated and are not restricted to numerical values.
    5. Metadata: While no metadata is strictly required for compliance, it is highly recommended for interoperability.

  10. Access community educational resources for ArviZ

    main

    Because ArviZ is backend-agnostic, learning often requires looking at resources that combine ArviZ with specific PPLs. Available resources include:

    Books

    Podcasts

    • Learning Bayesian Statistics
    • dats'n'stats

    Videos

    If you maintain a blog with 2 or more posts tagged with "ArviZ", you can submit a pull request to have it included in the official documentation.

  11. Verify your ArviZ installation

    main

    To verify that ArviZ and its sub-components (base, stats, and plots) are correctly installed and accessible, use the az.info attribute. This will print the current version and the status of the internal libraries.

    If you encounter import errors, enable logging to see more detailed information about which component failed to load.

    import arviz as az
print(az.info)
    # Troubleshooting: Enable logging to debug import errors
import logging
logging.basicConfig(level=logging.INFO)
import arviz as az

  12. Sort multidimensional variable dimensions

    main

    If your variables have multiple dimensions (e.g., experiment, subject, date), you can change their display/processing order by using the xarray.Dataset.transpose method on the posterior.

    # Define the desired dimension order
dim_order = ("chain", "draw", "subject", "date", "experiment")

# Transpose the posterior to the new order
experiments = experiments.posterior.transpose(*dim_order)

# Use the reordered object
az.summary(experiments)

# Or use it once without modifying the original object
az.summary(experiments.posterior.transpose(*dim_order))
    dim_order = ("chain", "draw", "subject", "date", "experiment")
experiments = experiments.posterior.transpose(*dim_order)
az.summary(experiments)

# One-time use
az.summary(experiments.posterior.transpose(*dim_order))