missingno Documentation

missingno is a Python package designed to visualize missing data (nullity) in tabular datasets. It converts data matrices into boolean masks (nullity matrices) to reveal patterns of missingness within and between columns. The package provides several specialized visualization tools:

• Bar Chart: Provides a snapshot of column-level missingness information.
• Matrix: A literal translation of the data table's nullity matrix, useful for spotting general patterns.
• Heatmap: Used to examine relationships and correlations within pairs of variables.
• Dendrogram: A hierarchically clustered dendrogram for understanding nullity correlations in higher-cardinality data.
• Geospatial Visualizations: Uses quadtree or convex hull algorithms to view geospatial data dependencies.

To get started with missingno for visualizing missing data in your datasets, install it using pip:

pip install missingno

To perform advanced customization using matplotlib, set inline=False in your missingno call. This causes the method to return the underlying matplotlib.axis.Axis object(s) instead of plotting. You can then access the axes and apply standard matplotlib operations.

# Example: Increasing y-axis label size for a dendrogram
mat = msno.dendrogram(collisions, inline=False)
mat.axes[0].tick_params(axis='y', labelsize=30)

The msno.bar function provides a simple bar chart visualization of nullity per column. It is a simpler format compared to the matrix view.

Options:

• log=True: Switch to a logarithmic scale for the bars.

The msno.matrix function creates a data-dense nullity matrix that allows you to visually identify patterns in data completion. It includes a sparkline on the right to summarize the general shape of data completeness and highlight rows with maximum/minimum nullity.

Key Features:

• Accommodates up to 50 labeled variables comfortably.
• Supports time-series data by specifying a periodicity using the freq keyword parameter (using pandas offset aliases).

The msno.heatmap function measures nullity correlation, showing how strongly the presence or absence of one variable affects another.

Correlation Scale:

• -1: One variable appears, the other definitely does not.
• 0: Variables appearing/not appearing have no effect on each other.
• 1: If one variable appears, the other definitely does too.

Note: Variables that are always full or always empty are automatically removed from the visualization as they have no meaningful correlation.

The msno.dendrogram function uses a hierarchical clustering algorithm (via scipy) to reveal deeper trends in variable completion than pairwise correlation heatmaps. It bins variables against one another based on nullity correlation (binary distance).

Interpretation:

• Top-down reading: Cluster leaves linked at a distance of zero fully predict one another's presence.
• Cluster height: For leaves that split close to zero but not at it, the height indicates how often records are "mismatched" or incorrectly filed.
• Large datasets: For extremely large datasets, the dendrogram automatically switches to a horizontal configuration.

Use msno.nullity_sort() to reshuffle rows by their completeness level. This is primarily used to improve the visual structure of matrix plots. It does not modify the underlying data.

• sort: Either 'ascending' or 'descending'.

# Example: Sort data by completeness in descending order
sorted_data = msno.nullity_sort(data, sort='descending')
msno.matrix(sorted_data.sample(250))

Use msno.nullity_filter() to drill down into large datasets by filtering columns based on their completeness.

• filter: Either 'top' (columns with at least p completeness) or 'bottom' (columns with at most p completeness).
• n: The maximum number of columns to return.
• p: The percentage cutoff (e.g., 0.9 for 90%).

# Example: Filter to at most 15 columns that are at most 99.9% complete
filtered_data = msno.nullity_filter(data, filter='bottom', n=15, p=0.999)
msno.matrix(filtered_data.sample(250))

Use these parameters to customize the msno.dendrogram() output:

• orientation: The orientation of the dendrogram. Defaults to top if $\le 50$ columns and left if $> 50$.
• method: The scipy.hierarchy linkage method used for clustering. Defaults to average.

The following parameters are available across matrix, bar, heatmap, dendrogram, and geoplot:

• figsize: The matplotlib figure size (default: (20, 12), though dendrogram may compute height dynamically).
• fontsize: The figure's font size (default: 16).
• labels: Boolean to display/hide column names. Defaults to True for dendrogram and heatmap. For matrix, it defaults to True if variables $\le 50$ and False if $> 50$.
• inline: If True (default), the chart is plotted immediately. If False, the method returns the visualization object instead of plotting.