DOCSEARCHbyAlgolia
Submit Docs

KeplerMapper Documentation

repository·master·Indexed 20 days ago

https://github.com/scikit-tda/kepler-mapper

A Python implementation of the Topological Data Analysis (TDA) Mapper algorithm for visualizing high-dimensional data structures. Compatible with the scikit-learn API, it allows users to project data using custom reducers (like t-SNE), generate topological graphs via the `map` method, and create interactive HTML visualizations with features like custom tooltips, dynamic min-intersection adjustment, and node repositioning. It can be used for anomaly detection, NLP tasks, and creating confidence graphs to represent model uncertainty.

Tokens
14.9K
Snippets
35
Records
70
Agent score
70%

What's inside KeplerMapper

  1. Overview of KeplerMapper

    master
    KeplerMapper is a Python library that implements the Mapper algorithm. It is primarily used for the visualization of high-dimensional data and 3D point cloud data. The library is designed to be compatible with Scikit-Learn, allowing you to use Scikit-Learn API compatible clustering and scaling algorithms as part of the Mapper process.

  2. What is the Mapper algorithm?

    master

    The MAPPER algorithm transforms high-dimensional data (such as point clouds) or function outputs (such as similarity measures) into a graph or simplicial complex. This graph serves as a compressed, meaningful summary of the dataset.

    How it works:

    1. Projection: The data is projected using a filter function.
    2. Covering: The filter function's range is covered by overlapping intervals (bins).
    3. Clustering: Points falling within each bin are clustered to form the nodes of the graph.
    4. Edges: An edge (vertex connection) is drawn between two nodes if a single data point appears in both nodes. This connectivity is enabled by the overlap of the bins.

  3. What is a Self-Guessing mapper?

    master

    Self-guessing is a concept where a learning set must be self-consistent: if a subset of the learning set is fed to a generalizer, the generalizer must be able to correctly guess the remaining parts of the learning set.

    In the context of this project, it relates to using models (like texture, rotation, or shape models) to impute non-visible or missing parts of data based on the visible parts, similar to how humans infer obscured parts of an image.

  4. What is Kepler Mapper and how does it work?

    master

    Kepler Mapper is a Python implementation of the Mapper algorithm used in Topological Data Analysis (TDA) to visualize the shape and structure of complex, high-dimensional data. It produces a network graph that represents topological descriptors like connected components, loops, and voids.

    The library uses an object-oriented API based on the Strategy Pattern, allowing users to swap out different components for the following three stages of the Mapper process:

    1. Clustering Strategy: Determines how data points are grouped. Kepler Mapper leverages Scikit-Learn-compatible clustering algorithms.
    2. Covering Scheme (Cover classes): Defines how the data space is partitioned into overlapping patches.
    3. Nerve Scheme (Nerve classes): Defines how the intersections of those patches are used to construct the network graph.

    By providing these interchangeable strategies, Kepler Mapper allows for highly flexible construction of Mapper network graphs tailored to specific datasets.

  6. Approximating Kolmogorov Complexity through compression

    master

    Kolmogorov Complexity ($K$) is the length of the shortest program that produces a string. Since $K$ is uncomputable, it can be approximated using real-world compression algorithms ($K'$):

    K'(x) = len(compress(x)) = Z(x)

    Where $Z(x)$ is the length of the compressed string. This approximation allows for ranking sequence continuations or measuring similarity between objects using the Normalized Compression Distance (NCD):

    NCD = (Z(x, y) - min{Z(x), Z(y)}) / max{Z(x), Z(y)}

    Where $Z(x, y)$ is the length of compressing the concatenation of $x$ and $y$.

  7. Link local clusters using link_local=True

    master
    To connect clusters that reside within the same hypercube, set the link_local parameter to True during initialization. This can result in a more cohesive topological representation, such as turning fragmented clusters into a continuous structure (e.g., transforming a low-resolution cat topology from disconnected components into a linked structure).

  8. Concept: The Self-Guessing Mapper framework

    master

    The Self-Guessing Mapper is a conceptual framework that uses KeplerMapper and filter functions to act as generalizers. It aims to build a model of perception by attempting to reconstruct original data points from a compressed representation.

    Core Workflow

    1. Space Compression: A set of filter functions {f(X)n} is selected to map high-dimensional data X into a lower-dimensional space. The goal is to find a set that maximizes accuracy while minimizing a cost function K (balancing complexity and accuracy).
    2. Filter Library: The framework leverages KeplerMapper's built-in projection capabilities, which include:
      • Subselected columns.
      • Statistical functions (mean, max, min, std).
      • Distance metrics and distance matrices.
      • Unsupervised dimensionality reduction (e.g., UMAP, t-SNE, Neural Gas) via the Scikit-learn API.
      • Supervised algorithms (e.g., XGBoost, Keras, KNN) via the Scikit-learn API.
    3. Ensemble Selection: Filter functions are ranked by accuracy and complexity, then combined into a stacked ensemble using forward selection based on local AUC evaluation.
    4. Self-Mapping: For each filter function, data is projected and covered with overlapping intervals. A self-supervised classifier (like a Decision Tree or MLP) is used to predict the original data points inside the interval using the projection values outside the interval as features.
    5. Reconstruction: KeplerMapper is used to reconstruct the predictions into a simplicial complex, which can be used to impute missing data or reconstruct Betti numbers.

  9. Use KeplerMapper for data projection and feature engineering

    master

    In the context of a self-guessing framework, KeplerMapper can be used to build a library of filter functions for data projection. This allows you to transform high-dimensional data into lower-dimensional features that can be used for modeling.

    Key projection capabilities include:

    • Subselected columns: Accessing specific data dimensions (e.g., specific axes or features).
    • Statistical functions: Applying transformations like mean, max, min, or std.
    • Distance metrics: Using various distance metrics or converting data into a distance matrix.
    • Unsupervised dimensionality reduction: Leveraging any Scikit-learn compatible algorithm such as UMAP, t-SNE, or neural gas.
    • Supervised algorithms: Utilizing Scikit-learn compatible supervised models like XGBoost, Keras, or KNN as part of the projection/filtering process.

  10. Understanding the Self-guessing mapper theory and limitations

    master

    The Self-guessing mapper is an experimental research-level implementation inspired by Algorithmic Information Theory (AIT). It attempts to find optimal filter functions by minimizing complexity.

    Key Concepts

    • Complexity Metrics: The approach discusses using Program Length (the complexity of the code/program) and Reference Length (the complexity of communicating a reference to an existing implementation, e.g., using scikit-learn instead of hand-coding an MLP) to rank model complexity.
    • Multi-scale mapping: The framework can map objects using multiple differently sized intervals/resolutions to capture both global complexity (low-dimensional) and fine-grained complexity (high-dimensional).

    Known Limitations and Drawbacks

    • Data Pre-processing Sensitivity: The mapper can be sensitive to data that is already pre-centered or pre-normalized. For example, using an $L_2$-norm as a filter function to describe a circle requires the circle to be centered at the origin.
    • Experimental Nature: The current implementation is considered highly experimental research-level code and may contain bugs or inefficiencies.
    • Segment-and-Center Problem: To model human perceptual reasoning, the framework would need to address how to segment and center objects within a focus area before applying filter functions.

  11. Configure colors for samples and nodes in HTML visualizations

    master

    When using kmapper.KeplerMapper.visualize, you can use color_values and colorscale to map sample values to colors via min-max scaling. These color values drive two visual components:

    1. Histogram distributions: The 'details' pane for a node shows a histogram of the color values of the samples within that node.
    2. Node colors: Each node is assigned a color based on its samples' color values, determined by the node_color_function.

    If you provide multiple sets of color_values or node_color_functions, the HTML visualization will include dropdown menus allowing users to toggle between them.