hdbscan Documentation

HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) is an unsupervised machine learning algorithm for clustering. Unlike DBSCAN, which uses a single epsilon value, HDBSCAN performs DBSCAN over varying epsilon values and integrates the results to find a clustering that provides the best stability. This enables the algorithm to identify clusters of varying densities and increases robustness to parameter selection.

Key features include:

• Clustering: Finds clusters of varying densities.
• Robust Single Linkage: Support for robust single linkage clustering.
• Outlier Detection: Includes GLOSH (Global-Local Outlier Score from Hierarchies) outlier detection.
• Cluster Exploration: Tools for visualizing and exploring cluster structures.
• Advanced Clustering: Support for prediction and soft clustering.

DBSCAN extracts dense clusters and classifies sparse background points as 'noise'. It requires tuning eps (epsilon, a distance threshold) and min_samples.

Limitations:

• eps can be difficult to choose and the algorithm is sensitive to it.
• It struggles with datasets containing clusters of varying densities; it may miss sparse clusters or lump dense clusters together depending on the parameters.

plot_clusters(data, cluster.DBSCAN, (), {'eps':0.025})

The alpha parameter (default 1.0) provides an alternative way to control how conservative the clustering is.

• Increasing alpha: Makes the clustering more conservative on a tight scale.

Note: It is recommended to prioritize tuning min_samples or cluster_selection_epsilon before adjusting alpha. Adjusting alpha requires recomputing the single linkage tree.

HDBSCAN is an evolution of DBSCAN designed to handle clusters of varying densities. It eliminates the need for an eps parameter by using a condensed tree approach to find stable clusters.

Key Parameters:

• min_cluster_size: The minimum number of points to consider a group a cluster. This is considered an intuitive parameter for EDA.
• min_samples: Inherited from DBSCAN for density-based space transformation. While less intuitive, the algorithm is generally not highly sensitive to it.

Advantages:

• Handles varying densities.
• Identifies noise.
• Stable across runs and subsampling.
• High performance (can utilize fastcluster if available).

import hdbscan

plot_clusters(data, hdbscan.HDBSCAN, (), {'min_cluster_size':15})

To compare clustering algorithms, you need numpy, matplotlib, seaborn, and scikit-learn. This setup includes configuring seaborn for better visualization and defining standard plotting keyword arguments.

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import sklearn.cluster as cluster
import time
%matplotlib inline
sns.set_context('poster')
sns.set_color_codes()
plot_kwds = {'alpha' : 0.25, 's' : 80, 'linewidths':0}

You can identify 'mixed' points (points that have a high likelihood of belonging to two different clusters) by analyzing the membership vectors. A common approach is to find points where the difference between the top two probabilities is very small, but the total sum of probabilities is high.

import numpy as np

def top_two_probs_diff(probs):
    sorted_probs = np.sort(probs)
    return sorted_probs[-1] - sorted_probs[-2]

# Compute the differences between the top two probabilities
diffs = np.array([top_two_probs_diff(x) for x in soft_clusters])

# Select indices with a small difference and a large total probability
mixed_points = np.where((diffs < 0.001) & (np.sum(soft_clusters, axis=1) > 0.5))[0]

You can use the BranchDetector class to find branches within clusters that were not generated by HDBSCAN (for example, labels from DBSCAN). To do this, pass the HDBSCAN clusterer object and your custom labels to the .fit() method.

Requirement: Custom clusters must be within-cluster-path-connected in HDBSCAN's Minimum Spanning Tree (MST). This means the MST edges between all points in the cluster must form a single connected component. If the custom clusters break this condition, the detector will return MST connected component labels instead of branches.

# Valid option: merging clusters manually to find new branching structures
custom_labels = clusterer.labels_.copy()
custom_labels[clusterer.labels_ == 3] = 2
branch_detector.fit(clusterer, custom_labels)
plot(branch_detector.labels_)

HDBSCAN follows the scikit-learn API. To cluster data, instantiate an hdbscan.HDBSCAN object and call .fit(data).

Results are stored in the .labels_ attribute:

• Integers starting from 0 represent cluster assignments.
• The value -1 indicates noise (samples not assigned to any cluster).

Additionally, you can access soft clustering membership scores via the .probabilities_ attribute. These scores range from 0.0 (noise) to 1.0 (core of the cluster).