API Reference / evaluation / metrics /

clustering.py

Clustering evaluation metrics.

Two kinds of measure live here, and mixing them up is the usual mistake.

External metrics compare a clustering against known ground-truth labels: adjusted_rand_score, rand_score, mutual_info_score, normalized_mutual_info_score, homogeneity_score, completeness_score, v_measure_score and fowlkes_mallows_score. They take (labels_true, labels_pred) and are invariant to how the clusters are named, so permuting labels changes nothing.

Internal metrics judge the clustering from the data geometry alone, with no ground truth: silhouette_score, silhouette_samples, davies_bouldin_score and calinski_harabasz_score. They take (X, labels) and are what you use to pick a number of clusters.

Two directions to watch. davies_bouldin_score is LOWER-is-better, unlike every other metric here. And rand_score and mutual_info_score are uncorrected, so they drift upward as the cluster count grows -- prefer adjusted_rand_score or normalized_mutual_info_score when comparing clusterings of different sizes.

python 
>>> from tuiml.evaluation.metrics import adjusted_rand_score, v_measure_score
>>> true = [0, 0, 1, 1, 2, 2]
>>> pred = [1, 1, 0, 0, 2, 2]          # same partition, different names
>>> adjusted_rand_score(true, pred)
1.0
>>> v_measure_score(true, pred)
1.0

Functions

Func

adjusted_rand_score

Line 52
adjusted_rand_score(labels_true: np.ndarray, labels_pred: np.ndarray) -> float

Compute Adjusted Rand Index (ARI).

The Rand index corrected for chance, using the contingency table n_{ij} with row sums a_i and column sums b_j:

\text{ARI} = \frac{\sum_{ij} \binom{n_{ij}}{2} - \left[\sum_i \binom{a_i}{2} \sum_j \binom{b_j}{2}\right] \Big/ \binom{n}{2}} {\tfrac{1}{2}\left[\sum_i \binom{a_i}{2} + \sum_j \binom{b_j}{2}\right] - \left[\sum_i \binom{a_i}{2} \sum_j \binom{b_j}{2}\right] \Big/ \binom{n}{2}}

Subtracting the expected index makes 0.0 the score of random labelling, so unlike rand_score the value does not drift upward with the number of clusters.

Parameters

labels_true
array-like of shape (n_samples,)
Ground-truth cluster labels.
labels_pred
array-like of shape (n_samples,)
Predicted cluster labels.

Returns

score
float
ARI in the range [-1, 1]. 1.0 is perfect agreement, 0.0 is the value expected from random labelling, and negative values mean the agreement is worse than chance.
python 
>>> from tuiml.evaluation.metrics import adjusted_rand_score
>>> adjusted_rand_score([0, 0, 1, 1], [0, 0, 1, 1])
1.0
>>> round(adjusted_rand_score([0, 0, 1, 1], [0, 1, 0, 1]), 2)
-0.5
Func

rand_score

Line 112
rand_score(labels_true: np.ndarray, labels_pred: np.ndarray) -> float

Compute Rand Index (RI).

The fraction of sample pairs that both labellings agree about, where a counts pairs together in both and b counts pairs apart in both:

\text{RI} = \frac{a + b}{\binom{n}{2}}

Not corrected for chance: random labellings score well above 0. Use adjusted_rand_score when comparing across different cluster counts.

Parameters

labels_true
array-like of shape (n_samples,)
Ground-truth cluster labels.
labels_pred
array-like of shape (n_samples,)
Predicted cluster labels.

Returns

score
float
RI score in the range [0, 1].
python 
>>> from tuiml.evaluation.metrics import rand_score
>>> round(rand_score([0, 0, 1, 1], [0, 0, 1, 1]), 3)
0.333
>>> rand_score([0, 0, 1, 1], [0, 1, 0, 1])
0.0
Func

silhouette_score

Line 169
silhouette_score(X: np.ndarray, labels: np.ndarray, metric: str='euclidean') -> float

Compute mean Silhouette Coefficient.

The mean silhouette over all samples. For one sample, with a(i) its mean distance to its own cluster and b(i) the mean distance to the nearest other cluster:

s(i) = \frac{b(i) - a(i)}{\max\{a(i),\, b(i)\}}

+1 means the sample sits well inside its cluster, 0 means it lies on a boundary, and negative means it is closer to a different cluster.

Parameters

X
array-like of shape (n_samples, n_features)
Feature matrix.
labels
array-like of shape (n_samples,)
Cluster label for each sample.
metric
str = 'euclidean'
Distance metric to use. One of 'euclidean', 'manhattan', or 'cosine'.

Returns

score
float
Mean silhouette score in the range [-1, 1].
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import silhouette_score
>>> X = np.array([[1, 2], [1, 4], [1, 0], [4, 2], [4, 4], [4, 0]])
>>> labels = np.array([0, 0, 0, 1, 1, 1])
>>> round(silhouette_score(X, labels), 3)
0.287
Func

silhouette_samples

Line 255
silhouette_samples(X: np.ndarray, labels: np.ndarray, metric: str='euclidean') -> np.ndarray

Compute Silhouette Coefficient for each sample.

Per-sample version of silhouette_score:

s(i) = \frac{b(i) - a(i)}{\max\{a(i),\, b(i)\}}

Useful for finding which individual points are badly clustered rather than only the overall average.

Parameters

X
array-like of shape (n_samples, n_features)
Feature matrix.
labels
array-like of shape (n_samples,)
Cluster label for each sample.
metric
str = 'euclidean'
Distance metric to use. One of 'euclidean', 'manhattan', or 'cosine'.

Returns

scores
np.ndarray of shape (n_samples,)
Silhouette coefficient for each sample, in the range [-1, 1].
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import silhouette_samples
>>> X = np.array([[1, 2], [1, 4], [1, 0], [4, 2], [4, 4], [4, 0]])
>>> labels = np.array([0, 0, 0, 1, 1, 1])
>>> np.round(silhouette_samples(X, labels), 3)
array([0.412, 0.225, 0.225, 0.412, 0.225, 0.225])
Func

davies_bouldin_score

Line 326
davies_bouldin_score(X: np.ndarray, labels: np.ndarray) -> float

Compute Davies-Bouldin Index.

Lower values indicate better clustering (minimum is 0).

Average over clusters of the worst-case similarity to any other cluster, where s_i is the mean distance from cluster i to its own centroid and d_{ij} the distance between centroids:

\text{DB} = \frac{1}{k} \sum_{i=1}^{k} \max_{j \neq i} \frac{s_i + s_j}{d_{ij}}

LOWER is better, and 0 is the best possible value: the opposite direction to most metrics in this module.

Parameters

X
array-like of shape (n_samples, n_features)
Feature matrix.
labels
array-like of shape (n_samples,)
Cluster label for each sample.

Returns

score
float
Davies-Bouldin score (lower is better).
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import davies_bouldin_score
>>> X = np.array([[1, 2], [1, 4], [1, 0], [4, 2], [4, 4], [4, 0]])
>>> labels = np.array([0, 0, 0, 1, 1, 1])
>>> round(davies_bouldin_score(X, labels), 3)
0.889
Func

calinski_harabasz_score

Line 397
calinski_harabasz_score(X: np.ndarray, labels: np.ndarray) -> float

Compute Calinski-Harabasz Index (Variance Ratio Criterion).

Higher values indicate better clustering.

Ratio of between-cluster to within-cluster dispersion, each corrected for its degrees of freedom:

\text{CH} = \frac{\operatorname{tr}(B_k) \big/ (k - 1)} {\operatorname{tr}(W_k) \big/ (n - k)}

Higher is better. The score is unbounded above, so it is meaningful for ranking candidate cluster counts on one dataset, not across datasets.

Parameters

X
array-like of shape (n_samples, n_features)
Feature matrix.
labels
array-like of shape (n_samples,)
Cluster label for each sample.

Returns

score
float
Calinski-Harabasz score (higher is better).
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import calinski_harabasz_score
>>> X = np.array([[1, 2], [1, 4], [1, 0], [4, 2], [4, 4], [4, 0]])
>>> labels = np.array([0, 0, 0, 1, 1, 1])
>>> round(calinski_harabasz_score(X, labels), 3)
3.375
Func

mutual_info_score

Line 469
mutual_info_score(labels_true: np.ndarray, labels_pred: np.ndarray) -> float

Compute Mutual Information between two clusterings.

How much knowing the cluster tells you about the true class:

\text{MI}(U, V) = \sum_{i}\sum_{j} \frac{n_{ij}}{n} \log \frac{n \, n_{ij}}{a_i b_j}

Measured in nats and unbounded above, which makes raw MI hard to compare; normalized_mutual_info_score rescales it to [0, 1].

Parameters

labels_true
array-like of shape (n_samples,)
Ground-truth cluster labels.
labels_pred
array-like of shape (n_samples,)
Predicted cluster labels.

Returns

score
float
Mutual information score (non-negative, unbounded above).
python 
>>> from tuiml.evaluation.metrics import mutual_info_score
>>> round(mutual_info_score([0, 0, 1, 1], [0, 0, 1, 1]), 3)
0.693
>>> mutual_info_score([0, 0, 1, 1], [0, 1, 0, 1])
0.0
Func

normalized_mutual_info_score

Line 524
normalized_mutual_info_score(labels_true: np.ndarray, labels_pred: np.ndarray, average_method: str='arithmetic') -> float

Compute Normalized Mutual Information (NMI).

Mutual information rescaled by the entropies of the two labellings:

\text{NMI}(U, V) = \frac{\text{MI}(U, V)} {\tfrac{1}{2}\left[H(U) + H(V)\right]}

Bounded in [0, 1], reaching 1.0 exactly when the two labellings agree up to relabelling.

Parameters

labels_true
array-like of shape (n_samples,)
Ground-truth cluster labels.
labels_pred
array-like of shape (n_samples,)
Predicted cluster labels.
average_method
str = 'arithmetic'
Normalizer to divide the mutual information by. One of 'arithmetic', 'geometric', 'min', or 'max', computed from the entropies of labels_true and labels_pred.

Returns

score
float
NMI score in the range [0, 1].
python 
>>> from tuiml.evaluation.metrics import normalized_mutual_info_score
>>> round(normalized_mutual_info_score([0, 0, 1, 1], [0, 0, 1, 1]), 3)
1.0
>>> normalized_mutual_info_score([0, 0, 1, 1], [0, 1, 0, 1])
0.0
Func

v_measure_score

Line 649
v_measure_score(labels_true: np.ndarray, labels_pred: np.ndarray, beta: float=1.0) -> float

Compute V-measure (harmonic mean of homogeneity and completeness).

The harmonic mean of homogeneity h and completeness c:

v = \frac{2 h c}{h + c}

Symmetric in the two labellings, and equal to normalized_mutual_info_score under arithmetic-mean normalization.

Parameters

labels_true
array-like of shape (n_samples,)
Ground-truth class labels.
labels_pred
array-like of shape (n_samples,)
Predicted cluster labels.
beta
float = 1.0
Weight of homogeneity relative to completeness. Values greater than 1.0 favor completeness; values less than 1.0 favor homogeneity.

Returns

score
float
V-measure score in the range [0, 1].
python 
>>> from tuiml.evaluation.metrics import v_measure_score
>>> round(v_measure_score([0, 0, 1, 1], [0, 0, 1, 1]), 3)
1.0
>>> v_measure_score([0, 0, 1, 1], [0, 1, 0, 1])
0.0
Func

homogeneity_score

Line 697
homogeneity_score(labels_true: np.ndarray, labels_pred: np.ndarray) -> float

Compute homogeneity metric (each cluster contains only members of a single class).

Whether each cluster contains only members of a single class:

h = 1 - \frac{H(C \mid K)}{H(C)}

Splitting one true class across many clusters does not hurt this score -- that is what completeness_score measures.

Parameters

labels_true
array-like of shape (n_samples,)
Ground-truth class labels.
labels_pred
array-like of shape (n_samples,)
Predicted cluster labels.

Returns

score
float
Homogeneity score in the range [0, 1].
python 
>>> from tuiml.evaluation.metrics import homogeneity_score
>>> round(homogeneity_score([0, 0, 1, 1], [0, 0, 1, 1]), 3)
1.0
>>> homogeneity_score([0, 0, 1, 1], [0, 1, 0, 1])
0.0
Func

completeness_score

Line 737
completeness_score(labels_true: np.ndarray, labels_pred: np.ndarray) -> float

Compute completeness metric (all members of a class are in the same cluster).

Whether all members of a class land in the same cluster:

c = 1 - \frac{H(K \mid C)}{H(K)}

The mirror image of homogeneity_score; putting everything in one cluster scores 1.0 here and poorly there.

Parameters

labels_true
array-like of shape (n_samples,)
Ground-truth class labels.
labels_pred
array-like of shape (n_samples,)
Predicted cluster labels.

Returns

score
float
Completeness score in the range [0, 1].
python 
>>> from tuiml.evaluation.metrics import completeness_score
>>> round(completeness_score([0, 0, 1, 1], [0, 0, 1, 1]), 3)
1.0
>>> completeness_score([0, 0, 1, 1], [0, 1, 0, 1])
0.0
Func

fowlkes_mallows_score

Line 777
fowlkes_mallows_score(labels_true: np.ndarray, labels_pred: np.ndarray) -> float

Compute Fowlkes-Mallows Index.

The geometric mean of the pairwise precision and recall, counting pairs of samples placed in the same cluster:

\text{FMI} = \frac{\text{TP}} {\sqrt{(\text{TP} + \text{FP})(\text{TP} + \text{FN})}}

Bounded in [0, 1]; unlike rand_score it ignores the true-negative pairs that dominate when there are many clusters.

Parameters

labels_true
array-like of shape (n_samples,)
Ground-truth cluster labels.
labels_pred
array-like of shape (n_samples,)
Predicted cluster labels.

Returns

score
float
Fowlkes-Mallows index in the range [0, 1].
python 
>>> from tuiml.evaluation.metrics import fowlkes_mallows_score
>>> fowlkes_mallows_score([0, 0, 1, 1], [0, 0, 1, 1])
1.0
>>> fowlkes_mallows_score([0, 0, 1, 1], [0, 1, 0, 1])
0.0