API Reference / algorithms / clustering /

gaussian_mixture.py

Expectation-Maximization clustering for Gaussian Mixture Models (GMM).

Classes

GaussianMixtureClusterer

class algorithms.clustering.gaussian_mixture.GaussianMixtureClusterer(DensityBasedClusterer)

Expectation-Maximization clustering for Gaussian Mixture Models.

Models the data as a mixture of K Gaussian distributions. Each component is characterized by its mean \mu_k, covariance \Sigma_k, and mixing weight \pi_k. Unlike hard-assignment methods like K-Means, EM provides soft probabilistic cluster assignments.
Constructor 
__init__(
    self,
    n_components: int = 2,
    max_iter: int = 200,
    tol: float = 1e-06,
    covariance_type: str = 'full',
    n_init: int = 10,
    random_state: Optional[int] = None,
)

Overview

The algorithm iteratively refines component parameters:

  1. Initialize means, covariances, and mixing weights
  2. E-step: Compute responsibilities (posterior probability that each
component generated each data point)
  1. M-step: Update parameters to maximize expected log-likelihood
  2. Repeat steps 2--3 until convergence or max_iter is reached
  3. Select the best result across n_init independent runs

Theory

The probability of an observation x under the mixture model is:

P(x) = \sum_{k=1}^{K} \pi_k \, \mathcal{N}(x \mid \mu_k, \Sigma_k)

E-step computes the responsibility of component k for point x_i:

\gamma_{ik} = \frac{\pi_k \, \mathcal{N}(x_i \mid \mu_k, \Sigma_k)}{\sum_{j=1}^{K} \pi_j \, \mathcal{N}(x_i \mid \mu_j, \Sigma_j)}

M-step updates the parameters:

\pi_k = \frac{N_k}{N}, \quad \mu_k = \frac{1}{N_k} \sum_{i} \gamma_{ik} x_i, \quad \Sigma_k = \frac{1}{N_k} \sum_{i} \gamma_{ik} (x_i - \mu_k)(x_i - \mu_k)^T

where N_k = \sum_i \gamma_{ik}.

Parameters

n_components
int = 2
The number of mixture components.
max_iter
int = 200
The maximum number of EM iterations to perform.
tol
float = 1e-6
The convergence threshold. EM iterations will stop when the lower bound average gain is below this threshold.
covariance_type
{"full", "diag", "spherical"} = "full"

String describing the type of covariance parameters to use:

  • •"full": Each component has its own general covariance matrix.
  • •"diag": Each component has its own diagonal covariance matrix.
  • •"spherical": Each component has its own single variance.
n_init
int = 10
The number of initializations to perform. The best results are kept.
random_state
int = None
Controls the random seed given to the method chosen to initialize the parameters.

Attributes

weights_
np.ndarray of shape (n_components,)
The weights of each mixture component.
means_
np.ndarray of shape (n_components, n_features)
The mean of each mixture component.
covariances_
np.ndarray
The covariance of each mixture component.
converged_
bool
True when convergence was reached in fit(), False otherwise.
n_iter_
int
Number of iterations used by the best fit of EM to reach the convergence.
log_likelihood_
float
Log-likelihood of the best fit of EM.

Notes

Complexity:

  • Training: O(n \cdot k \cdot m^2 \cdot i) for "full" covariance,
where n is samples, k is components, m is features, and i is iterations.
  • For "diag": O(n \cdot k \cdot m \cdot i).
  • For "spherical": O(n \cdot k \cdot i).
When to use GaussianMixtureClusterer:
  • When clusters have elliptical shapes of varying sizes and orientations
  • When soft (probabilistic) cluster assignments are needed
  • Density estimation and generative modeling
  • When the data is well-described by Gaussian distributions

References

Dempster1977
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1), pp. 1-22.
McLachlan2000
McLachlan, G. J. & Peel, D. (2000). Finite Mixture Models. Wiley Series in Probability and Statistics.

See Also

KMeansClusterer

Hard-assignment centroid-based clustering.
DBSCANClusterer

Density-based clustering with noise detection.

Gaussian mixture model clustering:

python 
>>> import numpy as np
>>> from tuiml.algorithms.clustering import GaussianMixtureClusterer
>>> X = np.array([[1, 2], [1, 4], [1, 0],
...               [10, 2], [10, 4], [10, 0]])
>>> gmm = GaussianMixtureClusterer(n_components=2)
>>> gmm.fit(X)
>>> gmm.predict([[0, 0], [12, 3]])
array([0, 1])

Methods

get_parameter_schema (cls) -> Dict[str, Dict[str, Any]]

Return parameter schema.

get_capabilities (cls) -> List[str]

Return algorithm capabilities.

get_complexity (cls) -> str

Return time/space complexity.

get_references (cls) -> List[str]

Return academic references.

fit (self, X: np.ndarray) -> 'GaussianMixtureClusterer'

Estimate model parameters with the EM algorithm.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Training data to cluster.
Returns
self
GaussianMixtureClusterer
Fitted estimator.
predict (self, X: np.ndarray) -> np.ndarray

Predict the labels for the data samples in X using trained model.

Parameters
X
np.ndarray of shape (n_samples, n_features)
New data to predict.
Returns
labels
np.ndarray of shape (n_samples,)
Component labels.
predict_proba (self, X: np.ndarray) -> np.ndarray

Predict posterior probability of each component given the data.

Parameters
X
np.ndarray of shape (n_samples, n_features)
New data to predict.
Returns
resp
np.ndarray of shape (n_samples, n_components)
Posterior probability of each Gaussian component for each sample in X.
score (self, X: np.ndarray) -> float

Compute the per-sample average log-likelihood of the given data X.

Parameters
X
np.ndarray of shape (n_samples, n_features)
New data to score.
Returns
score
float
Log-likelihood of X under the Gaussian mixture model.
__repr__ (self) -> str

String representation.