API Reference / algorithms / bayesian /

gaussian_processes.py

Gaussian Process regression implementation.

Classes

GaussianProcessesRegressor

class algorithms.bayesian.gaussian_processes.GaussianProcessesRegressor(Regressor)

Gaussian Process Regression (GPR) with uncertainty estimation.

Gaussian Processes are a non-parametric Bayesian approach to regression. They provide a probabilistic model that not only makes predictions but also estimates the predictive variance (uncertainty) of those predictions.
Constructor 
__init__(
    self,
    kernel: str = 'rbf',
    gamma: Any = 'scale',
    degree: int = 2,
    noise: float = 1.0,
    normalize: bool = True,
)

Overview

The algorithm performs regression through the following steps:

  1. Optionally normalise features and targets to zero mean, unit variance
  2. Compute the kernel matrix K between all training points
  3. Add noise to the diagonal for regularisation: K + \sigma_n^2 I
  4. Solve for dual coefficients via Cholesky decomposition
  5. At prediction time, compute the predictive mean and variance
using the kernel between test and training points

Theory

A Gaussian Process defines a distribution over functions:

f(\mathbf{x}) \sim \mathcal{GP}\bigl(m(\mathbf{x}),\, k(\mathbf{x}, \mathbf{x}')\bigr)

Given training data (X, \mathbf{y}) with noise \sigma_n^2, the predictive distribution at test points X_* is Gaussian:

\bar{f}_* &= K_*^\top (K + \sigma_n^2 I)^{-1} \mathbf{y} \ \text{cov}(f_*) &= K_{**} - K_*^\top (K + \sigma_n^2 I)^{-1} K_*

The log marginal likelihood used for model comparison is:

\log p(\mathbf{y} \mid X) = -\tfrac{1}{2} \mathbf{y}^\top K_y^{-1} \mathbf{y} - \tfrac{1}{2} \log |K_y| - \tfrac{n}{2} \log 2\pi

where K_y = K + \sigma_n^2 I.

Parameters

kernel
str = 'rbf'

The kernel function to use:

  • 'rbf' - Radial Basis Function (Squared Exponential)
  • 'poly' - Polynomial kernel
  • 'linear' - Linear (dot product) kernel
gamma
float or "scale" = "scale"
Kernel coefficient for 'rbf' and 'poly'. If "scale", uses 1.0 / (n_features * X.var()).
degree
int = 2
The degree of the polynomial kernel. Ignored for other kernels.
noise
float = 1.0
Noise level (Tikhonov regularisation). Added to the diagonal of the kernel matrix to account for noise in the observations and for numerical stability. Larger values increase smoothing.
normalize
bool = True
Whether to normalise the features and target variables to zero mean and unit variance before fitting. Highly recommended for GPR.

Attributes

alpha_
np.ndarray
Dual coefficients (weights) assigned to each training sample.
L_
np.ndarray
Lower triangular Cholesky factor of the kernel matrix.
X_train_
np.ndarray
Features used during training (normalised if normalize=True).
y_train_
np.ndarray
Targets used during training (normalised if normalize=True).
y_mean_
float
The mean value of the training targets.
y_std_
float
The standard deviation of the training targets.

Notes

Complexity:

  • Training: O(n^3) due to Cholesky decomposition of the n \times n kernel matrix
  • Prediction: O(n^2) per test point for the mean; O(n^2) for the variance
  • Memory: O(n^2) for storing the kernel matrix
When to use GaussianProcessesRegressor:
  • When you need calibrated uncertainty estimates alongside predictions
  • Small-to-medium datasets (up to a few thousand samples)
  • When the relationship between features and target is smooth
  • Bayesian optimisation and active learning settings
  • When model selection via the log marginal likelihood is desirable

References

Rasmussen2006
Rasmussen, C.E. and Williams, C.K.I. (2006). Gaussian Processes for Machine Learning. MIT Press.
Scholkopf2002
Scholkopf, B. and Smola, A.J. (2002). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press.

See Also

NaiveBayesClassifier

Probabilistic classifier using Bayes' theorem.

Basic regression with RBF kernel:

python 
>>> from tuiml.algorithms.bayesian import GaussianProcessesRegressor
>>> import numpy as np
>>>
>>> # Create sinusoidal training data
>>> X = np.array([[1], [3], [5], [7], [9]])
>>> y = np.sin(X).ravel()
>>>
>>> # Fit the Gaussian Process model
>>> gp = GaussianProcessesRegressor(kernel='rbf', noise=0.1)
>>> gp.fit(X, y)
GaussianProcessesRegressor(kernel='rbf', noise=0.1, n_train=5)
>>>
>>> # Predict with uncertainty
>>> mean, std = gp.predict([[4]], return_std=True)

Methods

get_parameter_schema (cls) -> Dict[str, Dict[str, Any]]

Return parameter schema.

get_capabilities (cls) -> List[str]

Return algorithm capabilities.

get_complexity (cls) -> str

Return time/space complexity.

get_references (cls) -> List[str]

Return academic references.

fit (self, X: np.ndarray, y: np.ndarray) -> 'GaussianProcessesRegressor'

Fit the Gaussian Process regression model.

Parameters
X
array-like of shape (n_samples, n_features)
Training data.
y
array-like of shape (n_samples,)
Target values.
Returns
self
GaussianProcessesRegressor
Returns the fitted estimator.
predict (self, X: np.ndarray, return_std: bool=False) -> Union[np.ndarray, Tuple[np.ndarray, np.ndarray]]

Predict using the Gaussian Process regression model.

Parameters
X
array-like of shape (n_samples, n_features)
Query points where the GP is evaluated.
return_std
bool = False
Whether or not to return the standard deviation of the predictive distribution at the query points.
Returns
y_mean
np.ndarray of shape (n_samples,)
Mean of predictive distribution at query points.
y_std
np.ndarray of shape (n_samples,)
Standard deviation of predictive distribution at query points. Only returned if return_std is True.
predict_proba (self, X: np.ndarray) -> Tuple[np.ndarray, np.ndarray]

Predict with uncertainty estimates.

Parameters
X
array-like of shape (n_samples, n_features)
Test features.
Returns
mean
np.ndarray of shape (n_samples,)
Predicted mean at each query point.
std
np.ndarray of shape (n_samples,)
Predicted standard deviation at each query point.
score (self, X: np.ndarray, y: np.ndarray) -> float

Compute the R-squared (coefficient of determination) score.

Parameters
X
array-like of shape (n_samples, n_features)
Test features.
y
array-like of shape (n_samples,)
True target values.
Returns
r2
float
R-squared score. Best possible score is 1.0; it can be negative if the model is worse than predicting the mean.
log_marginal_likelihood (self) -> float

Compute the log marginal likelihood of the training data.

Returns
log_likelihood
float
The log marginal likelihood value. Higher values indicate a better fit of the kernel and noise parameters to the data.
__repr__ (self) -> str

String representation.