API Reference / algorithms / bayesian /

bayesian_linear_regression.py

Bayesian Linear Regression implementation.

Classes

BayesianLinearRegressor

class algorithms.bayesian.bayesian_linear_regression.BayesianLinearRegressor(Regressor)

Bayesian Linear Regression with conjugate prior for uncertainty-aware predictions.

BayesianLinearRegressor places a Gaussian prior over the weight parameters and computes a closed-form posterior distribution using the conjugate normal-inverse-gamma model. This yields both point predictions and predictive uncertainty estimates for each input.
Constructor 
__init__(
    self,
    alpha: float = 1.0,
    beta: float = 1.0,
    fit_intercept: bool = True,
)

Overview

The algorithm proceeds as follows:

  1. Optionally augment the feature matrix with a bias column for the intercept
  2. Compute the posterior covariance S_N from the prior precision
and the data likelihood
  1. Compute the posterior mean \mathbf{m}_N as the MAP estimate
  2. At prediction time, compute the mean prediction and optionally the
predictive standard deviation that accounts for both parameter uncertainty and observation noise

Theory

Given a prior on the weights \mathbf{w} \sim \mathcal{N}(\mathbf{0}, \alpha^{-1} I) and a likelihood p(\mathbf{y} | X, \mathbf{w}) = \mathcal{N}(\Phi \mathbf{w}, \beta^{-1} I), the posterior is Gaussian:

p(\mathbf{w} | \mathbf{y}, X) = \mathcal{N}(\mathbf{m}_N, S_N)
where:
S_N = (\alpha I + \beta \, \Phi^\top \Phi)^{-1}
\mathbf{m}_N = \beta \, S_N \, \Phi^\top \mathbf{y}

The predictive distribution for a new input \mathbf{x}_* is:

p(y_* | \mathbf{x}_*, \mathbf{y}, X) = \mathcal{N}(\mathbf{m}_N^\top \phi_*, \sigma_*^2)

with predictive variance:

\sigma_*^2 = \beta^{-1} + \phi_*^\top S_N \, \phi_*

Parameters

alpha
float = 1.0
Prior precision (inverse variance) for the weight prior. Larger values impose stronger regularisation toward zero weights.
beta
float = 1.0
Noise precision (inverse variance) of the observation noise. Larger values assume less noise in the observations.
fit_intercept
bool = True
Whether to add an intercept (bias) term to the model by augmenting the feature matrix with a column of ones.

Attributes

coef_
np.ndarray of shape (n_features,)
Posterior mean of the weight vector (excluding intercept).
intercept_
float
The intercept term. Set to 0.0 if fit_intercept=False.
posterior_cov_
np.ndarray of shape (n_params, n_params)
Posterior covariance matrix :math:`S_N` over the weight parameters.

Notes

Complexity:

  • Training: O(m^2 n + m^3) where n = samples, m = features
  • Prediction (mean): O(m) per sample
  • Prediction (std): O(m^2) per sample
  • Memory: O(m^2) for storing the posterior covariance
When to use BayesianLinearRegressor:
  • When you need uncertainty estimates alongside predictions
  • Small-to-medium datasets where a linear model is appropriate
  • When you want built-in regularisation via the prior
  • Active learning or Bayesian optimisation settings
  • When interpretability of the model weights is important

References

Bishop2006
Bishop, C.M. (2006). Pattern Recognition and Machine Learning. Springer, Chapter 3.3. DOI: 10.1007/978-0-387-45528-0
Murphy2012
Murphy, K.P. (2012). Machine Learning: A Probabilistic Perspective. MIT Press, Chapter 7.6.

See Also

GaussianProcessesRegressor

Non-parametric Bayesian regression with kernels.
LinearRegression

Ordinary least squares linear regression.

Basic Bayesian linear regression with uncertainty:

python 
>>> from tuiml.algorithms.bayesian import BayesianLinearRegressor
>>> import numpy as np
>>> X = np.array([[1], [2], [3], [4], [5]])
>>> y = np.array([1.1, 2.0, 3.1, 3.9, 5.0])
>>> reg = BayesianLinearRegressor(alpha=1.0, beta=25.0)
>>> reg.fit(X, y)
BayesianLinearRegressor(alpha=1.0, beta=25.0)
>>> reg.predict(np.array([[3.5]]))
array([...])
>>> reg.predict_std(np.array([[3.5]]))
array([...])

Methods

get_parameter_schema (cls) -> Dict[str, Dict[str, Any]]

Return parameter schema.

get_capabilities (cls) -> List[str]

Return regressor capabilities.

get_complexity (cls) -> str

Return time/space complexity.

get_references (cls) -> List[str]

Return academic references.

fit (self, X: np.ndarray, y: np.ndarray) -> 'BayesianLinearRegressor'

Fit the Bayesian linear regression model.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Training features.
y
np.ndarray of shape (n_samples,)
Target values.
Returns
self
BayesianLinearRegressor
Returns the fitted instance.
predict (self, X: np.ndarray) -> np.ndarray

Predict target values (posterior mean predictions).

Parameters
X
np.ndarray of shape (n_samples, n_features)
Input features.
Returns
y_pred
np.ndarray of shape (n_samples,)
Predicted mean target values.
predict_std (self, X: np.ndarray) -> np.ndarray

Predict standard deviations of the predictive distribution.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Input features.
Returns
y_std
np.ndarray of shape (n_samples,)
Predicted standard deviations.
score (self, X: np.ndarray, y: np.ndarray) -> float

Compute the R-squared (coefficient of determination) score.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test features.
y
np.ndarray of shape (n_samples,)
True target values.
Returns
score
float
R-squared score.
__repr__ (self) -> str

String representation.