Ordinary Least Squares (OLS) linear regression with ridge regularization.
Classes
Linear Regression for predicting continuous values using ordinary least squares.
Implements ordinary least squares (OLS) linear regression with optional ridge regularization (L2) and automatic feature selection methods including M5 stepwise elimination and greedy backward elimination.
Constructor
__init__( self, ridge: float = 1e-08, attribute_selection: str = 'none', eliminate_colinear: bool = True, fit_intercept: bool = True, )
Overview
The algorithm fits a linear model through the following steps:
- Handle missing values by replacing with column means
- Standardize features to zero mean and unit variance
- Optionally eliminate highly colinear features (correlation > 0.99)
- Optionally perform feature selection (M5 or greedy backward elimination)
- Solve for coefficients using least squares with ridge regularization
- Transform coefficients back to the original feature scale
Theory
The model fits a linear equation relating features to the target:
y = X\beta + \epsilon
where \beta is the coefficient vector and \epsilon is the error term. The coefficients are estimated by minimizing the regularized sum of squared residuals:
\hat{\beta} = \arg\min_{\beta} \|y - X\beta\|_2^2 + \lambda \|\beta\|_2^2
The closed-form solution is:
\hat{\beta} = (X^T X + \lambda I)^{-1} X^T y
For numerical stability, the implementation uses np.linalg.lstsq with an augmented matrix rather than explicitly forming the Gram matrix.
Parameters
ridge
float
= 1e-8
Ridge regularization parameter (L2 penalty). A small value helps prevent numerical instability in matrix inversion.
attribute_selection
{"none", "m5", "greedy"}
= "none"
Feature selection method to use during fitting:
-
"none"- Use all features. -
"m5"- M5 method (stepwise backward elimination based on AIC). -
"greedy"- Greedy backward elimination based on coefficient significance.
eliminate_colinear
bool
= True
Whether to remove highly colinear attributes before fitting.
fit_intercept
bool
= True
Whether to fit an intercept (bias) term.
Attributes
coefficients_
np.ndarray
Regression coefficients of shape (n_features,).
intercept_
float
Intercept term.
selected_features_
np.ndarray
Indices of the features selected for the final model.
std_devs_
np.ndarray
Standard deviations of features used for scaling.
means_
np.ndarray
Means of features used for scaling.
Notes
Complexity:
- Training: O(n \cdot m^2 + m^3) for n samples and m features.
- Prediction: O(m) per sample.
- When the relationship between features and target is approximately linear
- When you need an interpretable model with explicit feature coefficients
- When the number of features is moderate relative to the number of samples
- As a baseline for comparison with more complex regression methods
References
Akaike1974
Akaike, H. (1974).
A new look at the statistical model identification.
IEEE Transactions on Automatic Control, 19(6), 716-723.
HoerlKennard1970
Hoerl, A.E. and Kennard, R.W. (1970).
Ridge Regression: Biased Estimation for Nonorthogonal Problems.
Technometrics, 12(1), 55-67.
Basic regression with automatic feature handling:
python
>>> import numpy as np
>>> from tuiml.algorithms.linear import LinearRegression
>>>
>>> # Generating sample data
>>> X = np.array([[1], [2], [3], [4]])
>>> y = np.array([2, 4, 6, 8])
>>>
>>> # Fit the model
>>> reg = LinearRegression()
>>> reg.fit(X, y)
>>>
>>> # Predict
>>> reg.predict([[5]])
array([10.])