Logistic Regression classifier with L2 regularization.
Classes
Logistic Regression classifier with L2 regularization.
Logistic regression models the probability of class membership using the logistic (sigmoid) function. It supports both binary and multiclass classification via the softmax generalization.
Constructor
__init__( self, max_iter: int = 1000, learning_rate: float = 1.0, ridge: Union[float, str] = 'auto', tol: float = 0.0001, solver: str = 'lbfgs', )
Overview
The algorithm fits a linear decision boundary by iterative optimization:
- Initialize weight matrix and bias to zeros
- Compute predicted probabilities using sigmoid (binary) or softmax (multiclass)
- Compute the cross-entropy loss with L2 regularization penalty
- Update weights with the selected solver (L-BFGS quasi-Newton by
solver="gd")
-
Repeat until convergence or
max_iteris reached
Theory
For binary classification, the model predicts the probability of the positive class using the sigmoid function:
P(y=1 \mid x) = \sigma(w^T x + b) = \frac{1}{1 + e^{-(w^T x + b)}}
For multiclass classification with c classes, the softmax function is applied:
P(y=k \mid x) = \frac{e^{w_k^T x + b_k}}{\sum_{j=1}^{c} e^{w_j^T x + b_j}}
The loss function minimized is the regularized cross-entropy:
\mathcal{L}(w) = -\frac{1}{n}\sum_{i=1}^{n} \sum_{k=1}^{c} y_{ik} \log P(y=k \mid x_i) + \frac{\lambda}{2} \|w\|_2^2
where \lambda is the ridge regularization parameter.
Parameters
max_iter
int
= 1000
Maximum number of solver iterations.
learning_rate
float
= 1.0
Step size for weight updates. Only used by
solver="gd"; L-BFGS performs its own line search.
ridge
float or "auto"
= "auto"
L2 regularization parameter (penalty) to prevent overfitting.
"auto" resolves to 1 / n_samples at fit time, which is equivalent to scikit-learn's default C=1.0.
tol
float
= 1e-4
Convergence tolerance. For
"lbfgs" this bounds the projected gradient norm; for "gd" training stops when the improvement in loss is less than this value.
solver
{"lbfgs", "gd"}
= "lbfgs"
Optimization algorithm.
"lbfgs" (quasi-Newton with line search, via SciPy) converges reliably without tuning; "gd" is the legacy fixed-step batch gradient descent.
Attributes
classes_
np.ndarray
Unique class labels found during training.
coef_
np.ndarray
Weight matrix of shape (n_classes, n_features).
intercept_
np.ndarray
Bias vector of shape (n_classes,).
n_iter_
int
Actual number of iterations run during training.
Notes
Complexity:
- Training: O(n \cdot m \cdot c \cdot \text{iterations}) where n is
- Prediction: O(m \cdot c) per sample.
- When you need a probabilistic classifier with calibrated probabilities
- Binary or multiclass problems with linearly separable (or near-linear) classes
- When model interpretability via feature coefficients is important
- As a baseline before trying more complex nonlinear models
References
LeCessie1992
le Cessie, S. and van Houwelingen, J.C. (1992).
Ridge Estimators in Logistic Regression.
Applied Statistics, 41(1), 191-201.
Bishop2006
Bishop, C.M. (2006).
Pattern Recognition and Machine Learning.
Springer, Chapter 4.
See Also
Basic binary classification with logistic regression:
python
>>> import numpy as np
>>> from tuiml.algorithms.linear import LogisticRegression
>>>
>>> # Generate binary classification data
>>> X = np.array([[0, 0], [1, 1], [2, 2], [3, 3]])
>>> y = np.array([0, 0, 1, 1])
>>>
>>> # Fit model
>>> clf = LogisticRegression()
>>> clf.fit(X, y)
>>>
>>> # Predict classes
>>> clf.predict([[0.5, 0.5], [4, 4]])
array([0, 1])
>>>
>>> # Get probabilities
>>> clf.predict_proba([[1.5, 1.5]])
array([[0.5, 0.5]])