API Reference / algorithms / linear /

sgd.py

Stochastic Gradient Descent (SGD) for linear classification and regression.

Classes

SGDClassifier

class algorithms.linear.sgd.SGDClassifier(Classifier)

Stochastic Gradient Descent classifier for large-scale linear classification.

Implements SGD learning for linear classifiers with support for multiple loss functions including hinge (linear SVM), log (logistic regression), and modified Huber. It is particularly efficient for large-scale datasets and online learning via the partial_fit method.
Constructor 
__init__(
    self,
    loss: str = 'hinge',
    learning_rate: float = 0.01,
    regularization: str = 'l2',
    lambda_: float = 0.0001,
    n_epochs: int = 100,
    random_state: Optional[int] = None,
    shuffle: bool = True,
    batch_size: int = 256,
)

Overview

The algorithm trains a linear classifier using mini-batch SGD:

  1. Initialize weights and bias to zero
  2. For each epoch, optionally shuffle the training data
  3. Process data in mini-batches for vectorized gradient computation
  4. Compute the loss gradient and regularization gradient for each batch
  5. Update weights using the learning rate (with inverse scaling schedule)
  6. For multiclass problems, train one-vs-all binary classifiers

Theory

At each step, the weights are updated using the gradient of the loss function with regularization:

w \leftarrow w - \eta \left(\nabla_w L(w, x, y) + \lambda \nabla R(w)\right)

The supported loss functions are:

  • Hinge loss (linear SVM): L = \max(0, 1 - y \cdot w^T x)
  • Log loss (logistic regression): L = \log(1 + e^{-y \cdot w^T x})
  • Modified Huber: a smooth approximation combining hinge and log loss
The learning rate follows an inverse scaling schedule:
\eta_t = \frac{\eta_0}{1 + t \cdot 10^{-4}}

Parameters

loss
{"hinge", "log", "modified_huber"} = "hinge"

The loss function to be used:

  • "hinge" - Gives a linear SVM.
  • "log" - Gives logistic regression, a probabilistic classifier.
  • "modified_huber" - Smooth loss that brings tolerance to outliers
as well as probability estimates.
learning_rate
float = 0.01
Initial learning rate for the weight updates.
regularization
{"l1", "l2", "elasticnet", "none"} = "l2"
The penalty (regularization term) to be used.
lambda_
float = 0.0001
Regularization strength; must be a positive float.
n_epochs
int = 100
Number of passes over the training data.
random_state
int or None = None
Seed used by the random number generator.
shuffle
bool = True
Whether the training data should be shuffled after each epoch.
batch_size
int = 256
Mini-batch size for gradient updates. Larger batches enable better vectorization but may converge differently than pure SGD (batch_size=1).

Attributes

weights_
np.ndarray
Weight vector of shape (n_features,) for binary or (n_classes, n_features) for multiclass.
bias_
float or np.ndarray
Bias term (intercept).
classes_
np.ndarray
Unique class labels found during training.

Notes

Complexity:

  • Training: O(n \cdot m \cdot \text{n\_epochs}) where n is
the number of samples and m is the number of features.
  • Prediction: O(m) per sample.
When to use SGDClassifier:
  • Large-scale datasets where batch solvers are too slow
  • Online or streaming learning scenarios (via partial_fit)
  • When you want to switch between SVM and logistic regression by changing the loss
  • When L1 regularization (sparsity) is desired

References

Bottou2010
Bottou, L. (2010). Large-Scale Machine Learning with Stochastic Gradient Descent. Proceedings of COMPSTAT 2010, pp. 177-186. DOI: 10.1007/978-3-7908-2604-3_16
Zhang2004
Zhang, T. (2004). Solving Large Scale Linear Prediction Problems Using Stochastic Gradient Descent Algorithms. Proceedings of the 21st International Conference on Machine Learning (ICML).

See Also

LogisticRegression

Batch gradient descent logistic regression with L2 regularization.
SGDRegressor

SGD-trained linear regressor for large-scale regression.

Binary classification with SGD using log loss:

python 
>>> import numpy as np
>>> from tuiml.algorithms.linear import SGDClassifier
>>>
>>> X = np.array([[0, 0], [1, 1], [2, 2], [3, 3]])
>>> y = np.array([0, 0, 1, 1])
>>>
>>> clf = SGDClassifier(loss='log', learning_rate=0.01)
>>> clf.fit(X, y)
>>>
>>> clf.predict([[2.5, 2.5]])
array([1])

Methods

get_parameter_schema (cls) -> Dict[str, Dict[str, Any]]
get_capabilities (cls) -> List[str]
get_complexity (cls) -> str
get_references (cls) -> List[str]
fit (self, X: np.ndarray, y: np.ndarray) -> 'SGDClassifier'

Fit the SGD classifier.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Training data.
y
np.ndarray of shape (n_samples,)
Target labels.
Returns
self
SGDClassifier
Fitted classifier.
predict (self, X: np.ndarray) -> np.ndarray

Predict class labels for samples.

Parameters
X
np.ndarray of shape (n_samples, n_features)
The input samples.
Returns
predictions
np.ndarray
Predicted class labels.
predict_proba (self, X: np.ndarray) -> np.ndarray

Predict class probabilities for samples.

Parameters
X
np.ndarray of shape (n_samples, n_features)
The input samples.
Returns
probabilities
np.ndarray of shape (n_samples, n_classes)
Predicted probabilities for each class.
partial_fit (self, X: np.ndarray, y: np.ndarray) -> 'SGDClassifier'

Update classifier with new samples (online learning).

Parameters
X
np.ndarray of shape (n_samples, n_features)
New training samples.
y
np.ndarray of shape (n_samples,)
New target labels.
Returns
self
SGDClassifier
Updated classifier.
__repr__ (self) -> str

SGDRegressor

class algorithms.linear.sgd.SGDRegressor(Regressor)

Stochastic Gradient Descent regressor for large-scale linear regression.

Implements SGD learning for linear regression with support for squared error and Huber loss functions. It is particularly efficient for large-scale datasets and supports online learning via the partial_fit method.
Constructor 
__init__(
    self,
    loss: str = 'squared_error',
    learning_rate: float = 0.01,
    regularization: str = 'l2',
    lambda_: float = 0.0001,
    n_epochs: int = 100,
    epsilon: float = 0.1,
    random_state: Optional[int] = None,
    shuffle: bool = True,
    batch_size: int = 256,
)

Overview

The algorithm trains a linear regression model using mini-batch SGD:

  1. Initialize weights and bias to zero
  2. For each epoch, optionally shuffle the training data
  3. Process data in mini-batches for vectorized gradient computation
  4. Compute the loss gradient (squared error or Huber) and regularization gradient
  5. Update weights using the learning rate (with inverse scaling schedule)
  6. Repeat for n_epochs passes over the data

Theory

The model fits a linear function f(x) = w^T x + b by minimizing the regularized empirical risk:

\min_{w, b} \frac{1}{n} \sum_{i=1}^{n} L(y_i, w^T x_i + b) + \lambda R(w)

Squared error loss:

L(y, \hat{y}) = \frac{1}{2}(y - \hat{y})^2

Huber loss (robust to outliers):

L_\delta(y, \hat{y}) = \begin{cases} \frac{1}{2}(y - \hat{y})^2 & \text{if } |y - \hat{y}| \leq \epsilon \ \epsilon |y - \hat{y}| - \frac{1}{2}\epsilon^2 & \text{otherwise} \end{cases}

The weight update rule at each step is:

w \leftarrow w - \eta_t \left(\nabla_w L + \lambda \nabla R(w)\right)

with the inverse scaling learning rate schedule \eta_t = \eta_0 / (1 + t \cdot 10^{-4}).

Parameters

loss
{"squared_error", "huber"} = "squared_error"

The loss function to be used:

  • "squared_error" - Ordinary least squares.
  • "huber" - Huber loss for robustness to outliers.
learning_rate
float = 0.01
Initial learning rate for the weight updates.
regularization
{"l1", "l2", "elasticnet", "none"} = "l2"
The penalty (regularization term) to be used.
lambda_
float = 0.0001
Regularization strength; must be a positive float.
n_epochs
int = 100
Number of passes over the training data.
epsilon
float = 0.1
The epsilon threshold for the Huber loss. Residuals smaller than epsilon are penalized quadratically; larger residuals are penalized linearly.
random_state
int or None = None
Seed used by the random number generator.
shuffle
bool = True
Whether the training data should be shuffled after each epoch.
batch_size
int = 256
Mini-batch size for gradient updates. Larger batches enable better vectorization but may converge differently than pure SGD (batch_size=1).

Attributes

weights_
np.ndarray
Weight vector of shape (n_features,).
bias_
float
Bias term (intercept).

Notes

Complexity:

  • Training: O(n \cdot m \cdot \text{n\_epochs}) where n is
the number of samples and m is the number of features.
  • Prediction: O(m) per sample.
When to use SGDRegressor:
  • Large-scale regression datasets where closed-form OLS is too slow
  • Online or streaming regression scenarios (via partial_fit)
  • When Huber loss is needed for robustness to outliers
  • When L1 regularization (sparse solutions) is desired

References

Bottou2010
Bottou, L. (2010). Large-Scale Machine Learning with Stochastic Gradient Descent. Proceedings of COMPSTAT 2010, pp. 177-186. DOI: 10.1007/978-3-7908-2604-3_16
Huber1964
Huber, P.J. (1964). Robust Estimation of a Location Parameter. The Annals of Mathematical Statistics, 35(1), 73-101.

See Also

LinearRegression

Closed-form OLS regression with ridge regularization.
SGDClassifier

SGD-trained linear classifier for classification tasks.

Regression with SGD using squared error loss:

python 
>>> import numpy as np
>>> from tuiml.algorithms.linear import SGDRegressor
>>>
>>> # Generate simple linear data
>>> X = np.array([[1], [2], [3], [4], [5]])
>>> y = np.array([2.0, 4.0, 6.0, 8.0, 10.0])
>>>
>>> # Fit the model
>>> reg = SGDRegressor(learning_rate=0.01, n_epochs=1000, random_state=42)
>>> reg.fit(X, y)
>>>
>>> # Predict
>>> reg.predict([[6]])

Methods

get_parameter_schema (cls) -> Dict[str, Dict[str, Any]]
get_capabilities (cls) -> List[str]
fit (self, X: np.ndarray, y: np.ndarray) -> 'SGDRegressor'

Fit the SGD regressor.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Training features.
y
np.ndarray of shape (n_samples,)
Target values.
Returns
self
SGDRegressor
Fitted regressor.
predict (self, X: np.ndarray) -> np.ndarray

Predict target values for samples.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Input samples.
Returns
predictions
np.ndarray of shape (n_samples,)
Predicted continuous values.
partial_fit (self, X: np.ndarray, y: np.ndarray) -> 'SGDRegressor'

Update regressor with new samples (online learning).

Parameters
X
np.ndarray of shape (n_samples, n_features)
New training samples.
y
np.ndarray of shape (n_samples,)
New target values.
Returns
self
SGDRegressor
Updated regressor.
__repr__ (self) -> str