API Reference / algorithms / svm /

svr.py

Support Vector Regressor (SVR) implementation.

Classes

SVR

class algorithms.svm.svr.SVR(Regressor)

Support Vector Regressor trained by Sequential Minimal Optimization.

Implements epsilon-insensitive Support Vector Regression (SVR) using the SMO algorithm. The model finds a function f(x) = w^T \phi(x) + b that has at most \epsilon deviation from the actual target values, while remaining as flat as possible.
Constructor 
__init__(
    self,
    C: float = 1.0,
    epsilon: float = 0.1,
    kernel: str = 'rbf',
    gamma: Any = 'scale',
    degree: int = 3,
    coef0: float = 0.0,
    tol: float = 0.001,
    max_iter: int = ...,
)

Overview

The SMO-based SVR training proceeds as follows:

  1. Map input features into a high-dimensional space via a kernel function
  2. Define an \epsilon-insensitive tube around the training targets
  3. Select pairs of dual variables (\alpha_i, \alpha_i^*) that violate KKT conditions
  4. Solve the two-variable sub-problem analytically to update the multipliers
  5. Update the bias term and check for convergence
  6. Repeat until all KKT conditions are satisfied or max_iter is reached

Theory

The SVR solves the following primal optimization problem:

\min_{w, b, \xi, \xi^*} \frac{1}{2}\|w\|^2 + C \sum_{i=1}^{n} (\xi_i + \xi_i^*)

Subject to:

y_i - w^T \phi(x_i) - b \leq \epsilon + \xi_i
w^T \phi(x_i) + b - y_i \leq \epsilon + \xi_i^*
\xi_i, \xi_i^* \geq 0

The corresponding dual formulation is:

\max_{\alpha, \alpha^*} -\frac{1}{2} \sum_{i,j} (\alpha_i - \alpha_i^*)(\alpha_j - \alpha_j^*) K(x_i, x_j) - \epsilon \sum_{i} (\alpha_i + \alpha_i^*) + \sum_{i} y_i (\alpha_i - \alpha_i^*)

Subject to:

0 \leq \alpha_i, \alpha_i^* \leq C
where:
  • C --- Regularization parameter trading off flatness against training error
  • \epsilon --- Width of the insensitive tube (errors within the tube are ignored)
  • K(x_i, x_j) --- Kernel function implementing the kernel trick
  • \alpha_i, \alpha_i^* --- Dual variables (Lagrange multipliers)
  • \xi_i, \xi_i^* --- Slack variables for points outside the \epsilon-tube
The prediction function is:
f(x) = \sum_{i \in SV} (\alpha_i - \alpha_i^*) K(x_i, x) + b

Parameters

C
float = 1.0
Regularization parameter. Larger values allow fewer points to lie outside the :math:`\epsilon`-tube but may cause overfitting.
epsilon
float = 0.1
Width of the epsilon-insensitive loss tube. Predictions within :math:`\epsilon` of the true value incur zero loss.
kernel
{'linear', 'poly', 'rbf'} = 'rbf'
Kernel type used for the non-linear mapping.
gamma
Union[str, float] = 'scale'

Kernel coefficient for 'rbf' and 'poly' kernels:

  • 'scale' -- Uses ``1 / (n_features * X.var())``
  • 'auto' -- Uses ``1 / n_features``
  • float -- User-defined positive coefficient
degree
int = 3
Degree of the polynomial kernel. Ignored by other kernels.
coef0
float = 0.0
Independent term in the polynomial kernel function.
tol
float = 1e-3
Tolerance for the stopping criterion in the optimization loop.
max_iter
int = 10000
Maximum number of optimization iterations.

Attributes

support_vectors_
np.ndarray
Training samples that lie on or outside the :math:`\epsilon`-tube boundary.
dual_coef_
np.ndarray
Coefficients :math:`(\alpha_i - \alpha_i^*)` of the support vectors in the prediction function.
intercept_
float
Intercept (bias) term :math:`b`.
n_support_
int
Total number of support vectors.

Notes

Complexity:

  • Training: O(n^2 \cdot p) to O(n^3) depending on the kernel cache, where n = samples, p = features
  • Prediction: O(n_{sv} \cdot p) per sample, where n_{sv} is the number of support vectors
When to use SVR:
  • Regression tasks where a non-linear relationship is expected
  • When robustness to outliers is important (errors within \epsilon are ignored)
  • Moderate-sized datasets (SVR scales quadratically with the number of samples)
  • When a sparse solution (few support vectors) is preferred

References

Smola2004
Smola, A.J. and Schoelkopf, B. (2004). A Tutorial on Support Vector Regression. Statistics and Computing, 14(3), pp. 199-222. DOI: 10.1023/B:STCO.0000035301.49549.88
Platt1999
Platt, J.C. (1999). Fast Training of Support Vector Machines Using Sequential Minimal Optimization. Advances in Kernel Methods --- Support Vector Learning, MIT Press, pp. 185-208.
Vapnik1995
Vapnik, V.N. (1995). The Nature of Statistical Learning Theory. Springer-Verlag, New York. DOI: 10.1007/978-1-4757-2440-0

See Also

SVC

Support Vector Classifier using SMO optimization.
RBFKernel

Radial Basis Function kernel.
PolynomialKernel

Polynomial kernel.

Basic regression with an RBF kernel:

python 
>>> from tuiml.algorithms.svm import SVR
>>> import numpy as np
>>>
>>> # Create sample data
>>> X_train = np.array([[1], [2], [3], [4], [5]], dtype=float)
>>> y_train = np.array([1.1, 2.0, 2.9, 4.1, 5.0])
>>>
>>> # Fit the regressor
>>> reg = SVR(C=1.0, epsilon=0.1, kernel='rbf')
>>> reg.fit(X_train, y_train)
SVR(...)
>>> predictions = reg.predict(X_train)

Methods

get_parameter_schema (cls) -> Dict[str, Dict[str, Any]]

Return parameter schema.

get_capabilities (cls) -> List[str]

Return algorithm capabilities.

get_complexity (cls) -> str

Return time/space complexity.

get_references (cls) -> List[str]

Return academic references.

fit (self, X: np.ndarray, y: np.ndarray) -> 'SVR'

Fit the SVR model using SMO algorithm.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Training features.
y
np.ndarray of shape (n_samples,)
Target values.
Returns
self
SVR
Returns the fitted instance.
predict (self, X: np.ndarray) -> np.ndarray

Predict target values for samples.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test features.
Returns
y_pred
np.ndarray of shape (n_samples,)
Predicted values.
score (self, X: np.ndarray, y: np.ndarray) -> float

Compute the :math:`R^2` (coefficient of determination) score.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test features.
y
np.ndarray of shape (n_samples,)
True target values.
Returns
r2
float
:math:`R^2` score, where 1.0 is a perfect fit.
__repr__ (self) -> str

String representation.