API Reference / algorithms / svm /

svc.py

Support Vector Classifier (SVC) implementation.

Classes

SVC

class algorithms.svm.svc.SVC(Classifier)

Support Vector Classifier using Sequential Minimal Optimization (SMO).

SVC is an efficient algorithm for training Support Vector Machines that breaks the large quadratic programming (QP) problem into a series of the smallest possible QP sub-problems, each solved analytically in closed form.
Constructor 
__init__(
    self,
    C: float = 1.0,
    kernel: Union[str, 'Kernel'] = 'rbf',
    gamma: Any = 'scale',
    degree: int = 3,
    coef0: float = 0.0,
    tol: float = 0.001,
    max_iter: int = ...,
)

Overview

The SMO-based SVC training proceeds as follows:

  1. Map input features into a (possibly high-dimensional) space via a kernel function
  2. Select two Lagrange multipliers that violate the KKT conditions
  3. Solve the two-variable QP sub-problem analytically to update the multipliers
  4. Update the bias (threshold) term and error cache
  5. Repeat steps 2-4 until all KKT conditions are satisfied or max_iter is reached
  6. For multiclass problems, use a one-vs-one decomposition with majority voting

Theory

The SVC solves the following primal optimization problem:

\min_{w, b, \xi} \frac{1}{2}\|w\|^2 + C \sum_{i=1}^{n} \xi_i

Subject to:

y_i (w^T \phi(x_i) + b) \geq 1 - \xi_i, \quad \xi_i \geq 0

The corresponding dual formulation is:

\max_{\alpha} \sum_{i=1}^{n} \alpha_i - \frac{1}{2} \sum_{i,j} \alpha_i \alpha_j y_i y_j K(x_i, x_j)

Subject to:

0 \leq \alpha_i \leq C, \quad \sum_{i=1}^{n} \alpha_i y_i = 0
where:
  • C --- Regularization parameter controlling the trade-off between margin width and training error
  • K(x_i, x_j) = \phi(x_i)^T \phi(x_j) --- Kernel function (the kernel trick)
  • \alpha_i --- Lagrange multipliers (dual coefficients)
  • \xi_i --- Slack variables allowing for soft-margin classification
The decision function for a new sample x is:
f(x) = \sum_{i \in SV} \alpha_i y_i K(x_i, x) + b

Parameters

C
float = 1.0
Regularization parameter. Larger values penalize misclassification more heavily, yielding a narrower margin with fewer training errors.
kernel
str or Kernel = 'rbf'
Kernel function. Can be 'linear', 'poly', 'rbf', 'sigmoid', or a Kernel object from tuiml.algorithms.svm.kernels.
gamma
Union[str, float] = 'scale'

Kernel coefficient for 'rbf', 'poly', and 'sigmoid' kernels:

  • 'scale' -- Uses ``1 / (n_features * X.var())``
  • 'auto' -- Uses ``1 / n_features``
  • float -- User-defined positive coefficient
degree
int = 3
Degree of the polynomial kernel. Ignored by other kernels.
coef0
float = 0.0
Independent term in the 'poly' and 'sigmoid' kernel functions.
tol
float = 1e-3
Tolerance for the stopping criterion in the SMO optimization loop.
max_iter
int = 1000
Maximum number of passes over the training set during optimization.

Attributes

classes_
np.ndarray
Unique class labels discovered during fit().
support_
np.ndarray
Indices of support vectors in the training data.
support_vectors_
np.ndarray
Training samples that lie on or within the margin boundary.
dual_coef_
np.ndarray
Coefficients (:math:`\alpha_i y_i`) of the support vectors in the decision function.
intercept_
np.ndarray
Intercept (bias) term :math:`b` of the decision function.
n_support_
np.ndarray
Number of support vectors for each class.
kernel_
Kernel
The actual kernel object used after fit().

Notes

Complexity:

  • Training: O(n^2 \cdot p) to O(n^3) depending on kernel cache efficiency, where n = samples, p = features
  • Prediction: O(n_{sv} \cdot p) per sample, where n_{sv} is the number of support vectors
When to use SVC:
  • Binary or multiclass classification with moderate-sized datasets
  • When a non-linear decision boundary is needed (via kernel trick)
  • High-dimensional data where the number of features exceeds the number of samples
  • When a sparse solution (few support vectors) is desirable

References

Platt1998
Platt, J.C. (1998). Sequential Minimal Optimization: A Fast Algorithm for Training Support Vector Machines. Microsoft Research Technical Report MSR-TR-98-14.
Vapnik1995
Vapnik, V.N. (1995). The Nature of Statistical Learning Theory. Springer-Verlag, New York. DOI: 10.1007/978-1-4757-2440-0
Keerthi2001
Keerthi, S.S., Shevade, S.K., Bhattacharyya, C. and Murthy, K.R.K. (2001). Improvements to Platt's SMO Algorithm for SVM Classifier Design. Neural Computation, 13(3), pp. 637-649. DOI: 10.1162/089976601300014493

See Also

SVR

Support Vector Regression using SMO optimization.
RBFKernel

Radial Basis Function kernel.
LinearKernel

Linear kernel for linearly separable data.

Basic binary classification with an RBF kernel:

python 
>>> from tuiml.algorithms.svm import SVC
>>> import numpy as np
>>>
>>> # Create sample data
>>> X_train = np.array([[0, 0], [1, 1], [2, 2], [3, 3]])
>>> y_train = np.array([0, 0, 1, 1])
>>>
>>> # Fit the classifier
>>> clf = SVC(C=1.0, kernel='rbf', gamma=0.5)
>>> clf.fit(X_train, y_train)
SVC(...)
>>> predictions = clf.predict(X_train)

Using a kernel object directly:

python 
>>> from tuiml.algorithms.svm import SVC
>>> from tuiml.algorithms.svm.kernels import RBFKernel
>>> kernel = RBFKernel(gamma=0.5)
>>> clf = SVC(C=1.0, kernel=kernel)
>>> clf.fit(X_train, y_train)
SVC(...)

Methods

get_parameter_schema (cls) -> Dict[str, Dict[str, Any]]

Return parameter schema.

get_capabilities (cls) -> List[str]

Return classifier capabilities.

get_complexity (cls) -> str

Return time/space complexity.

get_references (cls) -> List[str]

Return academic references.

fit (self, X: np.ndarray, y: np.ndarray) -> 'SVC'

Fit the SVC classifier.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Training features.
y
np.ndarray of shape (n_samples,)
Target labels.
Returns
self
SVC
Returns the fitted instance.
predict (self, X: np.ndarray) -> np.ndarray

Predict class labels for samples.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test features.
Returns
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
predict_proba (self, X: np.ndarray) -> np.ndarray

Predict class probabilities for samples.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test features.
Returns
proba
np.ndarray of shape (n_samples, n_classes)
Class probabilities.
decision_function (self, X: np.ndarray) -> np.ndarray

Compute the decision function values for input samples.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Input samples.
Returns
decision
np.ndarray of shape (n_samples,) or (n_samples, n_classifiers)
Decision function values. For binary, shape is (n_samples,). For multiclass one-vs-one, shape is (n_samples, n_classifiers).
kernel_ (self)

Return the actual kernel object used.

__repr__ (self) -> str

String representation.