API Reference / uncertainty / calibration /

platt.py

Platt (sigmoid) probability calibration.

Classes

PlattCalibrator

class uncertainty.calibration.platt.PlattCalibrator(Calibrator)

Probability calibration by fitting a sigmoid to held-out scores.

Platt scaling maps a raw score s onto a probability through a one-dimensional logistic link whose two parameters are fitted by maximum likelihood on a calibration set. It is the standard remedy for the margin-like, uncalibrated outputs of SVMs and boosted ensembles.
Constructor 
__init__(
    self,
    max_iter: int = 100,
    tol: float = 1e-10,
    regularize_targets: bool = True) -> None,
)

Overview

  1. Hold out a calibration set that the model was not trained on.
  2. Replace the hard labels with Platt's regularised targets, which pull the
fit away from 0 and 1 and prevent overfitting on small samples.
  1. Fit A and B by Newton descent on the log-likelihood.
  2. At transform time, apply the fitted sigmoid to new scores.

Theory

The calibration map is

P(y = 1 \mid s) = \frac{1}{1 + \exp(A s + B)}

fitted by minimising the regularised cross-entropy

-\sum_i t_i \log p_i + (1 - t_i) \log (1 - p_i)

where the targets follow Platt's correction for N_+ positive and N_- negative calibration samples:

t_i = \frac{N_+ + 1}{N_+ + 2} \ \text{if } y_i = 1, \quad t_i = \frac{1}{N_- + 2} \ \text{otherwise}

A negative A gives the usual increasing map; the sign is learned, so the calibrator also handles scores oriented the other way round.

Parameters

max_iter
int = 100
Maximum number of Newton iterations.
tol
float = 1e-10
Convergence tolerance on the gradient norm.
regularize_targets
bool = True
Whether to use Platt's regularised targets instead of the raw 0/1 labels. Strongly recommended on small calibration sets.

Attributes

a_
float
Fitted slope of the sigmoid.
b_
float
Fitted intercept of the sigmoid.
n_iter_
int
Newton iterations actually performed.
classes_
np.ndarray of shape (n_classes,)
Class labels seen during fit.
fitted_
bool
Whether fit has been called.

Notes

Complexity. O(n) per Newton iteration, O(1) memory beyond the score vector. Transform is O(m).

When to use. Platt scaling is the right default when the calibration set is small (a few hundred samples) or the miscalibration is a smooth monotone squashing — the typical SVM or AdaBoost case. When the calibration set is large or the distortion is not sigmoidal, prefer IsotonicCalibrator.

References

Platt1999
Platt, J. (1999). Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. Advances in Large Margin Classifiers, 61-74.
Lin2007
Lin, H.-T., Lin, C.-J., & Weng, R. C. (2007). A Note on Platt's Probabilistic Outputs for Support Vector Machines. Machine Learning, 68(3), 267-276. :doi:`10.1007/s10994-007-5018-6`

See Also

IsotonicCalibrator

Non-parametric calibration.
TemperatureScaler

Multiclass single-parameter calibration.
python 
>>> import numpy as np
>>> from tuiml.uncertainty import PlattCalibrator
>>> rng = np.random.default_rng(0)
>>> scores = np.concatenate([rng.normal(-1, 1, 200), rng.normal(1, 1, 200)])
>>> y = np.concatenate([np.zeros(200), np.ones(200)])
>>> cal = PlattCalibrator().fit(scores, y)
>>> proba = cal.transform(np.array([-2.0, 0.0, 2.0]))
>>> bool(proba[0] < proba[1] < proba[2])
True

Methods

fit (self, scores: np.ndarray, y: np.ndarray) -> 'PlattCalibrator'

Fit the sigmoid on held-out scores.

Parameters
scores
np.ndarray of shape (n_samples,) or (n_samples, 2)
Uncalibrated scores or decision values.
y
np.ndarray of shape (n_samples,)
True binary labels.
Returns
self
PlattCalibrator
The fitted calibrator.
transform (self, scores: np.ndarray) -> np.ndarray

Map raw scores onto calibrated probabilities.

Parameters
scores
np.ndarray of shape (n_samples,) or (n_samples, 2)
Uncalibrated scores.
Returns
proba
np.ndarray of shape (n_samples,)
Calibrated probability of the positive class.
predict_proba (self, scores: np.ndarray) -> np.ndarray

Return two-column calibrated probabilities.

Parameters
scores
np.ndarray of shape (n_samples,) or (n_samples, 2)
Uncalibrated scores.
Returns
proba
np.ndarray of shape (n_samples, 2)
Calibrated probabilities for the negative and positive class.
get_parameter_schema (cls) -> Dict[str, Any]

Return JSON Schema for constructor parameters.

__repr__ (self) -> str

Return a readable representation of the calibrator.