API Reference / uncertainty / calibration /

isotonic.py

Isotonic probability calibration via the pool-adjacent-violators algorithm.

Classes

IsotonicCalibrator

class uncertainty.calibration.isotonic.IsotonicCalibrator(Calibrator)

Non-parametric probability calibration by isotonic regression.

Fits a monotone, piecewise-constant map from raw classifier scores to calibrated probabilities. Unlike PlattCalibrator it assumes no functional form — only that a higher score should never mean a lower probability — so it corrects arbitrary monotone distortions.
Constructor 
__init__(
    self,
    out_of_bounds: str = 'clip',
    increasing: bool = True) -> None,
)

Overview

  1. Sort the held-out calibration scores in increasing order.
  2. Run the pool-adjacent-violators algorithm (PAVA) on the paired
binary outcomes, merging any adjacent blocks that violate monotonicity into their weighted mean.
  1. Store the resulting step function as (thresholds_, values_).
  2. At transform time, interpolate a new score into that step function.

Theory

Given calibration pairs (s_i, y_i) sorted by score, isotonic regression solves the constrained least-squares problem

\min_{p_1 \leq p_2 \leq \dots \leq p_n} \sum_{i=1}^{n} w_i (p_i - y_i)^2

PAVA solves this exactly in O(n) by maintaining a stack of blocks with non-decreasing means; whenever a new value violates the order, the offending blocks are pooled into their weighted average.

Because the fit is piecewise constant with at most n levels, isotonic calibration is more expressive than a sigmoid but needs more calibration data — roughly 1000 samples before it beats Platt scaling.

Parameters

out_of_bounds
{'clip', 'nan'} = 'clip'
Behaviour for scores outside the calibration range. 'clip' extends the boundary probabilities; 'nan' returns np.nan.
increasing
bool = True
Whether the calibration map is non-decreasing in the score. Set to False for scores where a lower value means a higher probability.

Attributes

thresholds_
np.ndarray of shape (n_blocks,)
Score breakpoints of the fitted step function.
values_
np.ndarray of shape (n_blocks,)
Calibrated probability of each block.
classes_
np.ndarray of shape (n_classes,)
Class labels seen during fit.
fitted_
bool
Whether fit has been called.

Notes

Complexity. Fitting is O(n \log n) (dominated by the sort; PAVA itself is O(n)), transform is O(m \log n) via binary search. The PAVA step runs in the shared C++ kernel tuiml._cpp_ext.stats.pool_adjacent_violators.

When to use. Prefer isotonic when the calibration set is large (\gtrsim 1000 samples) or the miscalibration is not sigmoidal — for example the systematic over-confidence of boosted ensembles. Prefer Platt scaling on small calibration sets, where isotonic overfits.

References

Zadrozny2002
Zadrozny, B., & Elkan, C. (2002). Transforming Classifier Scores into Accurate Multiclass Probability Estimates. KDD, 694-699. :doi:`10.1145/775047.775151`
Ayer1955
Ayer, M., Brunk, H. D., Ewing, G. M., Reid, W. T., & Silverman, E. (1955). An Empirical Distribution Function for Sampling with Incomplete Information. Annals of Mathematical Statistics, 26(4), 641-647. :doi:`10.1214/aoms/1177728423`

See Also

PlattCalibrator

Parametric sigmoid calibration.
TemperatureScaler

Single-parameter multiclass calibration.
expected_calibration_error

Measures the improvement.
python 
>>> import numpy as np
>>> from tuiml.uncertainty import IsotonicCalibrator
>>> scores = np.array([0.1, 0.2, 0.35, 0.4, 0.65, 0.7, 0.8, 0.95])
>>> y = np.array([0, 0, 0, 1, 0, 1, 1, 1])
>>> cal = IsotonicCalibrator()
>>> proba = cal.fit_transform(scores, y)
>>> bool(np.all(np.diff(proba) >= 0))
True
>>> float(cal.transform(np.array([0.9]))[0])
1.0

Methods

fit (self, scores: np.ndarray, y: np.ndarray, sample_weight: Optional[np.ndarray]=None) -> 'IsotonicCalibrator'

Fit the isotonic calibration map on held-out scores.

Parameters
scores
np.ndarray of shape (n_samples,) or (n_samples, 2)
Uncalibrated scores. A two-column array is read as binary probabilities and its positive column is used.
y
np.ndarray of shape (n_samples,)
True binary labels.
sample_weight
np.ndarray of shape (n_samples,)
Per-sample weights. Defaults to uniform.
Returns
self
IsotonicCalibrator
The fitted calibrator.
transform (self, scores: np.ndarray) -> np.ndarray

Map raw scores onto calibrated probabilities.

Parameters
scores
np.ndarray of shape (n_samples,) or (n_samples, 2)
Uncalibrated scores.
Returns
proba
np.ndarray of shape (n_samples,)
Calibrated probability of the positive class.
predict_proba (self, scores: np.ndarray) -> np.ndarray

Return two-column calibrated probabilities.

Parameters
scores
np.ndarray of shape (n_samples,) or (n_samples, 2)
Uncalibrated scores.
Returns
proba
np.ndarray of shape (n_samples, 2)
Calibrated probabilities for the negative and positive class.
get_parameter_schema (cls) -> Dict[str, Any]

Return JSON Schema for constructor parameters.

__repr__ (self) -> str

Return a readable representation of the calibrator.