API Reference / uncertainty / calibration /

temperature.py

Temperature and vector scaling for multiclass probability calibration.

Classes

TemperatureScaler

class uncertainty.calibration.temperature.TemperatureScaler(Calibrator)

Multiclass calibration by dividing logits by a single temperature.

Temperature scaling rescales every logit by one learned scalar T > 0. Because a positive scalar cannot change the ranking of the logits, the calibrated model has exactly the same accuracy as the original — only its confidence changes. This is the standard fix for the over-confidence of modern neural networks.
Constructor 
__init__(
    self,
    max_iter: int = 200,
    tol: float = 1e-06,
    log_t_bounds: tuple = ()) -> None,
)

Overview

  1. Collect logits on a held-out calibration set.
  2. Fit a single temperature T by minimising the negative
log-likelihood with a bounded 1-D search.
  1. At transform time, apply \text{softmax}(z / T).
A fitted T > 1 softens an over-confident model; T < 1 sharpens an under-confident one.

Theory

The calibrated probability of class k is

p_k = \frac{\exp(z_k / T)}{\sum_j \exp(z_j / T)}

and T minimises the calibration-set cross-entropy

\mathcal{L}(T) = -\frac{1}{n} \sum_{i=1}^{n} \log p_{y_i}(z_i / T)

which is convex in 1/T, so a golden-section search on \log T finds the global optimum without gradients.

Parameters

max_iter
int = 200
Maximum number of golden-section iterations.
tol
float = 1e-6
Convergence tolerance on the bracketed interval in log(T).
log_t_bounds
tuple of float, 4.0) = (-4.0
Search bracket for :math:`\log T`, i.e. roughly T in [0.018, 54.6].

Attributes

temperature_
float
The fitted temperature.
n_iter_
int
Golden-section iterations performed.
classes_
np.ndarray of shape (n_classes,)
Class labels seen during fit.
fitted_
bool
Whether fit has been called.

Notes

Complexity. O(n \cdot c) per iteration for n samples and c classes; the number of iterations is fixed by tol, not by n.

When to use. Use temperature scaling for any multiclass model whose ranking must not change — the accuracy-preserving property is the reason it is preferred over per-class isotonic calibration for deep networks. It cannot fix class-dependent bias; reach for VectorScaler when different classes are miscalibrated in different directions.

References

Guo2017
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. ICML, 1321-1330. :arxiv:`1706.04599`

See Also

VectorScaler

Per-class scale and bias.
PlattCalibrator

Binary sigmoid calibration.
expected_calibration_error

Measures the improvement.
python 
>>> import numpy as np
>>> from tuiml.uncertainty import TemperatureScaler
>>> from tuiml.uncertainty import expected_calibration_error
>>> rng = np.random.default_rng(0)
>>> y = rng.integers(0, 3, 300)
>>> noisy = np.where(rng.random(300) < 0.3, (y + 1) % 3, y)
>>> logits = np.eye(3)[noisy] * 6.0 + rng.normal(0, 1.0, (300, 3))
>>> scaler = TemperatureScaler().fit(logits, y)
>>> bool(scaler.temperature_ > 1.0)  # the model was over-confident
True
>>> proba = scaler.transform(logits)
>>> bool(expected_calibration_error(y, proba) < 0.1)
True

Methods

fit (self, scores: np.ndarray, y: np.ndarray) -> 'TemperatureScaler'

Fit the temperature on held-out logits.

Parameters
scores
np.ndarray of shape (n_samples, n_classes)
Uncalibrated logits. Probabilities are accepted and converted to logits internally via log.
y
np.ndarray of shape (n_samples,)
True labels.
Returns
self
TemperatureScaler
The fitted scaler.
transform (self, scores: np.ndarray) -> np.ndarray

Apply the fitted temperature and return calibrated probabilities.

Parameters
scores
np.ndarray of shape (n_samples, n_classes)
Uncalibrated logits or probabilities.
Returns
proba
np.ndarray of shape (n_samples, n_classes)
Calibrated probabilities, rows summing to one.
predict_proba (self, scores: np.ndarray) -> np.ndarray

Return calibrated probabilities; alias of transform.

Parameters
scores
np.ndarray of shape (n_samples, n_classes)
Uncalibrated logits or probabilities.
Returns
proba
np.ndarray of shape (n_samples, n_classes)
Calibrated probabilities.
get_parameter_schema (cls) -> Dict[str, Any]

Return JSON Schema for constructor parameters.

__repr__ (self) -> str

Return a readable representation of the scaler.

VectorScaler

class uncertainty.calibration.temperature.VectorScaler(Calibrator)

Multiclass calibration with a per-class scale and bias.

Vector scaling generalises TemperatureScaler by learning one weight and one bias per class, z_k \mapsto w_k z_k + b_k. It can correct class-dependent miscalibration that a single temperature cannot, at the cost of 2c parameters and the loss of the accuracy-preserving guarantee.
Constructor 
__init__(
    self,
    max_iter: int = 500,
    learning_rate: float = 0.05,
    tol: float = 1e-07) -> None,
)

Overview

  1. Collect logits on a held-out calibration set.
  2. Fit w and b by gradient descent on the cross-entropy.
  3. At transform time, apply \text{softmax}(w \odot z + b).

Theory

The calibration map is

p_k = \frac{\exp(w_k z_k + b_k)}{\sum_j \exp(w_j z_j + b_j)}

with the objective convex in (w, b), so plain gradient descent with a decaying step reaches the optimum. The gradient of the mean cross-entropy is

\nabla_{w_k} \mathcal{L} = \frac{1}{n} \sum_i (p_{ik} - y_{ik}) z_{ik}, \quad \nabla_{b_k} \mathcal{L} = \frac{1}{n} \sum_i (p_{ik} - y_{ik})

Parameters

max_iter
int = 500
Number of gradient-descent iterations.
learning_rate
float = 0.05
Initial step size; decayed as :math:`1/\sqrt{t}`.
tol
float = 1e-7
Stop when the loss improves by less than this between iterations.

Attributes

weights_
np.ndarray of shape (n_classes,)
Per-class logit scale.
bias_
np.ndarray of shape (n_classes,)
Per-class logit offset.
n_iter_
int
Iterations performed.
classes_
np.ndarray of shape (n_classes,)
Class labels seen during fit.
fitted_
bool
Whether fit has been called.

Notes

Complexity. O(n \cdot c) per iteration.

When to use. Use vector scaling when different classes are miscalibrated in different directions — typically under class imbalance. On small calibration sets it overfits where a single temperature would not, so compare the two with expected_calibration_error on a third split.

References

Guo2017
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. ICML, 1321-1330. :arxiv:`1706.04599`

See Also

TemperatureScaler

Single-parameter, accuracy preserving.
python 
>>> import numpy as np
>>> from tuiml.uncertainty import VectorScaler
>>> rng = np.random.default_rng(0)
>>> y = rng.integers(0, 3, 300)
>>> logits = np.eye(3)[y] * 5.0 + rng.normal(0, 1.0, (300, 3))
>>> scaler = VectorScaler(max_iter=200).fit(logits, y)
>>> proba = scaler.transform(logits)
>>> bool(np.allclose(proba.sum(axis=1), 1.0))
True

Methods

fit (self, scores: np.ndarray, y: np.ndarray) -> 'VectorScaler'

Fit per-class scale and bias on held-out logits.

Parameters
scores
np.ndarray of shape (n_samples, n_classes)
Uncalibrated logits or probabilities.
y
np.ndarray of shape (n_samples,)
True labels.
Returns
self
VectorScaler
The fitted scaler.
transform (self, scores: np.ndarray) -> np.ndarray

Apply the fitted scale and bias and return calibrated probabilities.

Parameters
scores
np.ndarray of shape (n_samples, n_classes)
Uncalibrated logits or probabilities.
Returns
proba
np.ndarray of shape (n_samples, n_classes)
Calibrated probabilities, rows summing to one.
predict_proba (self, scores: np.ndarray) -> np.ndarray

Return calibrated probabilities; alias of transform.

Parameters
scores
np.ndarray of shape (n_samples, n_classes)
Uncalibrated logits or probabilities.
Returns
proba
np.ndarray of shape (n_samples, n_classes)
Calibrated probabilities.
get_parameter_schema (cls) -> Dict[str, Any]

Return JSON Schema for constructor parameters.

__repr__ (self) -> str

Return a readable representation of the scaler.