Metrics for calibration quality and conformal prediction validity.
Functions
coverage_score(y_true: np.ndarray, prediction_sets: np.ndarray, classes: Optional[np.ndarray]=None) -> float
Fraction of samples whose true label falls inside the prediction set.
Parameters
y_true
np.ndarray of shape (n_samples,)
True labels, or true values for interval predictions.
prediction_sets
np.ndarray
Either a boolean membership mask of shape
(n_samples, n_classes) whose columns follow classes, or an interval array of shape (n_samples, 2) holding lower and upper bounds.
classes
np.ndarray of shape (n_classes,)
Column ordering of a boolean mask. Defaults to
np.unique(y_true), which matches the classes_ of every TuiML classifier. Pass it explicitly when the test split may not contain every class.
Returns
coverage
float
Empirical coverage in
[0, 1]. Should be at least 1 - alpha.
python
>>> import numpy as np
>>> from tuiml.uncertainty import coverage_score
>>> intervals = np.array([[0.0, 2.0], [1.0, 3.0], [5.0, 6.0]])
>>> float(coverage_score(np.array([1.0, 4.0, 5.5]), intervals))
0.6666666666666666average_set_size(prediction_sets: np.ndarray) -> float
Mean number of classes per prediction set.
Parameters
prediction_sets
np.ndarray of shape (n_samples, n_classes) of bool
Boolean membership mask.
Returns
size
float
Mean set size. Smaller is more informative at equal coverage.
python
>>> import numpy as np
>>> from tuiml.uncertainty import average_set_size
>>> sets = np.array([[True, False, False], [True, True, False]])
>>> float(average_set_size(sets))
1.5interval_width(intervals: np.ndarray) -> float
Mean width of prediction intervals.
Parameters
intervals
np.ndarray of shape (n_samples, 2)
Lower and upper bounds per sample.
Returns
width
float
Mean interval width. Smaller is sharper at equal coverage.
python
>>> import numpy as np
>>> from tuiml.uncertainty import interval_width
>>> float(interval_width(np.array([[0.0, 2.0], [1.0, 2.0]])))
1.5brier_score(y_true: np.ndarray, y_proba: np.ndarray, pos_label: Optional[int]=None) -> float
Mean squared error between predicted probabilities and outcomes.
The Brier score is a strictly proper scoring rule: it is minimised only by the true probabilities, so it penalises both miscalibration and poor discrimination.
Parameters
y_true
np.ndarray of shape (n_samples,)
True labels.
y_proba
np.ndarray of shape (n_samples,) or (n_samples, n_classes)
Predicted probabilities. A 1-D array is read as the probability of the positive class.
pos_label
int
Label treated as positive in the binary case. Defaults to the largest label present.
Returns
score
float
Brier score in
[0, 2]; lower is better. The binary case is bounded by [0, 1].
python
>>> import numpy as np
>>> from tuiml.uncertainty import brier_score
>>> float(brier_score(np.array([0, 1, 1]), np.array([0.1, 0.9, 0.8])))
0.019999999999999993expected_calibration_error(y_true: np.ndarray, y_proba: np.ndarray, n_bins: int=10, strategy: str='uniform') -> float
Weighted mean gap between confidence and accuracy across bins.
Parameters
y_true
np.ndarray of shape (n_samples,)
True labels.
y_proba
np.ndarray of shape (n_samples,) or (n_samples, n_classes)
Predicted probabilities.
n_bins
int
= 10
Number of confidence bins.
strategy
{'uniform', 'quantile'}
= 'uniform'
'uniform' uses equal-width bins; 'quantile' uses bins holding an equal number of samples, which is more stable when confidences concentrate near one.
Returns
ece
float
Expected calibration error in
[0, 1]; 0 means perfectly calibrated.
python
>>> import numpy as np
>>> from tuiml.uncertainty import expected_calibration_error
>>> y = np.array([0, 0, 1, 1])
>>> proba = np.array([0.0, 0.0, 1.0, 1.0])
>>> float(expected_calibration_error(y, proba, n_bins=5))
0.0maximum_calibration_error(y_true: np.ndarray, y_proba: np.ndarray, n_bins: int=10, strategy: str='uniform') -> float
Largest gap between confidence and accuracy over all bins.
Parameters
y_true
np.ndarray of shape (n_samples,)
True labels.
y_proba
np.ndarray of shape (n_samples,) or (n_samples, n_classes)
Predicted probabilities.
n_bins
int
= 10
Number of confidence bins.
strategy
{'uniform', 'quantile'}
= 'uniform'
Binning strategy.
Returns
mce
float
Maximum calibration error in
[0, 1]; reports the worst-case bin rather than the average, which matters for risk-sensitive deployment.
python
>>> import numpy as np
>>> from tuiml.uncertainty import maximum_calibration_error
>>> y = np.array([0, 0, 1, 1])
>>> proba = np.array([0.0, 0.0, 1.0, 1.0])
>>> float(maximum_calibration_error(y, proba, n_bins=5))
0.0reliability_curve(y_true: np.ndarray, y_proba: np.ndarray, n_bins: int=10, strategy: str='uniform') -> Tuple[np.ndarray, np.ndarray, np.ndarray]
Bin-wise mean confidence and observed accuracy for a reliability diagram.
Parameters
y_true
np.ndarray of shape (n_samples,)
True labels.
y_proba
np.ndarray of shape (n_samples,) or (n_samples, n_classes)
Predicted probabilities.
n_bins
int
= 10
Number of confidence bins.
strategy
{'uniform', 'quantile'}
= 'uniform'
Binning strategy.
Returns
mean_confidence
np.ndarray of shape (n_nonempty_bins,)
Mean predicted confidence per bin.
accuracy
np.ndarray of shape (n_nonempty_bins,)
Observed accuracy per bin.
counts
np.ndarray of shape (n_nonempty_bins,)
Number of samples per bin.
python
>>> import numpy as np
>>> from tuiml.uncertainty import reliability_curve
>>> y = np.array([0, 1, 1, 1])
>>> conf, acc, counts = reliability_curve(y, np.array([0.1, 0.8, 0.9, 0.95]), n_bins=2)
>>> int(counts.sum())
4