API Reference / uncertainty / conformal /

venn_abers.py

Venn-Abers predictors: calibrated probability intervals.

Classes

VennAbersCalibrator

class uncertainty.conformal.venn_abers.VennAbersCalibrator(Calibrator)

Probability intervals with a validity guarantee, not point estimates.

Every other calibrator returns a single number and asks you to trust it. A Venn-Abers predictor returns a pair [p_0, p_1] that provably brackets a perfectly calibrated probability — it reports its own calibration uncertainty. The interval is wide where calibration data is thin and narrow where it is plentiful.
Constructor 
__init__(
    self,
    increasing: bool = True) -> None,
)

Overview

  1. For a test score s, hypothetically append it to the calibration
set twice: once labelled 0, once labelled 1.
  1. Run isotonic regression on each augmented set.
  2. The two fitted values at s are p_0 and p_1.
  3. The true calibrated probability is guaranteed to lie between them.

Theory

Under exchangeability, the multiprobability prediction \{p_0, p_1\} is valid: one of the two is the perfectly calibrated probability of the test label. The width p_1 - p_0 shrinks as O(1/n) in the calibration size, so it doubles as a diagnostic — a wide interval says the calibration set does not pin the probability down in that score region.

For a single actionable number, the standard summary is

p = \frac{p_1}{1 - p_0 + p_1}

which transform returns; predict_proba_interval returns the pair itself.

Each hypothesis needs its own isotonic fit, so the implementation runs one PAVA pass per distinct insertion position per direction, using the shared C++ kernel and caching by position. Test points are deliberately not batched into a single fit: a batch of identical hypothesised labels perturbs the isotonic regression far more than the one point the definition adds, which inflates the interval.

Parameters

increasing
bool = True
Whether the calibration map is non-decreasing in the score.

Attributes

calibration_scores_
np.ndarray of shape (n_calibration,)
Sorted calibration scores.
calibration_labels_
np.ndarray of shape (n_calibration,)
Binary labels aligned with calibration_scores_.
classes_
np.ndarray of shape (n_classes,)
Class labels seen during fit.
fitted_
bool
Whether fit has been called.

Notes

Complexity. Fitting is O(n \log n) — just a sort. Predicting is O(u n) for u distinct insertion positions among the m test scores, since each needs its own PAVA pass; repeated positions are cached. The O(n \log n) GCM formulation of Vovk et al. would remove the linear factor and is the natural next optimisation.

When to use. Reach for Venn-Abers when a miscalibrated probability is expensive and you need to know how much to trust the calibration itself — medical triage, pricing, any decision with an asymmetric cost. For a plain point probability with less machinery, use IsotonicCalibrator.

References

Vovk2014
Vovk, V., & Petej, I. (2014). Venn-Abers Predictors. UAI, 829-838. :arxiv:`1211.0025`
Vovk2015
Vovk, V., Petej, I., & Fedorova, V. (2015). Large-Scale Probabilistic Predictors with and without Guarantees of Validity. NeurIPS, 892-900.

See Also

IsotonicCalibrator

Point probabilities from the same PAVA kernel.
SplitConformalClassifier

Set-valued rather than probability-valued output.
python 
>>> import numpy as np
>>> from tuiml.uncertainty import VennAbersCalibrator
>>> rng = np.random.default_rng(0)
>>> scores = rng.uniform(0, 1, 400)
>>> y = (rng.uniform(0, 1, 400) < scores).astype(int)
>>> va = VennAbersCalibrator().fit(scores, y)
>>> p0, p1 = va.predict_proba_interval(np.array([0.2, 0.8]))
>>> bool(np.all(p0 <= p1))
True
>>> proba = va.transform(np.array([0.2, 0.8]))
>>> bool(proba[0] < proba[1])
True

Methods

fit (self, scores: np.ndarray, y: np.ndarray) -> 'VennAbersCalibrator'

Store the calibration set that later hypotheses are appended to.

Parameters
scores
np.ndarray of shape (n_samples,) or (n_samples, 2)
Uncalibrated scores.
y
np.ndarray of shape (n_samples,)
True binary labels.
Returns
self
VennAbersCalibrator
The fitted calibrator.
predict_proba_interval (self, scores: np.ndarray) -> tuple

Return the Venn-Abers probability interval for each score.

Parameters
scores
np.ndarray of shape (n_samples,) or (n_samples, 2)
Uncalibrated scores.
Returns
p0
np.ndarray of shape (n_samples,)
Fitted probability under the hypothesis that the test label is 0.
p1
np.ndarray of shape (n_samples,)
Fitted probability under the hypothesis that the test label is 1. Always at least p0; the width reports calibration uncertainty.
transform (self, scores: np.ndarray) -> np.ndarray

Collapse the probability interval to a single calibrated value.

Parameters
scores
np.ndarray of shape (n_samples,) or (n_samples, 2)
Uncalibrated scores.
Returns
proba
np.ndarray of shape (n_samples,)
The standard p1 / (1 - p0 + p1) summary of the interval.
predict_proba (self, scores: np.ndarray) -> np.ndarray

Return two-column calibrated probabilities.

Parameters
scores
np.ndarray of shape (n_samples,) or (n_samples, 2)
Uncalibrated scores.
Returns
proba
np.ndarray of shape (n_samples, 2)
Calibrated probabilities for the negative and positive class.
get_parameter_schema (cls) -> Dict[str, Any]

Return JSON Schema for constructor parameters.

__repr__ (self) -> str

Return a readable representation of the calibrator.