API Reference / uncertainty / conformal /

aps.py

Adaptive prediction sets (APS) and their regularised variant (RAPS).

Classes

APSConformalClassifier

class uncertainty.conformal.aps.APSConformalClassifier(SplitConformalClassifier)

Adaptive prediction sets that grow where the model is uncertain.

The LAC score behind SplitConformalClassifier minimises average set size, but it does so by leaving hard samples under-covered and easy ones over-covered. APS instead accumulates probability mass down the sorted class ranking, producing small sets on easy inputs and large sets on ambiguous ones — much better conditional coverage at a modest cost in average size.
Constructor 
__init__(
    self,
    estimator: Any,
    alpha: float = 0.1,
    randomized: bool = True,
    calibration_size: float = 0.25,
    random_state: Optional[int] = None) -> None,
)

Overview

  1. Sort each sample's class probabilities in decreasing order.
  2. The nonconformity of the true label is the total probability mass down
to and including it, minus a uniform random fraction of its own mass.
  1. Calibrate the corrected quantile of those scores as usual.
  2. A test set includes classes in rank order until the accumulated mass
exceeds the threshold.

Theory

For sorted probabilities \hat{p}_{(1)} \geq \dots \geq \hat{p}_{(c)} and the true label at rank r, the score is

s = \sum_{j=1}^{r} \hat{p}_{(j)} - u \cdot \hat{p}_{(r)}, \quad u \sim \mathrm{Uniform}(0, 1)

The randomised term u is what makes coverage exact rather than merely conservative: without it the discrete jumps between ranks force the set to over-cover. Set randomized=False for deterministic, reproducible sets at the cost of slight over-coverage.

Parameters

estimator
Classifier
A TuiML classifier exposing predict_proba.
alpha
float = 0.1
Miscoverage level.
randomized
bool = True
Whether to apply the uniform randomisation that makes coverage exact.
calibration_size
float = 0.25
Fraction of the training data held out for calibration.
random_state
int
Seed for the split and for the randomisation term.

Attributes

classes_
np.ndarray of shape (n_classes,)
Class labels seen during fit.
scores_
np.ndarray of shape (n_calibration,)
Cumulative-mass nonconformity scores.
quantile_
float
The conformal threshold on accumulated probability mass.
fitted_
bool
Whether fit has been called.

Notes

Complexity. O(n c \log c) for the per-sample sort, on top of one estimator fit.

When to use. Prefer APS over LAC whenever coverage must hold across subgroups, not just on average — the LAC set is smaller overall but systematically fails the hard tail. Its known weakness is a long tail of very large sets when the probability estimates are noisy; RAPS fixes exactly that.

References

Romano2020
Romano, Y., Sesia, M., & Candès, E. J. (2020). Classification with Valid and Adaptive Coverage. NeurIPS, 3581-3591. :arxiv:`2006.02544`

See Also

RAPSConformalClassifier

Penalises the long tail of large sets.
SplitConformalClassifier

Smaller sets, worse conditional coverage.
MondrianConformalClassifier

Exact per-class coverage.
python 
>>> import numpy as np
>>> from tuiml.uncertainty import APSConformalClassifier
>>> from tuiml.algorithms.trees import DecisionTreeClassifier
>>> rng = np.random.default_rng(0)
>>> X = rng.normal(size=(400, 4))
>>> y = (X[:, 0] + X[:, 1] > 0).astype(int)
>>> cp = APSConformalClassifier(DecisionTreeClassifier(max_depth=4),
...                             alpha=0.1, random_state=0)
>>> cp.fit(X, y)
APSConformalClassifier(estimator=DecisionTreeClassifier(), alpha=0.1)
>>> cp.predict_set(X[:5]).shape
(5, 2)

Methods

predict_set (self, X: np.ndarray) -> np.ndarray

Predict adaptive prediction sets.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test features.
Returns
include
np.ndarray of shape (n_samples, n_classes) of bool
Class-membership mask; sets grow on ambiguous samples.
get_parameter_schema (cls) -> Dict[str, Any]

Return JSON Schema for constructor parameters.

RAPSConformalClassifier

class uncertainty.conformal.aps.RAPSConformalClassifier(APSConformalClassifier)

Regularised adaptive prediction sets — APS without the long tail.

APS produces well-adapted sets but occasionally enormous ones, because noisy tail probabilities let the cumulative mass creep past the threshold over many low-ranked classes. RAPS adds a penalty that grows with rank, so admitting a badly-ranked class becomes progressively more expensive. The result keeps APS's conditional coverage while sharply bounding the worst-case set size.
Constructor 
__init__(
    self,
    estimator: Any,
    alpha: float = 0.1,
    lambda_penalty: float = 0.01,
    k_reg: int = 1,
    randomized: bool = True,
    calibration_size: float = 0.25,
    random_state: Optional[int] = None) -> None,
)

Overview

  1. Compute the APS cumulative-mass score.
  2. Add lambda_penalty for every class ranked beyond k_reg.
  3. Calibrate and predict exactly as APS does, with the penalty applied on
both sides so the guarantee is preserved.

Theory

With the true label at rank r, the RAPS score is

s = \sum_{j=1}^{r} \hat{p}_{(j)} - u \cdot \hat{p}_{(r)} + \lambda \cdot \max(0,\ r - k_{\text{reg}})

Because the penalty is a deterministic function of rank and is applied identically at calibration and prediction time, the exchangeability argument is untouched — the 1 - \alpha guarantee still holds. The penalty only reshapes which sets achieve it.

Parameters

estimator
Classifier
A TuiML classifier exposing predict_proba.
alpha
float = 0.1
Miscoverage level.
lambda_penalty
float = 0.01
Penalty added per rank beyond k_reg. Larger values shrink the tail harder; too large and every set collapses to k_reg classes.
k_reg
int = 1
Rank beyond which the penalty applies. A good default is the typical number of plausible classes.
randomized
bool = True
Whether to apply the uniform randomisation term.
calibration_size
float = 0.25
Fraction of the training data held out for calibration.
random_state
int
Seed for the split and the randomisation.

Attributes

classes_
np.ndarray of shape (n_classes,)
Class labels seen during fit.
scores_
np.ndarray of shape (n_calibration,)
Penalised cumulative-mass scores.
quantile_
float
The conformal threshold.
fitted_
bool
Whether fit has been called.

Notes

Complexity. Identical to APS: O(n c \log c).

When to use. Use RAPS on problems with many classes, where APS's tail of huge sets makes the output unusable. On binary or few-class problems the penalty has little to bite on and plain APS is simpler. Tune lambda_penalty and k_reg on a validation split against average_set_size at fixed coverage.

References

Angelopoulos2021
Angelopoulos, A. N., Bates, S., Malik, J., & Jordan, M. I. (2021). Uncertainty Sets for Image Classifiers using Conformal Prediction. ICLR. :arxiv:`2009.14193`

See Also

APSConformalClassifier

The unregularised version.
average_set_size

The quantity this class minimises.
python 
>>> import numpy as np
>>> from tuiml.uncertainty import RAPSConformalClassifier
>>> from tuiml.algorithms.trees import DecisionTreeClassifier
>>> rng = np.random.default_rng(0)
>>> X = rng.normal(size=(600, 4))
>>> y = rng.integers(0, 4, 600)
>>> cp = RAPSConformalClassifier(DecisionTreeClassifier(max_depth=3),
...                              alpha=0.2, lambda_penalty=0.05, random_state=0)
>>> cp.fit(X, y)
RAPSConformalClassifier(estimator=DecisionTreeClassifier(), alpha=0.2)
>>> cp.predict_set(X[:5]).shape
(5, 4)

Methods

get_parameter_schema (cls) -> Dict[str, Any]

Return JSON Schema for constructor parameters.