API Reference / uncertainty / conformal /

split.py

Split (inductive) conformal prediction for classifiers and regressors.

Classes

SplitConformalClassifier

class uncertainty.conformal.split.SplitConformalClassifier(ConformalPredictor, SetPredictorMixin)

Split conformal prediction sets with a finite-sample coverage guarantee.

Wraps any TuiML classifier and returns, instead of a single label, a set of labels that contains the truth with probability at least 1 - \alpha. The guarantee is distribution-free: it needs no assumption about the model or the data beyond exchangeability, and holds at finite sample size rather than asymptotically.
Constructor 
__init__(
    self,
    estimator: Any,
    alpha: float = 0.1,
    score: str = 'lac',
    calibration_size: float = 0.25,
    random_state: Optional[int] = None) -> None,
)

Overview

  1. Split the training data into a proper training part and a calibration
part.
  1. Fit the wrapped estimator on the proper training part only.
  2. Score each calibration sample by its nonconformity — how poorly the
model predicted its true label.
  1. Take the \lceil (n+1)(1-\alpha) \rceil / n empirical quantile
of those scores as the threshold.
  1. A test label joins the prediction set when its nonconformity falls below
that threshold.

Theory

With the least-ambiguous-set score s(x, y) = 1 - \hat{p}_y(x) and the corrected quantile \hat{q} of the calibration scores, the set

C(x) = \{ y : 1 - \hat{p}_y(x) \leq \hat{q} \}

satisfies

P\left( Y_{n+1} \in C(X_{n+1}) \right) \geq 1 - \alpha

The finite-sample correction is what makes this exact: using the plain 1-\alpha quantile would undercover by roughly 1/n.

Coverage is marginal, averaged over the data draw. It says nothing about coverage for a particular subgroup — see MondrianConformalClassifier for class-conditional validity.

Parameters

estimator
Classifier
A TuiML classifier exposing predict_proba.
alpha
float = 0.1
Miscoverage level; the target coverage is 1 - alpha.
score
{'lac', 'margin'} = 'lac'
Nonconformity score. 'lac' (least ambiguous set-valued classifier) uses :math:`1 - \hat{p}_y` and gives the smallest average set size; 'margin' uses the gap to the best competing class and adapts better to hard samples.
calibration_size
float = 0.25
Fraction of the training data held out for calibration.
random_state
int
Seed for the train/calibration split.

Attributes

classes_
np.ndarray of shape (n_classes,)
Class labels seen during fit.
scores_
np.ndarray of shape (n_calibration,)
Nonconformity scores on the calibration set.
quantile_
float
The conformal threshold derived from scores_.
fitted_
bool
Whether fit has been called.

Notes

Complexity. One estimator fit plus O(n \log n) for the quantile. Prediction costs one predict_proba call plus O(mc).

When to use. Use split conformal whenever a calibrated set is more useful than a point label — triage, selective prediction, or any setting where abstention is allowed. It is the cheapest conformal method: one model fit. When data is scarce and holding out 25% hurts, use CVPlusRegressor or its classification analogue instead. Calibration needs at least \lceil 1/\alpha \rceil - 1 samples; below that no finite threshold can certify the level and the predictor returns the full label set.

References

Vovk2005
Vovk, V., Gammerman, A., & Shafer, G. (2005). Algorithmic Learning in a Random World. Springer. :doi:`10.1007/b106715`
Sadinle2019
Sadinle, M., Lei, J., & Wasserman, L. (2019). Least Ambiguous Set-Valued Classifiers with Bounded Error Levels. Journal of the American Statistical Association, 114(525), 223-234. :doi:`10.1080/01621459.2017.1395341`

See Also

APSConformalClassifier

Adaptive sets with better conditional coverage.
MondrianConformalClassifier

Class-conditional coverage.
SplitConformalRegressor

Prediction intervals.
coverage_score

Verifies the guarantee empirically.
python 
>>> import numpy as np
>>> from tuiml.uncertainty import SplitConformalClassifier
>>> from tuiml.algorithms.trees import DecisionTreeClassifier
>>> rng = np.random.default_rng(0)
>>> X = rng.normal(size=(400, 4))
>>> y = (X[:, 0] + X[:, 1] > 0).astype(int)
>>> cp = SplitConformalClassifier(DecisionTreeClassifier(max_depth=4),
...                               alpha=0.1, random_state=0)
>>> cp.fit(X, y)
SplitConformalClassifier(estimator=DecisionTreeClassifier(), alpha=0.1)
>>> sets = cp.predict_set(X[:5])
>>> sets.shape
(5, 2)

Methods

fit (self, X: np.ndarray, y: np.ndarray) -> 'SplitConformalClassifier'

Fit the estimator on a training split and calibrate on the rest.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Training features.
y
np.ndarray of shape (n_samples,)
Training labels.
Returns
self
SplitConformalClassifier
The fitted predictor.
fit_calibrated (self, X_train: np.ndarray, y_train: np.ndarray, X_cal: np.ndarray, y_cal: np.ndarray) -> 'SplitConformalClassifier'

Fit with an explicit, caller-supplied calibration set.

Parameters
X_train
np.ndarray of shape (n_train, n_features)
Proper training features.
y_train
np.ndarray of shape (n_train,)
Proper training labels.
X_cal
np.ndarray of shape (n_calibration, n_features)
Calibration features, disjoint from the training set.
y_cal
np.ndarray of shape (n_calibration,)
Calibration labels.
Returns
self
SplitConformalClassifier
The fitted predictor.
predict_set (self, X: np.ndarray) -> np.ndarray

Predict a boolean class-membership mask.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test features.
Returns
include
np.ndarray of shape (n_samples, n_classes) of bool
include[i, k] is True when class k is in the prediction set of sample i.
get_parameter_schema (cls) -> Dict[str, Any]

Return JSON Schema for constructor parameters.

SplitConformalRegressor

class uncertainty.conformal.split.SplitConformalRegressor(ConformalPredictor)

Split conformal prediction intervals with guaranteed coverage.

Turns any TuiML regressor into an interval predictor whose intervals contain the truth with probability at least 1 - \alpha, without any distributional assumption. The interval is the point prediction plus and minus a single calibrated radius.
Constructor 
__init__(
    self,
    estimator: Any,
    alpha: float = 0.1,
    calibration_size: float = 0.25,
    normalize: bool = False,
    random_state: Optional[int] = None) -> None,
)

Overview

  1. Split the training data into a proper training part and a calibration
part.
  1. Fit the wrapped regressor on the proper training part only.
  2. Take the absolute residual |y - \hat{y}| on each calibration
sample as its nonconformity score.
  1. The corrected empirical quantile of those residuals is the interval
radius.

Theory

With \hat{q} the corrected quantile of the calibration residuals,

C(x) = \left[ \hat{f}(x) - \hat{q}, \ \hat{f}(x) + \hat{q} \right]

satisfies P(Y_{n+1} \in C(X_{n+1})) \geq 1 - \alpha.

The width is constant across the input space, which is exactly its weakness: a homoscedastic interval over-covers where the model is confident and under-covers where it is not. Setting normalize=True divides residuals by a fitted difficulty estimate to restore local adaptivity, and ConformalizedQuantileRegressor does so directly by conformalising quantile predictions.

Parameters

estimator
Regressor
A TuiML regressor.
alpha
float = 0.1
Miscoverage level; the target coverage is 1 - alpha.
calibration_size
float = 0.25
Fraction of the training data held out for calibration.
normalize
bool = False
Whether to scale residuals by a fitted difficulty model, producing locally adaptive interval widths.
random_state
int
Seed for the train/calibration split.

Attributes

scores_
np.ndarray of shape (n_calibration,)
Absolute (optionally normalised) calibration residuals.
quantile_
float
Interval radius derived from scores_.
difficulty_estimator_
Regressor or None
Model of the log absolute residual, fitted only when normalize.
fitted_
bool
Whether fit has been called.

Notes

Complexity. One estimator fit (two when normalize=True) plus O(n \log n) for the quantile.

When to use. This is the default interval method: cheapest to fit and exactly valid. Prefer CVPlusRegressor when data is too scarce to hold out a calibration split, and ConformalizedQuantileRegressor when the noise is strongly heteroscedastic.

References

Lei2018
Lei, J., G'Sell, M., Rinaldo, A., Tibshirani, R. J., & Wasserman, L. (2018). Distribution-Free Predictive Inference for Regression. Journal of the American Statistical Association, 113(523), 1094-1111. :doi:`10.1080/01621459.2017.1307116`
Papadopoulos2002
Papadopoulos, H., Proedrou, K., Vovk, V., & Gammerman, A. (2002). Inductive Confidence Machines for Regression. ECML, 345-356. :doi:`10.1007/3-540-36755-1_29`

See Also

CVPlusRegressor

Uses all data, no held-out split.
ConformalizedQuantileRegressor

Heteroscedastic intervals.
interval_width

Compares interval sharpness.
python 
>>> import numpy as np
>>> from tuiml.uncertainty import SplitConformalRegressor
>>> from tuiml.algorithms.trees import DecisionTreeRegressor
>>> rng = np.random.default_rng(0)
>>> X = rng.normal(size=(400, 3))
>>> y = X[:, 0] * 2.0 + rng.normal(0, 0.5, 400)
>>> cp = SplitConformalRegressor(DecisionTreeRegressor(max_depth=5),
...                              alpha=0.1, random_state=0)
>>> cp.fit(X, y)
SplitConformalRegressor(estimator=DecisionTreeRegressor(), alpha=0.1)
>>> intervals = cp.predict_interval(X[:5])
>>> intervals.shape
(5, 2)

Methods

fit (self, X: np.ndarray, y: np.ndarray) -> 'SplitConformalRegressor'

Fit the regressor on a training split and calibrate on the rest.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Training features.
y
np.ndarray of shape (n_samples,)
Training targets.
Returns
self
SplitConformalRegressor
The fitted predictor.
predict_interval (self, X: np.ndarray) -> np.ndarray

Predict lower and upper bounds for each sample.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test features.
Returns
intervals
np.ndarray of shape (n_samples, 2)
Column 0 holds the lower bound, column 1 the upper bound.
get_parameter_schema (cls) -> Dict[str, Any]

Return JSON Schema for constructor parameters.