API Reference / uncertainty / conformal /

cv_plus.py

Cross-validation-based conformal regression (CV+ and jackknife+).

Classes

CVPlusRegressor

class uncertainty.conformal.cv_plus.CVPlusRegressor(ConformalPredictor)

CV+ prediction intervals that use every training sample twice.

Split conformal throws away a quarter of the training data to calibrate. CV+ instead cross-fits: each fold's model is calibrated on the fold it never saw, so all the data trains the ensemble and all of it calibrates. The price is k model fits instead of one.
Constructor 
__init__(
    self,
    estimator: Any,
    alpha: float = 0.1,
    cv: int = 5,
    aggregate: str = 'median',
    shuffle: bool = True,
    random_state: Optional[int] = None) -> None,
)

Overview

  1. Partition the training data into cv folds.
  2. For each fold, fit a model on the other folds and record the absolute
residual of every sample in the held-out fold.
  1. At prediction time, every fold model predicts the test point.
  2. The interval bounds are quantiles of the fold predictions shifted by
the out-of-fold residuals, so a residual is only ever paired with a model that did not train on it.

Theory

Let \hat{f}_{-k(i)} be the model fitted without the fold containing sample i, and R_i = |y_i - \hat{f}_{-k(i)}(x_i)| its out-of-fold residual. The CV+ interval is

C(x) = \left[ q^-_{\alpha}\left\{ \hat{f}_{-k(i)}(x) - R_i \right\},\ \ q^+_{\alpha}\left\{ \hat{f}_{-k(i)}(x) + R_i \right\} \right]

where q^- and q^+ are the \lfloor \alpha(n+1) \rfloor smallest and largest order statistics. Unlike split conformal, the guarantee is the slightly weaker

P\left( Y_{n+1} \in C(X_{n+1}) \right) \geq 1 - 2\alpha

in the worst case, though empirically CV+ achieves close to 1 - \alpha and is never observed to fall below it on real data. The factor-of-two slack is the cost of reusing the data.

Setting cv=n_samples recovers the jackknife+ — see JackknifePlusRegressor.

Parameters

estimator
Regressor
A TuiML regressor. It is deep-copied once per fold, so the instance passed in is never mutated.
alpha
float = 0.1
Miscoverage level.
cv
int = 5
Number of cross-fitting folds. More folds means more training data per model and more compute.
aggregate
{'median', 'mean'} = 'median'
How the fold models are combined for the point prediction returned by predict. Interval bounds always use the order statistics above, independent of this choice.
shuffle
bool = True
Whether to permute the samples before folding. Leave enabled unless the row order is itself meaningful.
random_state
int
Seed for the fold shuffle.

Attributes

estimators_
list of Regressor
One fitted model per fold.
fold_index_
np.ndarray of shape (n_samples,)
Fold assignment of each training sample.
scores_
np.ndarray of shape (n_samples,)
Out-of-fold absolute residuals.
quantile_
float
The corrected residual quantile, reported for comparison with split conformal. The interval itself uses the full residual vector.
fitted_
bool
Whether fit has been called.

Notes

Complexity. cv model fits at training time and cv predictions per test batch, plus O(n \log n) for the order statistics. This is cv times the cost of split conformal in both phases — the reason JackknifePlusRegressor is only practical on small data.

When to use. Use CV+ when data is scarce enough that holding out a calibration split visibly hurts the model, and when cv extra fits are affordable. On large data, split conformal gives a strictly stronger guarantee for a fraction of the compute.

References

Barber2021
Barber, R. F., Candès, E. J., Ramdas, A., & Tibshirani, R. J. (2021). Predictive Inference with the Jackknife+. Annals of Statistics, 49(1), 486-507. :doi:`10.1214/20-AOS1965`

See Also

SplitConformalRegressor

One fit, stronger guarantee, needs a held-out split.
JackknifePlusRegressor

The leave-one-out limit of this method.
coverage_score

Verifies the guarantee empirically.
python 
>>> import numpy as np
>>> from tuiml.uncertainty import CVPlusRegressor
>>> from tuiml.algorithms.trees import DecisionTreeRegressor
>>> rng = np.random.default_rng(0)
>>> X = rng.normal(size=(200, 3))
>>> y = X[:, 0] * 2.0 + rng.normal(0, 0.5, 200)
>>> cp = CVPlusRegressor(DecisionTreeRegressor(max_depth=4), cv=5, random_state=0)
>>> cp.fit(X, y)
CVPlusRegressor(estimator=DecisionTreeRegressor(), alpha=0.1, cv=5)
>>> cp.predict_interval(X[:4]).shape
(4, 2)

Methods

fit (self, X: np.ndarray, y: np.ndarray) -> 'CVPlusRegressor'

Cross-fit the estimator and collect out-of-fold residuals.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Training features.
y
np.ndarray of shape (n_samples,)
Training targets.
Returns
self
CVPlusRegressor
The fitted predictor.
predict (self, X: np.ndarray) -> np.ndarray

Aggregate the fold models into a single point prediction.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test features.
Returns
y_pred
np.ndarray of shape (n_samples,)
Median (or mean) of the fold predictions.
predict_interval (self, X: np.ndarray) -> np.ndarray

Predict CV+ lower and upper bounds for each sample.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test features.
Returns
intervals
np.ndarray of shape (n_samples, 2)
Column 0 holds the lower bound, column 1 the upper bound.
get_parameter_schema (cls) -> Dict[str, Any]

Return JSON Schema for constructor parameters.

__repr__ (self) -> str

Return a readable representation of the predictor.

JackknifePlusRegressor

class uncertainty.conformal.cv_plus.JackknifePlusRegressor(CVPlusRegressor)

Jackknife+ intervals — the leave-one-out limit of CV+.

Each training sample is held out on its own, so every residual comes from a model fitted on all n - 1 remaining samples. This gives the tightest intervals of the family, because each model sees the most data, and the strongest empirical coverage. It also costs n model fits.
Constructor 
__init__(
    self,
    estimator: Any,
    alpha: float = 0.1,
    aggregate: str = 'median') -> None,
)

Overview

  1. For each training sample, fit a model on all the others.
  2. Record that sample's leave-one-out absolute residual.
  3. Form the interval from the order statistics of the leave-one-out
predictions shifted by those residuals, exactly as in CV+.

Theory

Jackknife+ is CVPlusRegressor with cv = n_samples. It inherits the worst-case 1 - 2\alpha bound

P\left( Y_{n+1} \in C(X_{n+1}) \right) \geq 1 - 2\alpha

but is provably at least as tight as CV+ with fewer folds, and in practice covers at very close to the nominal 1 - \alpha.

Note the distinction from the plain jackknife, which shifts a single model's prediction by leave-one-out residuals: that has no coverage guarantee at all and can fail badly when the fitting algorithm is unstable. The "+" is what pairs each residual with its own leave-one-out model.

Parameters

estimator
Regressor
A TuiML regressor, deep-copied once per sample.
alpha
float = 0.1
Miscoverage level.
aggregate
{'median', 'mean'} = 'median'
Fold aggregation for the point prediction.

Attributes

estimators_
list of Regressor
One fitted model per training sample.
scores_
np.ndarray of shape (n_samples,)
Leave-one-out absolute residuals.
fitted_
bool
Whether fit has been called.

Notes

Complexity. n model fits and n predictions per test batch. This is only practical for a few hundred samples with a cheap estimator; beyond that use CVPlusRegressor with cv=10, which is close in tightness and orders of magnitude cheaper.

References

Barber2021
Barber, R. F., Candès, E. J., Ramdas, A., & Tibshirani, R. J. (2021). Predictive Inference with the Jackknife+. Annals of Statistics, 49(1), 486-507. :doi:`10.1214/20-AOS1965`

See Also

CVPlusRegressor

The k-fold version; far cheaper.
SplitConformalRegressor

One fit, needs a held-out split.
python 
>>> import numpy as np
>>> from tuiml.uncertainty import JackknifePlusRegressor
>>> from tuiml.algorithms.linear import LinearRegression
>>> rng = np.random.default_rng(0)
>>> X = rng.normal(size=(60, 2))
>>> y = X[:, 0] * 2.0 + rng.normal(0, 0.3, 60)
>>> cp = JackknifePlusRegressor(LinearRegression(), alpha=0.1)
>>> cp.fit(X, y)
JackknifePlusRegressor(estimator=LinearRegression(), alpha=0.1)
>>> cp.predict_interval(X[:3]).shape
(3, 2)

Methods

fit (self, X: np.ndarray, y: np.ndarray) -> 'JackknifePlusRegressor'

Fit one model per left-out training sample.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Training features.
y
np.ndarray of shape (n_samples,)
Training targets.
Returns
self
JackknifePlusRegressor
The fitted predictor.
__repr__ (self) -> str

Return a readable representation of the predictor.