API Reference / algorithms / gradient_boosting /

ngboost.py

NGBoost - Natural Gradient Boosting for probabilistic prediction.

A native, pure-NumPy implementation of Duan et al. (ICML 2020). Unlike the other members of gradient_boosting, nothing here wraps an external boosting library: the base learners are TuiML's own DecisionTreeRegressor.

Classes

NGBoostRegressor

class algorithms.gradient_boosting.ngboost.NGBoostRegressor(_NGBoostBase, Regressor)

NGBoost fits a whole predictive distribution, not just a mean.

Ordinary gradient boosting drives one number per sample towards the truth. NGBoost drives all the parameters of a probability distribution — for the Normal, both \mu and \log \sigma — using the natural gradient of a proper scoring rule. The natural gradient is the ordinary gradient premultiplied by the inverse Riemannian metric of the score, which makes each step invariant to how the distribution happens to be parameterised. Boosting on the raw gradient instead is badly conditioned: the \mu and \log \sigma directions live on different scales, and the fit drifts towards whichever one the parameterisation happens to favour.
Constructor 
__init__(
    self,
    dist: str = 'normal',
    scoring: str = 'log',
    n_estimators: int = 100,
    learning_rate: float = 0.1,
    max_depth: int = 3,
    min_samples_split: int = 2,
    min_samples_leaf: int = 1,
    natural_gradient: bool = True,
    minibatch_frac: float = 1.0,
    tol: float = 1e-05,
    random_state: Optional[int] = None,
    verbose: bool = False,
)

Overview

  1. Initialise every sample with the marginal MLE of the distribution.
  2. At each stage compute the natural gradient
I(\theta)^{-1} \nabla_\theta S(\theta, y) per sample.
  1. Fit one DecisionTreeRegressor per
distribution parameter against that natural gradient.
  1. Line-search the stage scaling that minimises the mean score, shrink it
by learning_rate, and take the step.

Theory

For a proper scoring rule S the induced Riemannian metric is

I(\theta) = \mathbb{E}_{y \sim P_\theta} \left[ \nabla_\theta S(\theta, y)\, \nabla_\theta S(\theta, y)^T \right]

which for the log score is the Fisher information. With the Normal parameterised as \theta = (\mu, \log \sigma) and z = (y - \mu)/\sigma, the negative log-likelihood has gradient

\nabla_\theta S = \left( -\frac{z}{\sigma},\; 1 - z^2 \right)

and Fisher information

I(\theta) = \begin{pmatrix} \sigma^{-2} & 0 \ 0 & 2 \end{pmatrix}

so the natural gradient collapses to the strikingly simple, scale-free

\tilde{\nabla}_\theta S = \left( \mu - y,\; \tfrac{1}{2}(1 - z^2) \right)

The CRPS alternative,

\mathrm{CRPS}(\theta, y) = \sigma \left[ z(2\Phi(z) - 1) + 2\varphi(z) - \pi^{-1/2} \right]

is also proper, and its own metric is used when scoring="crps".

Parameters

dist
str = "normal"
Predictive distribution: "normal", "lognormal" (requires strictly positive targets) or "exponential" (requires non-negative targets).
scoring
str = "log"
Proper scoring rule. "log" is the negative log-likelihood; "crps" is the continuous ranked probability score and is available for dist="normal" and "lognormal".
n_estimators
int = 100
Maximum number of boosting stages.
learning_rate
float = 0.1
Shrinkage applied to each stage on top of the line-searched scaling.
max_depth
int = 3
Maximum depth of each base learner.
min_samples_split
int = 2
Minimum samples required to split an internal node of a base learner.
min_samples_leaf
int = 1
Minimum samples required at a leaf of a base learner.
natural_gradient
bool = True
Boost on the natural gradient. Set to False to recover ordinary (non-invariant) gradient boosting of the same score.
minibatch_frac
float = 1.0
Fraction of rows sampled without replacement per stage.
tol
float = 1e-5
Stop once a stage improves the mean training score by less than this.
random_state
int
Seed for the minibatch sampler and the base-learner tie-breaking.
verbose
bool = False
Print the training score after each stage.

Attributes

dist_
object
The fitted distribution helper.
init_params_
np.ndarray of shape (n_params,)
Marginal parameter vector the boosting started from.
estimators_
list of list of DecisionTreeRegressor
One inner list per stage, one tree per distribution parameter.
scalings_
list of float
Per-stage step size, learning_rate times the line-searched scaling.
train_score_
list of float
Mean training score after each stage, including the initialisation.
n_estimators_
int
Number of stages actually fitted (may be below n_estimators when the tol early stop triggers).
n_features_in_
int
Number of features seen during fit().

Notes

Complexity:

  • Fitting: O(M \cdot k \cdot n p \log n) for M stages and
k distribution parameters, plus a constant-size line search per stage.
  • Prediction: O(M \cdot k \cdot d) per sample for depth
d.

When to use NGBoostRegressor:

  • When a calibrated predictive interval matters as much as the point
estimate — risk pricing, forecasting, anything downstream of a decision threshold.
  • When the noise is heteroscedastic, so a single global error bar would
misstate the uncertainty for most samples.
  • Prefer a plain boosted regressor when only the conditional mean is
wanted: NGBoost pays for its second head in accuracy and runtime.

References

Duan2020
Duan, T., Avati, A., Ding, D.Y., Thai, K.K., Basu, S., Ng, A.Y., Schuler, A. (2020). NGBoost: Natural Gradient Boosting for Probabilistic Prediction. Proceedings of the 37th International Conference on Machine Learning (ICML), PMLR 119, 2690-2700. DOI: 10.48550/arXiv.1910.03225
Amari1998
Amari, S. (1998). Natural Gradient Works Efficiently in Learning. Neural Computation, 10(2), 251-276. DOI: 10.1162/089976698300017746
Gneiting2007
Gneiting, T., Raftery, A.E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102(477), 359-378. DOI: 10.1198/016214506000001437

See Also

NGBoostClassifier

Categorical counterpart.
XGBoostRegressor

Point-estimate boosting.
DecisionTreeRegressor

The base learner.
python 
>>> import numpy as np
>>> from tuiml.algorithms.gradient_boosting import NGBoostRegressor
>>> rng = np.random.default_rng(0)
>>> X = rng.uniform(-3, 3, size=(300, 1))
>>> # Noise grows with x: a single global error bar cannot describe this.
>>> y = X[:, 0] ** 2 + rng.normal(0, 0.2 + 0.5 * np.abs(X[:, 0]))
>>> model = NGBoostRegressor(n_estimators=60, random_state=0).fit(X, y)
>>> params = model.predict_dist(X)
>>> sorted(params)
['loc', 'scale']
>>> # The fitted scale tracks the true |x|-driven noise.
>>> bool(np.corrcoef(params["scale"], np.abs(X[:, 0]))[0, 1] > 0.7)
True
>>> lower, upper = model.predict_interval(X, alpha=0.05).T
>>> bool(np.mean((y >= lower) & (y <= upper)) > 0.85)
True

Methods

get_parameter_schema (cls) -> Dict[str, Dict[str, Any]]

Return JSON Schema for constructor parameters.

get_capabilities (cls) -> List[str]

Return supported capabilities.

get_complexity (cls) -> str

Return complexity analysis.

get_references (cls) -> List[str]

Return academic citations.

fit (self, X: np.ndarray, y: np.ndarray) -> 'NGBoostRegressor'

Fit the boosted distributional model.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Training data.
y
np.ndarray of shape (n_samples,)
Continuous targets.
Returns
self
NGBoostRegressor
Fitted estimator.
predict (self, X: np.ndarray) -> np.ndarray

Predict the conditional mean of the fitted distribution.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test samples.
Returns
y_pred
np.ndarray of shape (n_samples,)
Predicted mean for each sample.
predict_dist (self, X: np.ndarray) -> Dict[str, np.ndarray]

Return the predicted distribution parameters.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test samples.
Returns
params
dict of str to np.ndarray
{"loc", "scale"} for normal and lognormal (the log-normal's parameters describe :math:`\log y`), {"scale"} for exponential. Each value has shape (n_samples,).
predict_interval (self, X: np.ndarray, alpha: float=0.05) -> np.ndarray

Return equal-tailed prediction intervals.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test samples.
alpha
float = 0.05
Miscoverage level; alpha=0.05 gives a nominal 95% interval.
Returns
interval
np.ndarray of shape (n_samples, 2)
Column 0 is the lower bound, column 1 the upper bound.
score_samples (self, X: np.ndarray, y: np.ndarray) -> np.ndarray

Return the per-sample value of the fitted scoring rule.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test samples.
y
np.ndarray of shape (n_samples,)
True targets.
Returns
score
np.ndarray of shape (n_samples,)
Lower is better.
__repr__ (self) -> str

Return a concise representation of the estimator.

NGBoostClassifier

class algorithms.gradient_boosting.ngboost.NGBoostClassifier(_NGBoostBase, Classifier)

NGBoost for categorical targets: boosting on natural-gradient logits.

The classification counterpart of NGBoostRegressor. The predictive distribution is categorical over K classes, parameterised by K - 1 logits against a reference class, and the boosting stages follow the natural gradient of the log score — the ordinary gradient premultiplied by the inverse Fisher information of the multinomial. Binary problems are the K = 2 case and reduce to a Bernoulli with a single logit.
Constructor 
__init__(
    self,
    n_estimators: int = 100,
    learning_rate: float = 0.1,
    max_depth: int = 3,
    min_samples_split: int = 2,
    min_samples_leaf: int = 1,
    natural_gradient: bool = True,
    minibatch_frac: float = 1.0,
    tol: float = 1e-05,
    random_state: Optional[int] = None,
    verbose: bool = False,
)

Overview

  1. Initialise every sample with the marginal class log-odds.
  2. Compute the natural gradient of the negative log-likelihood per sample.
  3. Fit one DecisionTreeRegressor per logit.
  4. Line-search the stage scaling, shrink by learning_rate, step.

Theory

With class 0 as reference, p = \mathrm{softmax}(0, \eta) and

\nabla_\eta S = p_{1:K} - \mathbb{1}\{y\}, \qquad I(\eta) = \mathrm{diag}(p_{1:K}) - p_{1:K} p_{1:K}^T

The reduced Fisher information is non-singular and its inverse is available in closed form,

I(\eta)^{-1} = \mathrm{diag}(p_j^{-1}) + p_0^{-1} \mathbf{1} \mathbf{1}^T

so the natural gradient needs no linear solve. For K = 2 this collapses to (p - y) / (p(1 - p)), the Bernoulli case.

Parameters

n_estimators
int = 100
Maximum number of boosting stages.
learning_rate
float = 0.1
Shrinkage applied to each stage on top of the line-searched scaling.
max_depth
int = 3
Maximum depth of each base learner.
min_samples_split
int = 2
Minimum samples required to split an internal node of a base learner.
min_samples_leaf
int = 1
Minimum samples required at a leaf of a base learner.
natural_gradient
bool = True
Boost on the natural gradient rather than the ordinary one.
minibatch_frac
float = 1.0
Fraction of rows sampled without replacement per stage.
tol
float = 1e-5
Stop once a stage improves the mean training score by less than this.
random_state
int
Seed for the minibatch sampler and the base-learner tie-breaking.
verbose
bool = False
Print the training score after each stage.

Attributes

classes_
np.ndarray of shape (n_classes,)
Sorted class labels seen during fit().
dist_
object
The fitted categorical distribution helper.
init_params_
np.ndarray of shape (n_classes - 1,)
Marginal log-odds the boosting started from.
estimators_
list of list of DecisionTreeRegressor
One inner list per stage, one tree per logit.
scalings_
list of float
Per-stage step size.
train_score_
list of float
Mean training negative log-likelihood after each stage.
n_estimators_
int
Number of stages actually fitted.
n_features_in_
int
Number of features seen during fit().

Notes

Complexity:

  • Fitting: O(M (K-1) n p \log n) for M stages.
  • Prediction: O(M (K-1) d) per sample.
When to use NGBoostClassifier:
  • When well-behaved class probabilities matter more than raw accuracy.
  • When the natural-gradient step's parameterisation invariance is wanted
on a multiclass problem with very unbalanced classes.
  • Prefer a plain boosted classifier when only the argmax is consumed.

References

Duan2020
Duan, T., Avati, A., Ding, D.Y., Thai, K.K., Basu, S., Ng, A.Y., Schuler, A. (2020). NGBoost: Natural Gradient Boosting for Probabilistic Prediction. Proceedings of the 37th International Conference on Machine Learning (ICML), PMLR 119, 2690-2700. DOI: 10.48550/arXiv.1910.03225
Amari1998
Amari, S. (1998). Natural Gradient Works Efficiently in Learning. Neural Computation, 10(2), 251-276. DOI: 10.1162/089976698300017746

See Also

NGBoostRegressor

Continuous counterpart.
XGBoostClassifier

Point-estimate boosting.
DecisionTreeRegressor

The base learner.
python 
>>> import numpy as np
>>> from tuiml.algorithms.gradient_boosting import NGBoostClassifier
>>> rng = np.random.default_rng(0)
>>> X = rng.normal(size=(200, 3))
>>> y = (X[:, 0] + 0.5 * X[:, 1] > 0).astype(int)
>>> model = NGBoostClassifier(n_estimators=40, random_state=0).fit(X, y)
>>> model.classes_.tolist()
[0, 1]
>>> proba = model.predict_proba(X)
>>> proba.shape
(200, 2)
>>> bool(np.allclose(proba.sum(axis=1), 1.0))
True
>>> float((model.predict(X) == y).mean()) > 0.9
True

Methods

get_parameter_schema (cls) -> Dict[str, Dict[str, Any]]

Return JSON Schema for constructor parameters.

get_capabilities (cls) -> List[str]

Return supported capabilities.

get_complexity (cls) -> str

Return complexity analysis.

get_references (cls) -> List[str]

Return academic citations.

fit (self, X: np.ndarray, y: np.ndarray) -> 'NGBoostClassifier'

Fit the boosted categorical model.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Training data.
y
np.ndarray of shape (n_samples,)
Class labels.
Returns
self
NGBoostClassifier
Fitted estimator.
predict_proba (self, X: np.ndarray) -> np.ndarray

Predict class probabilities.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test samples.
Returns
proba
np.ndarray of shape (n_samples, n_classes)
Row-stochastic class probabilities, ordered as classes_.
predict (self, X: np.ndarray) -> np.ndarray

Predict class labels.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test samples.
Returns
y_pred
np.ndarray of shape (n_samples,)
Predicted labels drawn from classes_.
predict_dist (self, X: np.ndarray) -> Dict[str, np.ndarray]

Return the predicted distribution parameters.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test samples.
Returns
params
dict of str to np.ndarray
{"proba": array of shape (n_samples, n_classes)} — the parameters of the predictive categorical distribution.
predict_interval (self, X: np.ndarray, alpha: float=0.05) -> np.ndarray

Return the categorical analogue of a prediction interval.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test samples.
alpha
float = 0.05
Miscoverage level.
Returns
mask
np.ndarray of shape (n_samples, n_classes), dtype=bool
mask[i, k] is True when class classes_[k] belongs to the credible set of sample i. Every row contains at least the most probable class.
score_samples (self, X: np.ndarray, y: np.ndarray) -> np.ndarray

Return the per-sample negative log-likelihood.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test samples.
y
np.ndarray of shape (n_samples,)
True labels.
Returns
score
np.ndarray of shape (n_samples,)
Lower is better.
__repr__ (self) -> str

Return a concise representation of the estimator.