API Reference / automl /

ensembling.py

Greedy ensemble selection over an AutoML trial pool.

A search leaves behind dozens of fitted models and keeps one. Caruana et al. (2004) showed that a weighted average of the pool, assembled greedily, is reliably better than its single best member and costs nothing more to build: the models are already fitted, and selection works purely on their stored validation predictions.

This module implements that procedure and the small combiner that carries the result. It is not a replacement for VotingClassifier, which builds an ensemble by fitting a declared list of learners with fixed weights; here the members are already fitted, the pool is whatever the search happened to produce, and the weights are what selection discovers.

References

Caruana2004
Caruana, R., Niculescu-Mizil, A., Crew, G., & Ksikes, A. (2004). Ensemble selection from libraries of models. Proceedings of the 21st International Conference on Machine Learning (ICML), 18. :doi:`10.1145/1015330.1015432`

Classes

GreedyEnsemble

class automl.ensembling.GreedyEnsemble

A weighted average of already-fitted models, chosen by selection.

Built by greedy_selection from an AutoML trial pool and used as the final predictor when the ensemble scores better than the best single trial. It holds fitted models, so it has no fit: everything it needs was learned during the search.
Constructor 
__init__(
    self,
    models: Sequence[Any],
    weights: np.ndarray,
    task: str = 'classification',
    classes: Optional[np.ndarray] = None,
)

Parameters

models
list of Algorithm
The fitted pool members, in the same order as the predictions passed to greedy_selection.
weights
np.ndarray of shape (n_models,)
Selection frequencies. Members with weight zero are ignored.
task
{"classification", "regression"}
Whether to average class probabilities or predicted values.
classes
np.ndarray
Class labels, in the column order of the averaged probability matrix. Required for classification.

Attributes

members_
list of tuple
The (weight, model) pairs that actually contribute, heaviest first.

See Also

greedy_selection

Builds the weights.
VotingClassifier

Fits and combines a declared list of learners instead of a discovered pool.
python 
>>> import numpy as np
>>> from tuiml.algorithms.trees import DecisionTreeClassifier
>>> from tuiml.automl.ensembling import GreedyEnsemble
>>> from tuiml.datasets import load_iris
>>> data = load_iris()
>>> trees = [DecisionTreeClassifier(max_depth=d).fit(data.X, data.y)
...          for d in (1, 3)]
>>> ensemble = GreedyEnsemble(trees, np.array([0.25, 0.75]),
...                           task="classification",
...                           classes=np.unique(data.y))
>>> ensemble.predict(data.X[:3]).tolist()
[0, 0, 0]

Methods

predict_proba (self, X: np.ndarray) -> np.ndarray

Predict weighted-average class probabilities.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Input samples.
Returns
proba
np.ndarray of shape (n_samples, n_classes)
Averaged class probabilities, columns ordered as classes_.
Raises
RuntimeError
If the ensemble was built for regression.
predict (self, X: np.ndarray) -> np.ndarray

Predict labels (classification) or values (regression).

Parameters
X
np.ndarray of shape (n_samples, n_features)
Input samples.
Returns
y_pred
np.ndarray of shape (n_samples,)
Ensemble predictions.
describe (self) -> List[Dict[str, Any]]

Return the contributing members as plain dicts.

Returns
rows
list of dict
One {"model", "weight"} dict per contributing member, heaviest first.
__repr__ (self) -> str

Return a short summary naming the members and their weights.

Functions

Func

greedy_selection

Line 29
greedy_selection(predictions: Sequence[np.ndarray], y_true: np.ndarray, scorer: Callable[..., float], *, n_rounds: int=25, decode: Optional[Callable[..., np.ndarray]]=None) -> Tuple[np.ndarray, float, List[int]]

Select an ensemble greedily, with replacement, from a pool of models.

Overview

  1. Start from the empty ensemble.
  2. In each round, try appending every pool member to the current
ensemble, score the resulting average, and keep the best append.
  1. Repeat for n_rounds. Selection is with replacement: a strong
model can be added several times, which is exactly how the procedure expresses a non-uniform weight without ever solving for one.
  1. Return the prefix of the selection sequence with the highest score,
so that rounds which only made things worse are discarded.

Theory

After k rounds the ensemble prediction is the running mean of the selected members,

\bar{p}_k = \frac{1}{k} \sum_{i=1}^{k} p_{s_i},

so the number of times model j was selected, divided by k, is its weight. Because each round maximises the score of \bar{p}_k directly, selection optimises the target metric itself rather than a differentiable surrogate, and it cannot do worse than the single best model, which is always available as the first pick.

Parameters

predictions
sequence of np.ndarray
One validation prediction array per pool member, all the same shape: (n_samples, n_classes) probabilities for classification, or (n_samples,) values for regression.
y_true
np.ndarray of shape (n_samples,)
Validation targets.
scorer
callable
Function (y_true, y_pred) -> float, higher is better.
n_rounds
int = 25
Number of greedy rounds, i.e. the ensemble's size counted with multiplicity.
decode
callable
Maps an averaged prediction array to the form scorer expects, e.g. probabilities to hard class labels. Identity when omitted.

Returns

weights
np.ndarray of shape (n_models,)
Selection frequency per pool member, summing to 1.
score
float
The score of the returned ensemble on the validation predictions.
order
list of int
The indices selected, in the order they were picked.

Raises

ValueError
If predictions is empty.
python 
>>> import numpy as np
>>> from tuiml.automl.ensembling import greedy_selection
>>> y = np.array([0, 0, 1, 1])
>>> good = np.array([[.9, .1], [.8, .2], [.2, .8], [.1, .9]])
>>> bad = np.array([[.4, .6], [.3, .7], [.6, .4], [.7, .3]])
>>> accuracy = lambda t, p: float(np.mean(t == p))
>>> weights, score, order = greedy_selection(
...     [good, bad], y, accuracy, n_rounds=4,
...     decode=lambda p: p.argmax(axis=1))
>>> weights.tolist()
[1.0, 0.0]
>>> score
1.0
Func

align_proba

Line 293
align_proba(model: Any, X: np.ndarray, classes: np.ndarray) -> np.ndarray

Return a model's class probabilities in a fixed column order.

Pool members may know a different subset of classes, or the same classes in a different order, than the ensemble as a whole. Averaging their raw matrices would silently add unrelated columns together, so every matrix is projected onto the ensemble's classes first.

Parameters

model
Algorithm
A fitted classifier.
X
np.ndarray of shape (n_samples, n_features)
Input samples.
classes
np.ndarray of shape (n_classes,)
The target column order.

Returns

proba
np.ndarray of shape (n_samples, n_classes)
Probabilities, one column per entry of classes.