API Reference / evaluation / tuning /

hyperband.py

Hyperband - successive halving without having to guess the schedule.

Classes

HyperbandSearchCV

class evaluation.tuning.hyperband.HyperbandSearchCV(SuccessiveHalvingSearchCV)

Successive halving run at several aggression levels, and the best kept.

SuccessiveHalvingSearchCV forces a choice nobody can make well in advance: many candidates on little data, or few candidates on plenty? Guess too aggressive and a slow-starting configuration is killed in round one; too conservative and the budget is wasted on obvious losers. Hyperband refuses the choice and runs the whole spectrum, spending a comparable budget on each.
Constructor 
__init__(
    self,
    estimator,
    param_distributions,
    factor: int = 3,
    resource: str = 'n_samples',
    min_resource: Union[int, str] = 'auto',
    max_resource: Union[int, str] = 'auto',
    n_brackets: Union[int, str] = 'auto',
    scoring: Union[str, Any] = 'accuracy',
    cv: int = 5,
    refit: bool = True,
    verbose: int = 0,
    n_jobs: int = 1,
    random_seed: Optional[int] = None,
    progress_callback: Optional[Any] = None,
)

Overview

  1. Derive a set of brackets from the resource range. The first is
maximally aggressive — the largest pool at the smallest resource; the last runs a handful of candidates at full resource, which is plain random search.
  1. Run successive halving inside each bracket.
  2. Report the best configuration across all of them.

Theory

With R the maximum resource and \eta the elimination factor, there are s_{\max} = \lfloor \log_\eta R \rfloor + 1 brackets. Bracket s starts with

n_s = \left\lceil \frac{s_{\max}}{s + 1} \eta^{s} \right\rceil \quad \text{candidates at resource} \quad r_s = R \eta^{-s}

so aggression falls and per-candidate budget rises as s decreases. Each bracket costs about the same, and the total is roughly s_{\max} times a single successive-halving run.

That is the trade: Hyperband spends a constant factor more than one well-chosen halving schedule, in exchange for never needing to have chosen it. Because the last bracket is ordinary random search at full resource, Hyperband cannot do much worse than random search given the same budget — which is the guarantee that makes it a safe default.

Parameters

estimator
Algorithm
Model template to tune.
param_distributions
dict, ParameterDistribution or ParameterGrid
Space to sample candidates from. A plain dict is accepted and wrapped, as RandomSearchCV does.
factor
int = 3
Elimination factor, shared by every bracket.
resource
str = 'n_samples'
What is scaled between rounds; see SuccessiveHalvingSearchCV.
min_resource
int or str = 'auto'
Smallest resource any bracket may start at.
max_resource
int or str = 'auto'
Full resource, reached by the last round of the first bracket.
n_brackets
int or str = 'auto'
Brackets to run. 'auto' uses the full :math:`s_{\max} + 1` set; a smaller number keeps the most aggressive brackets, which is the right economy when the budget is tight.
scoring
str or callable = 'accuracy'
Metric used to rank configurations.
cv
int = 5
Folds per evaluation.
refit
bool = True
Whether to refit the best configuration on the full data.
random_seed
int
Seed for candidate sampling and subsampling.

Attributes

best_params_
dict
Best configuration across all brackets.
best_score_
float
Its score, measured at that bracket's final resource.
best_estimator_
Any
Refitted estimator, when refit=True.
cv_results_
dict
Per-candidate record, with bracket, round and resource.
brackets_
list of dict
Per-bracket n_candidates, min_resource and best_score.

Notes

Complexity. Roughly n_brackets times one successive-halving run, which is still far below evaluating every candidate at full resource.

Scores across brackets are comparable only at the top. Each bracket's winner was measured at that bracket's final resource, which is the full resource for the first bracket but less for later ones when the schedule does not divide evenly. best_score_ therefore favours brackets that reached further; read brackets_ to see what each achieved.

Measured on load_breast_cancer tuning a RandomForest, cv=3, averaged over 3 seeds: Hyperband scored 0.7374 in 6.0 s against random search's 0.7425 in 18.7 s — 3.1x faster for half a point of score — while a single aggressive halving schedule scored 0.7226 in 5.0 s. Hyperband's extra brackets are what buy back that difference.

When to use. Hyperband is the sensible default for expensive fits when nothing is known about how the score responds to resource. Prefer SuccessiveHalvingSearchCV directly when that response is known — you then spend the whole budget on the right schedule rather than a constant factor of it on several. Prefer BayesianSearchCV when fits are expensive but the resource cannot be varied, since it economises on which points to try rather than on how long to try them.

References

Li2018
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., & Talwalkar, A. (2018). Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization. Journal of Machine Learning Research, 18(185), 1-52. :arxiv:`1603.06560`

See Also

SuccessiveHalvingSearchCV

One bracket, chosen by you.
RandomSearchCV

What Hyperband's last bracket reduces to.
python 
>>> import numpy as np
>>> from tuiml.evaluation.tuning import HyperbandSearchCV
>>> from tuiml.base.tuning import ParameterDistribution
>>> from tuiml.algorithms.trees import RandomForestClassifier
>>> from tuiml.datasets import load_iris
>>> data = load_iris()
>>> space = ParameterDistribution({'max_depth': (2, 12, 'int'),
...                                'n_estimators': (5, 30, 'int')})
>>> search = HyperbandSearchCV(
...     RandomForestClassifier(), space, factor=3, n_brackets=2,
...     cv=3, random_seed=0)
>>> search.fit(data.X, data.y)
HyperbandSearchCV(factor=3, n_brackets=2)
>>> len(search.brackets_)
2
>>> bool(search.best_score_ > 0.8)
True

Methods

fit (self, X: np.ndarray, y: np.ndarray) -> 'HyperbandSearchCV'

Run every bracket and keep the best configuration overall.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Training features.
y
np.ndarray of shape (n_samples,)
Target values.
Returns
self
HyperbandSearchCV
The fitted searcher.
__repr__ (self) -> str

Return a readable representation of the searcher.