Successive halving - multi-fidelity hyperparameter search.
Classes
class evaluation.tuning.successive_halving.SuccessiveHalvingSearchCV(BaseTuner)
Search many configurations cheaply, then spend the budget on survivors.
__init__( self, estimator, param_distributions, n_candidates: int = 27, factor: int = 3, resource: str = 'n_samples', min_resource: Union[int, str] = 'auto', max_resource: Union[int, str] = 'auto', aggressive_elimination: bool = False, scoring: Union[str, Any] = 'accuracy', cv: int = 5, refit: bool = True, verbose: int = 0, n_jobs: int = 1, random_seed: Optional[int] = None, progress_callback: Optional[Any] = None, )
Overview
-
Start with
n_candidatesconfigurations and a small resource — by
- Evaluate them all by cross-validation at that resource.
- Keep the best 1/\eta of them and multiply the resource by
- Repeat until one configuration remains or the full resource is reached.
Theory
With n candidates and elimination factor \eta, each round keeps n_i = \lfloor n / \eta^i \rfloor candidates at resource r_i = r_{\min} \eta^i. Because the survivor count falls at the same rate the resource grows, every round costs roughly the same, and the total is about \log_\eta(n) times one full-budget evaluation — against n for an exhaustive search.
The assumption that buys this is that rank is roughly preserved across resource levels: a configuration that looks bad on 10% of the data is unlikely to be the best on 100%. That mostly holds and occasionally does not, which is the method's one real failure mode. A configuration whose advantage only appears with enough data — a high-capacity model that needs volume to beat a simpler one — can be eliminated in round one and never reconsidered. Raise min_resource when that risk is real.
Parameters
estimator
param_distributions
RandomSearchCV does.
n_candidates
factor makes the rounds come out even.
factor
factor. Three is the usual choice; larger is more aggressive and more likely to discard a late bloomer.
resource
'n_samples' grows the training subsample; any other string names an integer estimator parameter to grow instead — 'n_estimators' for a forest, for example, which is often the better resource because it costs nothing in statistical power.
min_resource
'auto' picks :math:`\max(n / \text{factor}^{\text{rounds}},\ 20)` for samples, or 1 for a parameter resource.
max_resource
'auto' is the full training-set size for 'n_samples', or the estimator's current value otherwise.
aggressive_elimination
max_resource. Useful when the candidate pool is large relative to the resource range.
scoring
cv
refit
random_seed
Attributes
best_params_
best_score_
best_estimator_
refit=True.
cv_results_
round and resource each score was measured at.
n_rounds_
n_candidates_per_round_
resources_per_round_
Notes
Complexity. About \log_\eta(n) rounds of roughly equal cost, so total work is close to \log_\eta(n) full-budget evaluations rather than n.
Scores are not comparable across rounds. A score from round 0 was measured on a fraction of the data and is usually pessimistic; best_score_ is always taken from the final round so the reported number means what it appears to mean. Read cv_results_['round'] before comparing entries.
Measured on load_breast_cancer tuning a RandomForest over three parameters, cv=3, averaged over 3 seeds:
==================== ========= ======== ========== searcher score time speed-up ==================== ========= ======== ========== RandomSearchCV (27) 0.7425 18.7 s 1.0x SuccessiveHalving(27) 0.7226 5.0 s 3.8x HyperbandSearchCV 0.7374 6.0 s 3.1x ==================== ========= ======== ==========
Read that as the trade it is: both finish in about a quarter of the time, but plain halving gave up two points of score while Hyperband gave up half a point. That is the hedging working — halving committed to one aggressive schedule and sometimes killed the eventual winner early, which is the failure mode described above.
Choosing a parameter as the resource rather than the sample count did better than either: growing n_estimators from 1 to 27 across rounds finished in 1.3 s, fourteen times faster than random search, at a comparable score. When the estimator has a natural budget knob, use it — unlike subsampling it costs no statistical power.
When to use. Use this when a single fit is expensive and the candidate pool is large — exactly where random search wastes most of its budget. It is pointless when fits are cheap, since the bookkeeping then costs more than it saves, and unsuitable when the resource cannot be varied meaningfully. When the total budget rather than the pool size is what you control, HyperbandSearchCV removes the choice of min_resource by trying several.
References
See Also
>>> import numpy as np
>>> from tuiml.evaluation.tuning import SuccessiveHalvingSearchCV
>>> from tuiml.base.tuning import ParameterDistribution
>>> from tuiml.algorithms.trees import RandomForestClassifier
>>> from tuiml.datasets import load_iris
>>> data = load_iris()
>>> space = ParameterDistribution({'max_depth': (2, 12, 'int'),
... 'n_estimators': (5, 40, 'int')})
>>> search = SuccessiveHalvingSearchCV(
... RandomForestClassifier(), space, n_candidates=9, factor=3,
... cv=3, random_seed=0)
>>> search.fit(data.X, data.y)
SuccessiveHalvingSearchCV(n_candidates=9, factor=3)
>>> search.n_candidates_per_round_
[9, 3, 1]
>>> bool(search.best_score_ > 0.8)
TrueMethods
fit
(self, X: np.ndarray, y: np.ndarray) -> 'SuccessiveHalvingSearchCV'
fit
(self, X: np.ndarray, y: np.ndarray) -> 'SuccessiveHalvingSearchCV'
Run the halving schedule and keep the best surviving configuration.
Parameters
X
y
Returns
self