API Reference / automl /

automl.py

Automatic model selection and tuning under a wall-clock budget.

AutoMLClassifier and AutoMLRegressor take a dataset and a number of seconds, and give back a fitted model, a leaderboard of everything they tried, and -- the part that makes the result portable -- best_spec_, the winning configuration written as a train spec. The spec is plain JSON-compatible data, so the outcome of a search can be committed to a repository, diffed, reviewed, and replayed without re-running the search.

Classes

AutoMLClassifier

class automl.automl.AutoMLClassifier(_AutoMLBase, Classifier)

Automatic classifier selection and tuning under a time budget.

Searches TuiML's own classifiers -- no external AutoML backend -- and returns the best one it found, plus everything needed to reproduce it.

Overview

  1. Rank the library's classifiers cheap-and-strong first
(build_portfolio).
  1. Evaluate every candidate at its defaults, then narrow the field by
successive halving while a Gaussian process picks where to sample configurations derived from each algorithm's own parameter schema (search_space_for) until the wall-clock budget is spent.
  1. Optionally combine the fitted trials with greedy ensemble selection
(greedy_selection).
  1. Publish the winner as best_spec_, a train spec.

Parameters

time_budget
float = 60.0
Wall-clock seconds the search may spend. A model fit cannot be interrupted once started, so the budget bounds when a new trial may begin, not when the last one ends: a candidate is admitted only while a worst-case trial still fits in the time left. Overshoot is therefore bounded by roughly one trial, and is largest when the first expensive candidate is admitted on the evidence of cheap ones.
metric
str or callable = None
Metric to optimise: a function name from metrics, or a callable (y_true, y_pred) -> float. Loss-style names are negated automatically. Defaults to "accuracy_score".
cv
int or None = 3
Cross-validation folds used to score each trial. None (or 1) switches to a single stratified 75/25 holdout, which is roughly cv times cheaper per trial.
random_state
int
Seed for the splits and for hyperparameter sampling.
ensemble
bool = True
Run greedy ensemble selection over the trial pool and predict with the ensemble if it beats the best single model.
candidates
sequence of str or type
Explicit candidates to search, overriding portfolio discovery. Order is respected.
max_candidates
int
Truncate the discovered portfolio to its top max_candidates.
n_ensemble_rounds
int = 25
Greedy selection rounds (the ensemble size counted with multiplicity).
halving_factor
int = 3
Aggressiveness of the candidate elimination in pass two. Each rung keeps the top 1 / halving_factor of the field and multiplies the draws per survivor by the same amount. 1 disables elimination and gives every candidate an equal share of the budget, which is what plain round-robin random search does.
acquisition
{'ei', 'pi', 'ucb'} = 'ei'
Acquisition policy the surrogate maximises when proposing the next configuration. See AcquisitionFunction.
verbose
int = 0
Print one line per trial when greater than 0.

Attributes

best_spec_
dict
The winning single model as a train spec: {"model": {"name": ..., "params": {...}}, "pipeline": [], "evaluation": {...}}. JSON-serialisable, so a search result can be committed and replayed.
best_estimator_
Classifier
The best single model, refitted on all the training data.
best_name_, best_params_, best_score_
str, dict, float
The winner's registry name, parameters, and validation score.
leaderboard_
list of dict
Every successful trial as {"rank", "name", "score", "params"}, best first.
failed_
list of dict
Candidates that raised, with the error text.
ensemble_
GreedyEnsemble or None
The selected ensemble, when ensemble=True and it beat the best single model.
portfolio_
list of str
The candidates the search considered, in the order it tried them.
eliminated_
list of str
Candidates dropped by halving, in the order they were cut. Reading it against leaderboard_ shows where the budget stopped going.
survivors_
list of str
Candidates still alive when the budget ran out.
rungs_
int
Halving rungs completed.
search_time_
float
Wall-clock seconds the search actually used.

Notes

Complexity. One fit per trial (cv + 1 fits when cv >= 2); the trial count is whatever the budget allows. Ensemble selection is O(R \cdot M \cdot n) for R rounds, M pool members and n validation rows, with no model fitting at all.

When to use. As a strong first pass on a new tabular dataset, or to generate a starting train() spec to hand-tune afterwards. When the algorithm is already known, tune it directly with RandomSearchCV and skip the portfolio.

References

Caruana2004
Caruana, R., Niculescu-Mizil, A., Crew, G., & Ksikes, A. (2004). Ensemble selection from libraries of models. Proceedings of the 21st International Conference on Machine Learning (ICML), 18. :doi:`10.1145/1015330.1015432`
Feurer2015
Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., & Hutter, F. (2015). Efficient and robust automated machine learning. Advances in Neural Information Processing Systems, 28.

See Also

AutoMLRegressor

The regression counterpart.
search_space_for

Derives the per-algorithm search space.
build_portfolio

Chooses and ranks the candidates.
python 
>>> from tuiml.automl import AutoMLClassifier
>>> from tuiml.datasets import load_iris
>>> data = load_iris()
>>> automl = AutoMLClassifier(time_budget=5, cv=3, random_state=0).fit(
...     data.X, data.y)
>>> automl.best_spec_["model"]["name"]                  # doctest: +SKIP
'LogisticRegression'
>>> automl.leaderboard_[0]["score"] > 0.9
True

The spec is the deliverable -- replay it with train:

python 
>>> import tuiml
>>> spec = dict(automl.best_spec_, data={"X": data.X, "y": data.y})
>>> model = tuiml.train(spec)                            # doctest: +SKIP

Methods

predict_proba (self, X: np.ndarray) -> np.ndarray

Predict class probabilities with the winning model or ensemble.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Input samples.
Returns
proba
np.ndarray of shape (n_samples, n_classes)
Class probabilities, columns ordered as classes_.
get_capabilities (cls) -> List[str]

Return classifier capabilities.

AutoMLRegressor

class automl.automl.AutoMLRegressor(_AutoMLBase, Regressor)

Automatic regressor selection and tuning under a time budget.

The regression counterpart of AutoMLClassifier: same three-stage search (portfolio, schema-derived sampling, greedy ensembling) and the same best_spec_ deliverable, scoring with R^2 by default and averaging predicted values rather than class probabilities.

Parameters

time_budget
float = 60.0
Wall-clock seconds the search may spend. A model fit cannot be interrupted once started, so the budget bounds when a new trial may begin, not when the last one ends: a candidate is admitted only while a worst-case trial still fits in the time left. Overshoot is therefore bounded by roughly one trial, and is largest when the first expensive candidate is admitted on the evidence of cheap ones.
metric
str or callable = None
Metric to optimise, from metrics or a callable. Error metrics such as "mean_squared_error" are negated automatically so that higher always means better. Defaults to "r2_score".
cv
int or None = 3
Cross-validation folds per trial; None uses a 75/25 holdout.
random_state
int
Seed for the splits and for hyperparameter sampling.
ensemble
bool = True
Average the trial pool by greedy selection when that scores better.
candidates
sequence of str or type
Explicit candidates, overriding portfolio discovery.
max_candidates
int
Truncate the discovered portfolio.
n_ensemble_rounds
int = 25
Greedy selection rounds.
verbose
int = 0
Print one line per trial when greater than 0.

Attributes

best_spec_
dict
The winning model as a train spec.
best_estimator_
Regressor
The best single model, refitted on all the training data.
best_name_, best_params_, best_score_
str, dict, float
The winner's registry name, parameters, and validation score.
leaderboard_
list of dict
Every successful trial, best first.
failed_
list of dict
Candidates that raised, with the error text.
ensemble_
GreedyEnsemble or None
The selected ensemble, when it beat the best single model.
portfolio_
list of str
Candidates considered, in the order they were tried.
search_time_
float
Wall-clock seconds actually used.

Notes

Complexity. cv + 1 fits per trial; the trial count is set by the budget. Ensemble selection fits nothing.

When to use. As a first pass on a new tabular regression problem. For a known algorithm, tune it directly with RandomSearchCV.

References

Caruana2004
Caruana, R., Niculescu-Mizil, A., Crew, G., & Ksikes, A. (2004). Ensemble selection from libraries of models. Proceedings of the 21st International Conference on Machine Learning (ICML), 18. :doi:`10.1145/1015330.1015432`

See Also

AutoMLClassifier

The classification counterpart.
build_portfolio

Chooses and ranks the candidates.
python 
>>> from tuiml.automl import AutoMLRegressor
>>> from tuiml.datasets import load_cpu
>>> data = load_cpu()
>>> automl = AutoMLRegressor(time_budget=5, cv=3, random_state=0).fit(
...     data.X, data.y)
>>> automl.best_score_ > 0.5
True
>>> sorted(automl.best_spec_)
['evaluation', 'model', 'pipeline']

Methods

get_capabilities (cls) -> List[str]

Return regressor capabilities.