Sequential model-based hyperparameter search using Gaussian Processes.
Where GridSearchCV enumerates and RandomSearchCV samples blindly, Bayesian optimization learns as it goes: it fits a cheap probabilistic surrogate to the (configuration, score) pairs observed so far and spends its next expensive model fit wherever that surrogate says the payoff is greatest.
The module supplies the three pieces of that loop:
-
GaussianProcess— the surrogate. A from-scratch GP regressor with
-
AcquisitionFunction— the decision rule. Turns the surrogate's mean
-
BayesianSearchCV— the tuner. Warms up
Reach for this module when a single estimator fit is expensive enough that thinking about the next configuration is cheaper than trying another one at random — large models, big datasets, or budgets in the tens of evaluations rather than thousands. For cheap fits the GP overhead dominates and random search is the better tool.
Everything is implemented on NumPy and SciPy; no external Bayesian optimization library is required.
References
Classes
Gaussian Process surrogate model with an RBF kernel.
AcquisitionFunction needs in order to decide where to sample next.__init__( self, kernel: str = 'rbf', length_scale: float = 1.0, noise: float = 1e-10, )
Overview
-
fitbuilds the kernel matrix K over the training inputs and
noise to its diagonal for numerical stability.
-
It inverts that matrix once and caches the result in
K_inv. -
predictforms the cross-kernel K_* between the query points
Theory
The squared-exponential (RBF) kernel with length scale \ell is
Conditioning a zero-mean GP prior on observations (X, y) gives the posterior at test points X_*
where K = k(X, X), K_* = k(X_*, X), K_{**} = k(X_*, X_*) and \sigma_n^{2} is noise. predict returns \mu_* and, on request, the square root of the diagonal of \Sigma_*, clipped at zero so round-off cannot produce a negative variance.
A small \ell makes the surrogate wiggly and trusts observations only very locally; a large \ell smooths the surface and lets a single observation influence distant regions.
Parameters
kernel
'rbf' (squared exponential) is implemented; the value is stored but not otherwise consulted.
length_scale
noise
numpy.linalg.inv complains about a singular matrix.
Attributes
X_train
fit. None before fitting.
y_train
fit.
K_inv
Notes
Complexity. fit is O(n^{3}) for the explicit matrix inversion and O(n^{2}) in memory; predict is O(m\,n) for the mean and O(m^{2} + m\,n^{2}) when standard deviations are requested. This is fine for the tens to low hundreds of observations a hyperparameter search produces, and unsuitable for large-scale regression.
When to use. As the surrogate inside BayesianSearchCV. It is a deliberately minimal implementation: no kernel hyperparameter marginal-likelihood optimization, no Cholesky solve, no per-dimension length scales.
References
See Also
Fit the surrogate to four observations and query an untried point:
>>> import numpy as np
>>> from tuiml.evaluation.tuning.bayesian_search import GaussianProcess
>>> X = np.array([[0.0], [1.0], [2.0], [3.0]])
>>> y = np.array([0.0, 0.8, 0.9, 0.1])
>>> gp = GaussianProcess(length_scale=1.0, noise=1e-6)
>>> gp.fit(X, y)
>>> mu, sigma = gp.predict(np.array([[1.5]]), return_std=True)
>>> round(float(mu[0]), 3)
1.019
>>> round(float(sigma[0]), 3)
0.1The posterior interpolates the observations it has already seen:
>>> round(float(gp.predict(np.array([[1.0]]))[0]), 3)
0.8Methods
fit
(self, X: np.ndarray, y: np.ndarray)
fit
(self, X: np.ndarray, y: np.ndarray)
Condition the Gaussian Process on observed points.
Parameters
X
y
Returns
None
X_train, y_train and K_inv are set.
Raises
numpy.linalg.LinAlgError
noise or drop duplicate rows from X.
predict
(self, X: np.ndarray, return_std: bool=False) -> Union[np.ndarray, Tuple[np.ndarray, np.ndarray]]
predict
(self, X: np.ndarray, return_std: bool=False) -> Union[np.ndarray, Tuple[np.ndarray, np.ndarray]]
Evaluate the GP posterior at new points.
Parameters
X
fit.
return_std
True, also return the posterior standard deviation, which is what acquisition functions use to quantify exploration value.
Returns
y_mean
return_std=False.
y_std
return_std=True, as the second element of a tuple.
Raises
ValueError
fit has not been called yet.
Acquisition functions that turn a GP posterior into a sampling decision.
GaussianProcess and the best score observed so far, it scores candidate points so that higher is better; the optimizer then maximizes it to choose the next configuration to actually train. All three variants balance the same tension: the posterior mean \mu(x) pulls toward regions already known to be good (exploitation) while the posterior standard deviation \sigma(x) pulls toward regions the surrogate has not pinned down (exploration).__init__( self, kind: str = 'ei', xi: float = 0.01, kappa: float = 2.576, )
Theory
Write y^{+} for the incumbent best observed score and
Expected Improvement (kind='ei') is the expected amount by which x beats the incumbent, which has the closed form
where \Phi and \phi are the standard normal CDF and PDF. The first term rewards a high mean, the second rewards uncertainty, and \xi sets how much improvement must be expected before a point counts as promising. EI is the usual default because it is scale-aware: it answers "by how much", not merely "how likely".
Probability of Improvement (kind='poi') keeps only the probability that the incumbent is beaten,
which ignores the size of the gain and therefore exploits greedily unless \xi is raised.
Upper Confidence Bound (kind='ucb') is an explicit optimistic bound and the only variant that does not reference y^{+},
with \kappa directly dialing exploration. The default \kappa = 2.576 is the two-sided 99% normal quantile.
Parameters
kind
'ei' for Expected Improvement, 'ucb' for Upper Confidence Bound, 'poi' for Probability of Improvement. Any other value raises at call time, not at construction.
xi
'ei' and 'poi'. Larger values explore more; 0.0 is purely greedy. Ignored by 'ucb'.
kappa
'ucb'; larger values favor uncertain regions. Ignored by 'ei' and 'poi'.
Notes
Complexity. One GP prediction with standard deviations plus O(m) arithmetic for m candidate points, so the cost is dominated by predict.
When to use. Keep 'ei' unless you have a reason not to. Choose 'ucb' when you want a single, interpretable exploration knob, and 'poi' when any improvement at all is what matters. Standard deviations are floored at 1e-9 before division, so already-observed points where \sigma \to 0 yield a near-zero score rather than a divide-by-zero.
References
See Also
Score two candidate points against a surrogate whose incumbent is 0.9:
>>> import numpy as np
>>> from tuiml.evaluation.tuning.bayesian_search import (
... AcquisitionFunction, GaussianProcess)
>>> X = np.array([[0.0], [1.0], [2.0], [3.0]])
>>> y = np.array([0.0, 0.8, 0.9, 0.1])
>>> gp = GaussianProcess(length_scale=1.0, noise=1e-6)
>>> gp.fit(X, y)
>>> candidates = np.array([[1.5], [2.5]])
>>> ei = AcquisitionFunction(kind='ei', xi=0.01)
>>> [round(float(v), 4) for v in ei(candidates, gp, y_best=0.9)]
[0.1157, 0.0]UCB and POI rank the same candidates with different appetites for risk:
>>> ucb = AcquisitionFunction(kind='ucb', kappa=2.576)
>>> [round(float(v), 4) for v in ucb(candidates, gp, y_best=0.9)]
[1.2752, 0.8316]
>>> poi = AcquisitionFunction(kind='poi')
>>> [round(float(v), 4) for v in poi(candidates, gp, y_best=0.9)]
[0.8626, 0.0006]Methods
__call__
(self, X: np.ndarray, gp: GaussianProcess, y_best: float) -> np.ndarray
__call__
(self, X: np.ndarray, gp: GaussianProcess, y_best: float) -> np.ndarray
Score candidate points with the configured acquisition policy.
Parameters
X
gp
y_best
'ei' and 'poi'; ignored by 'ucb'.
Returns
values
kind settings.
Raises
ValueError
self.kind is not one of 'ei', 'ucb' or 'poi'.
Bayesian optimization search for hyperparameter tuning.
__init__( self, estimator, param_space: Dict, n_iterations: int = 50, acquisition: str = 'ei', n_random_starts: int = 10, xi: float = 0.01, kappa: float = 2.576, scoring: Union[str, Callable] = 'accuracy', cv: int = 5, refit: bool = True, verbose: int = 0, n_jobs: int = 1, random_seed: Optional[int] = None, progress_callback: Optional[Callable] = None, **kwargs, )
Parameters
estimator
param_space
Dictionary with parameters as keys and search spaces as values. Values can be:
- •List of values: discrete search
- •Tuple (min, max): continuous search
- •Tuple (min, max, 'int'): integer search
n_iterations
acquisition
n_random_starts
xi
kappa
scoring
cv
refit
verbose
random_state
Attributes
best_params_
best_score_
best_estimator_
cv_results_
gp_
>>> from tuiml.algorithms.ensemble import BaggingClassifier
>>> from tuiml.datasets import load_iris
>>> from tuiml.evaluation.splitting import train_test_split
>>> from tuiml.evaluation.tuning import BayesianSearchCV
>>> X, y = load_iris()
>>> X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)A search space maps each parameter to a (low, high) range for floats
or a (low, high, 'int') triple for integers:
>>> param_space = {
... 'n_estimators': (5, 20, 'int'),
... 'bag_size_percent': (50.0, 100.0),
... }
>>> search = BayesianSearchCV(
... estimator=BaggingClassifier(),
... param_space=param_space,
... n_iterations=5,
... cv=2,
... )
>>> search = search.fit(X_train, y_train) # doctest: +SKIP
>>> sorted(search.best_params_) # doctest: +SKIP
['bag_size_percent', 'n_estimators']