Sequential Model-Based Methods#

Sequential model-based optimizers build a model of the objective function and use it to decide where to sample next. They are best for expensive evaluations where you want to minimize the number of function calls.

These methods learn from past evaluations to make smarter decisions about which parameters to try next.

Bayesian Optimization#

Uses Gaussian Process regression to model the objective and acquisition functions to balance exploration and exploitation.

from hyperactive.opt.gfo import BayesianOptimizer

optimizer = BayesianOptimizer(
    search_space=search_space,
    n_iter=5,
    experiment=objective,
)

Tree-Structured Parzen Estimators (TPE)#

Models the distribution of good and bad parameters separately.

Forest Optimizer#

Uses Random Forest to model the objective function.

Lipschitz Optimization and DIRECT Algorithm#

Use Lipschitz continuity assumptions to guide the search.

When to Use Model-Based Methods#

Sequential model-based optimizers are best suited for:

  • Expensive evaluations: Training ML models, simulations, real-world experiments

  • Limited budget: When you can only afford 10-100 evaluations

  • Smooth landscapes: When the objective function has some continuity

Consider using simpler methods if:

  • Evaluations are cheap (random search may find good solutions faster)

  • The search space is very large or high-dimensional

  • The landscape is highly non-smooth or discontinuous