Experiments

An experiment is the objective function that evaluates parameters and returns a score. It encapsulates your optimization problem separately from the optimizer, so you can swap algorithms without changing your evaluation code. Hyperactive supports custom functions and built-in classes for common ML tasks with cross-validation.

---

What is an Experiment?

An experiment is any function that takes parameters and returns a score. Hyperactive will search for parameters that maximize this score.

def experiment(params):
    # Your evaluation logic here
    return score  # Hyperactive maximizes this

---

Two Approaches

Choose based on your use case:

For any optimization problem

Custom Functions

Write a function that evaluates parameters and returns a score. Full flexibility for simulations, engineering, research, or any custom logic.

def experiment(params):
    result = run_simulation(params)
    return result.quality

For machine learning tasks

Built-in Classes

Pre-built experiments for sklearn, sktime, and PyTorch. Handles cross-validation, scoring, and best practices automatically.

from hyperactive.experiment.integrations import SklearnCvExperiment

experiment = SklearnCvExperiment(model, X, y, cv=5)

---

Custom Functions

The simplest form of experiment. Takes a dictionary of parameters and returns a number.

Basic Example

def objective(params):
    x = params["x"]
    y = params["y"]
    # Hyperactive MAXIMIZES this score
    return -(x**2 + y**2)

Important

Hyperactive maximizes the score. To minimize a loss, negate it:

return -loss  # Convert minimization to maximization

Mathematical Functions

Optimizing benchmark or mathematical functions:

import numpy as np
from hyperactive.opt.gfo import BayesianOptimizer

# Ackley function (a common benchmark)
def ackley(params):
    x = params["x"]
    y = params["y"]

    term1 = -20 * np.exp(-0.2 * np.sqrt(0.5 * (x**2 + y**2)))
    term2 = -np.exp(0.5 * (np.cos(2 * np.pi * x) + np.cos(2 * np.pi * y)))
    result = term1 + term2 + np.e + 20

    return -result  # Negate to maximize (minimize the Ackley function)

search_space = {
    "x": np.linspace(-5, 5, 100),
    "y": np.linspace(-5, 5, 100),
}

optimizer = BayesianOptimizer(
    search_space=search_space,
    n_iter=50,
    experiment=ackley,
)
best_params = optimizer.solve()

External Simulations

Your function can call any Python code, including simulations, APIs, or file I/O:

import subprocess

def run_simulation(params):
    # Run an external simulation with the given parameters
    result = subprocess.run(
        ["./my_simulation", str(params["param1"]), str(params["param2"])],
        capture_output=True,
        text=True,
    )
    # Parse the output and return the score
    score = float(result.stdout.strip())
    return score

---

Built-in Experiments

For common ML tasks, Hyperactive provides ready-to-use experiment classes.

Class	Use Case	Install
SklearnCvExperiment	Scikit-learn models with CV	Included
SktimeForecastingExperiment	Time series forecasting	pip install hyperactive[sktime]
TorchTrainerExperiment	PyTorch Lightning models	pip install hyperactive[torch]

SklearnCvExperiment

The most common choice for tuning sklearn classifiers and regressors.

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import KFold
from sklearn.metrics import accuracy_score
from hyperactive.experiment.integrations import SklearnCvExperiment
from hyperactive.opt.gfo import HillClimbing

X, y = load_iris(return_X_y=True)

experiment = SklearnCvExperiment(
    estimator=RandomForestClassifier(random_state=42),
    X=X,
    y=y,
    cv=KFold(n_splits=5, shuffle=True, random_state=42),
    scoring=accuracy_score,  # Optional: defaults to estimator's score method
)

search_space = {
    "n_estimators": list(range(10, 200, 10)),
    "max_depth": list(range(1, 20)),
    "min_samples_split": list(range(2, 10)),
}

optimizer = HillClimbing(
    search_space=search_space,
    n_iter=30,
    experiment=experiment,
)
best_params = optimizer.solve()

Key parameters:

• estimator: Any sklearn estimator

• X, y: Your training data

• cv: Number of cross-validation folds (default: 5)

• scoring: Scoring metric (default: estimator’s default)

SktimeForecastingExperiment

For time series forecasting optimization:

from sktime.forecasting.naive import NaiveForecaster
from sktime.datasets import load_airline
from hyperactive.experiment.integrations import SktimeForecastingExperiment
from hyperactive.opt.gfo import RandomSearch

y = load_airline()

experiment = SktimeForecastingExperiment(
    estimator=NaiveForecaster(),
    y=y,
    fh=[1, 2, 3],  # Forecast horizon
)

search_space = {
    "strategy": ["mean", "last", "drift"],
}

optimizer = RandomSearch(
    search_space=search_space,
    n_iter=10,
    experiment=experiment,
)
best_params = optimizer.solve()

TorchTrainerExperiment

For PyTorch Lightning model optimization:

from hyperactive.experiment.integrations import TorchExperiment

experiment = TorchExperiment(
    model_class=MyLightningModel,
    datamodule=my_datamodule,
    trainer_kwargs={"max_epochs": 10},
)

---

Benchmark Functions

Standard test functions for evaluating optimizers:

from hyperactive.experiment.bench import Ackley, Sphere, Parabola

# Use benchmark as experiment
ackley = Ackley(dim=2)

optimizer = BayesianOptimizer(
    search_space=ackley.search_space,
    n_iter=50,
    experiment=ackley,
)

Available benchmarks include: Ackley, Rastrigin, Rosenbrock, Sphere, and more. These are useful for comparing optimizer performance on known landscapes.

---

Direct Evaluation

Experiments can be evaluated directly using the score() method:

from hyperactive.experiment.integrations import SklearnCvExperiment

experiment = SklearnCvExperiment(
    estimator=RandomForestClassifier(),
    X=X, y=y, cv=5,
)

# Evaluate specific parameters
params = {"n_estimators": 100, "max_depth": 10}
score, additional_info = experiment.score(params)

print(f"Score: {score}")
print(f"Additional info: {additional_info}")

This is useful for debugging or manual exploration before running optimization.

---

Error Handling

Robust experiments handle invalid parameter combinations gracefully:

def robust_objective(params):
    try:
        score = compute_score(params)
        return score
    except Exception:
        return -np.inf  # Return bad score on failure

Tip

Returning -np.inf signals an invalid configuration. The optimizer will learn to avoid this region of the search space.

---

Best Practices

Return Meaningful Scores

Ensure your score reflects what you want to optimize. Higher is better (Hyperactive maximizes).

Handle Errors Gracefully

Return -np.inf for invalid configurations instead of raising exceptions.

Consider Compute Time

For expensive experiments, use BayesianOptimizer or TPEOptimizer which learn from previous evaluations.

Use Reproducibility

Set random seeds inside your experiment for consistent results across runs.

---

Quick Reference

# Custom function
def experiment(params):
    result = evaluate(params)
    return score  # Higher is better

# Sklearn integration
from hyperactive.experiment.integrations import SklearnCvExperiment
experiment = SklearnCvExperiment(model, X, y, cv=5)

# Use with any optimizer
from hyperactive.opt.gfo import HillClimbing
optimizer = HillClimbing(search_space, experiment=experiment)
best = optimizer.solve()