Bayesian optimisation, animated

Before the theory, an animated overview: Gaussian process regression (the uncertainty band tightening onto the data), then the decision loop with Expected Improvement, and finally the exploration/exploitation trade-off via UCB.

Why Bayesian optimisation?

Picture training a large neural network on an HPC cluster. Each hyperparameter configuration requires a full retraining: two hours of compute. You have a budget of fifty evaluations. Which ones should you run?

Gradient descent assumes we know \(f\) and \(\nabla f\) — here we know neither. Grid search wastes the budget on blind coverage. BO answers a simple question: how do we intelligently decide where to evaluate next, given everything observed so far?

The central idea: replace the costly function with a probabilistic surrogate, cheap to query, that returns not only a prediction but also an uncertainty. We exploit that uncertainty to target the next evaluations. It is a one-step lookahead: each round we make the move that maximises the expected one-move gain — a myopic policy, but robust and cheap in practice.

The surrogate: Gaussian processes

The surrogate is almost always a Gaussian process (GP): a distribution over functions, fully determined by a mean \(\mu(\cdot)\) and a covariance \(k(\cdot,\cdot)\).

Key property: any finite subset of points follows a multivariate Gaussian. After observing noisy data \((\mathbf{x},\mathbf{y})\) with \(\varepsilon \sim \mathcal{N}(0,\sigma_n^2)\), we predict at any test point:

Adding \(\sigma_n^2\) on the diagonal makes the mean smooth the observations rather than pass through them, and the variance stays strictly positive everywhere. The GP thus provides both a prediction and its uncertainty — exactly what makes it the ideal surrogate.

Kernels (covariance functions)

With the mean usually zero, it is the kernel that characterises the GP. It must be symmetric and positive semi-definite. Two families dominate.

The Fit – Choose – Evaluate loop

BO is a sequential process: each round it fits the surrogate, chooses the most promising point, evaluates it, then repeats.

Maximising \(\alpha_n\) (the inner loop) must be far cheaper than one evaluation of \(f\).

Acquisition functions

The acquisition function \(\alpha_n(x)\) captures the benefit of evaluating at \(x\). Write \(\mu_n(x)\), \(\sigma_n(x)\) for the predictive mean and standard deviation, and \(y_n^*\) for the best observed value. Three main families exist.

Simulation: UCB in action

Slide \(\beta\) and the number of observations. The yellow square marks the point UCB would choose next: watch how a high \(\beta\) pushes toward uncertain regions (exploration), a low \(\beta\) toward the mean's peaks (exploitation).

Simulation: compare PI, EI and UCB

Same surrogate, same observations, three different acquisitions. Step through the loop and compare where each strategy would evaluate (yellow square).

Applications

Hyperparameter tuning

Tuning a TCN or WaveNet on HPC, where each configuration requires a full retraining, is exactly BO's canonical use case. With fifty well-placed evaluations, you cover the space far better than a grid of hundreds of points.

Uncertainty quantification for frost risk

A GP gives a full predictive law. For a frost alert, the right quantity is not just \(\hat{T}_c < 3°\mathrm{C}\), but the probability \(\Pr(T_c < 3°\mathrm{C})\), read off the predictive band. The alert fires as soon as the lower bound crosses the threshold — ahead of the mean.

Link to kriging and the Kalman filter

GP regression is kriging in spatial statistics. And the GP's marginal likelihood is precisely the quantity the Kalman filter optimises implicitly — a direct bridge to my Kalman filter article.

Limitations and extensions