Skip to content

feat(bo): Bayesian optimization with a Gaussian process model, log-EI and a Latin hypercube design - #382

Merged
tachsin merged 13 commits into
mainfrom
feat/bo-foundations
Oct 2, 2026
Merged

tachsin merged 13 commits into
mainfrom
feat/bo-foundations

Conversation

@tachsin

@tachsin tachsin commented Oct 1, 2026 •

Copy link
Copy Markdown
Owner

The first part of batch B in the optimization plan: Bayesian optimization, for expensive black-box functions. It follows the decisions in #381 and #379.

Foundations:

  • Portable erf, erfc and the scaled erfcx in math.
  • Real::latin_hypercube.
  • EI, log-EI, PI and UCB, with their gradients, in algorithm::bo::acquisition. Log-EI's gradient goes through the Mills ratio, with an asymptotic series below z = −64, its coefficients checked with mpmath.

model::gp, public and marked unstable for one release:

  • Gaussian process regression with a constant mean, profiled at its GLS estimate (Jones et al. 1998, eq. 5), and an ARD Matérn 5/2 or squared exponential kernel.
  • Hyperparameters by maximum log marginal likelihood, with genoxide's own L-BFGS-B and GPML eq. 5.9's gradient, from several seeded starts.
  • predict and predict_with_gradient.
  • Checked against GPML: eqs. 2.20, 2.25-2.26 with Algorithm 2.1, 2.30, 4.17, 5.8 and 5.9.

Bo:

  • A Latin hypercube of 2(n + 1) points first, then one point per generation: the model is fitted, then log-EI is maximized from 1,000 raw samples and 10 L-BFGS-B starts with its analytic gradient.
  • Acquisitions: log-EI (default), EI, PI and UCB.
  • Output::Standardize (default) or Output::Log, for objectives like Goldstein-Price.
  • It never asks a point twice. Invalid points enter the model at the worst valid value.
  • Re-evaluation and checkpoints; the same bits on every thread count.

Noise: none by default, which differs from the plan. The jitter is the only nugget, so the model interpolates deterministic functions. Learned noise blurred the minimum: within 80 evaluations over 20 seeds, f* + 1e-4 was reached by:

Problem Noise-free Learned noise
Branin 20/20 14/20
Six-hump camel 20/20 17/20
Hartmann 3 20/20 12/20

Noise::Learned stays available for noisy functions.

In Python and the CLI: gx.Bo and gx.model.gp.GaussianProcess; type = "bo".

The bayesian_optimization example: Branin, with the posterior mean and log-EI drawn per step in a new surrogate plot. It ends 2.1e-5 above the global minimum after 31 evaluations, using a final L-BFGS-B polish on the GP mean and one evaluation. Rust and Python give identical output and traces.

Tests:

  • The GP against hand-worked formulas to 1e-13.
  • Every gradient against finite differences.
  • Branin, six-hump camel, Hartmann 3, and Goldstein-Price with Log, each reaching f* + 1e-3.
  • Reproducibility, checkpoints and validation.
  • Two portable_runs entries, and a Rust and Python cross-check.

Coming in B2: batch BO (Kriging believer, constant liar), AsyncEngine, constrained BO, integer genes, and their examples.

tachsin added 12 commits October 2, 2026 11:29
A public Gaussian process, documented as unstable for one release: a constant
mean, an ARD Matérn 5/2 kernel by default or the squared exponential, learned
or fixed noise, inputs scaled to the unit cube by the bounds of a Real genome
and outputs standardized. The hyperparameters maximize the log marginal
likelihood (Rasmussen and Williams, eq. 2.30) with the mean at its generalized
least squares estimate, by genoxide's own L-BFGS-B on their logarithms with the
analytic gradient of eq. 5.9, from a fixed or warm start and random starts on
derived streams, the winner by value then start. The posterior mean and
variance (eq. 2.25, 2.26, Algorithm 2.1) come with their gradients. Cholesky
factorizations by linalg, with a jitter raised tenfold from 1e-10 of the
diagonal when needed.

L-BFGS-B gains a crate-private minimizer driven by hand with a supplied
gradient, for such inner problems.
An ask / tell algorithm on Real genomes for expensive black-box functions: an
initial design of 2(n + 1) points by default (the initial genomes, then a Latin
hypercube), then one point per generation, chosen by fitting the Gaussian
process of model::gp to every evaluation and maximizing an acquisition function
from raw samples and the best point evaluated with L-BFGS-B and the
acquisition's analytic gradient. Log-EI by default, EI, PI (maximized through
its logarithm) and UCB, settable during a run. An output transform: the values
standardized by default, or the logarithm of their distance above the best for
objectives that span orders of magnitude. Invalid points enter the model at the
worst value; a point is never asked twice. Reproducible on any platform and
thread count, re-evaluation and checkpoints included.

The acquisition functions gain their derivatives with respect to the mean and
the standard deviation, log-EI's through an asymptotic series where the Mills
ratio's difference cancels.
…ic functions

genoxide's fitness functions are deterministic, and a Gaussian process that
interpolates their values resolves the small differences near a minimum that a
learned noise smooths over. Measured with Bo over 20 seeds and 80 evaluations
to f* + 1e-4: Branin reached in 20 runs, the six-hump camel in
[-3, 3] x [-2, 2] in 20 and Hartmann 3 in 20, against 14, 17 and 12 with noise
learned from a least of 1e-6, which settled above that least (an absolute
standard deviation of about 0.03 on Branin). Hartmann 6 reached its minimum in
as many runs, far more precisely. Noise::Learned stays for noisy functions.

The documented measurements of the log transform are redone with the new
default, and the tests' budgets with them.
…ition per step

Bo's defaults on Branin's function: 6 points of a Latin hypercube, then 24
chosen by log-EI, which end 9.6e-5 above a global minimum; the Gaussian
process of the 30 evaluations, its mean minimized by L-BFGS-B from the best
point, then evaluated once: 2.1e-5 above it, in 31 evaluations. Over seeds 1
to 20, the polish brings 14 runs within 1e-4 at 30 evaluations, against 4 by
the search alone. Rust and Python print the same rows and write the same
trace, whose frames hold the model's mean and the acquisition on a 25 x 25
grid, for a new plot kind of the site's player, `surrogate`, and a new
category, `bayesian`.
gx.Bo(real, ...) with the Rust builder's settings, the acquisitions as "log-ei",
"ei", gx.ProbabilityOfImprovement(xi) and gx.UpperConfidenceBound(beta), and
gx.RunningBo for a control: the acquisition, which can change, the model that
chose the last point and the acquisition's values. gx.model.gp.GaussianProcess
fits and queries the Rust model on numpy arrays. A run in Python gives the
bits of the same run in Rust, which tests on both sides check.
The genoxide program runs Bo with every builder setting: the acquisition as
"log-ei", "ei", { type = "pi", xi } or { type = "ucb", beta }, the kernel, the
noise as a number or { learned = <least> }, the output transform and the starts
of both maximizations. Checkpoints resume a run as it would have gone on.
AGENTS.md gains a section on Bo with a program that runs in CI (the search,
then a polish of the model's mean), rows in both method tables and in
troubleshooting, and the panics of a point of the wrong length. The README, the
crate docs, the Python README, both llms.txt, the features list and the
packages' descriptions name Bayesian optimization; the roadmap marks the first
part of batch B done and lists the second.

The plan records what was verified in Rasmussen and Williams (2006), Jones,
Schonlau and Welch (1998), Loeppky, Sacks and Welch (2009), Snoek, Larochelle
and Adams (2012) and Srinivas et al. (2010), and where the implementation
departs from its design notes, with the measurements: no noise by default, the
log transform's offset, re-evaluation.
…'t factor

The final factorization of a fit can only fail if rounding defeats a jitter of
the diagonal's scale; the fit then returns Error::InvalidSetting and Bo asks a
random point instead. The module docs' link to Lbfgsb loses its redundant
target, which the doc build rejected.
@tachsin
tachsin force-pushed the feat/bo-foundations branch from 9c2d994 to 27852c1 Compare October 2, 2026 09:55
@tachsin tachsin changed the title feat(bo): acquisition functions, Latin hypercube designs and portable erf, erfc and erfcx feat(bo): Bayesian optimization with a Gaussian process model, log-EI and a Latin hypercube design Oct 2, 2026
@tachsin
tachsin marked this pull request as ready for review October 2, 2026 09:55
@tachsin
tachsin merged commit d186c7f into main Oct 2, 2026
22 checks passed
@tachsin
tachsin deleted the feat/bo-foundations branch October 2, 2026 10:26
@github-actions github-actions Bot mentioned this pull request Oct 2, 2026
tachsin added a commit that referenced this pull request Oct 2, 2026
…ization, completing batch B (#415)

The second part of batch B in the [optimization
plan](docs/optimization-plan.md), after #382: Bayesian optimization in
batches, asynchronously, with constraints, and on integer genes.

**Batches:** `.batch(q)` with `.fantasy(bo::Fantasy::KrigingBeliever)`,
the default, or `ConstantLiar(Lie::Min | Mean | Max)` (Ginsbourger, Le
Riche and Carraro 2010, Algorithms 1 and 2, checked).
- Each point of a batch is chosen after the earlier ones are added with
a fantasized value, through `GaussianProcess::with_points`.
Hyperparameters are not refit within a batch.
- With `Engine::parallel(true)`, a batch is evaluated in parallel, with
the same bits on any number of threads.
- `batch(1)` reproduces single-point runs bit for bit.
- The believer is the default on evidence. With batches of 4 over 20
seeds, it ties the lowest lie and beats the mean and highest lies. Under
`AsyncEngine` it reached Hartmann 3 in 5 of 5 runs, against 2 of 5 for
the lowest lie.

**Asynchronous:** `Bo` implements `Incremental`, so `AsyncEngine`
proposes a point while others are pending, with the pending points
fantasized. A checkpoint holds them, and a resumed run matches an
uninterrupted one. `Incremental` gains defaulted `prepare`, `wants` and
`receive_evaluation`, and `AsyncEngine` evaluates the wanted extras.

**Constraints** (Gardner et al. 2014, checked):
- Constraint values come from `Constrained` or a problem's own. Each
constraint gets its own GP.
- EI is multiplied by the probability of feasibility, and log-EI adds
its logarithm. Before any feasible point exists, that probability alone
is maximized.
- UCB with constraints is an error.

**Integer genes:** `Bo<R: bo::Space = Real>`, with `Space` implemented
for `Real` and `Integer`.
- The model sees lattice points only (Garrido-Merchán and
Hernández-Lobato 2020, eq. 7).
- The acquisition is maximized on the lattice: exhaustively when the
lattice is small, otherwise by raw samples and a ±1 hill climb.
- A fully evaluated lattice ends the run as `Converged`.

**In Python and the CLI:**
- Python: `gx.Bo(..., batch=, fantasy=)`, `run(f, constraints=m)`, and
integer genomes.
- CLI: `batch`, `fantasy`, `asynchronous = true`, integer genomes, and
constraint values from fitness programs.

**Examples** (Rust and Python identical, except the asynchronous one):
- **`bo_hartmann6`:** batches of 4 come within 1e-4 of the global
minimum in 15 rounds (74 evaluations), against 36 rounds one point at a
time. The global minimum depends on the seed: batches reach it in 13 of
20 seeds and single points in 12 of 20, the rest stopping at the local
minimum −3.2032. The README says so.
- **`bo_constrained`:** Gramacy et al.'s toy problem. It comes within
1e-5 of f* = 0.5997880520 (checked with mpmath and a grid scan) in 21
evaluations, and every one of 20 seeds within 39. Given only the total
violation, the same search ends 0.34 above.
- **`bo_asynchronous`** (Rust only): Hartmann 3 to 1e-4 with 4 workers
and evaluations of 10-50 ms, in about 0.4 s, against 0.8 s for batches
of 4. Its timings are masked in CI.

**Tests:**
- Batch and fantasy behaviour; the same bits on 1, 2 and 8 threads.
- Async with one worker reproducible.
- Constrained BO on the Gramacy problem and CEC 2006 G24, and the
probability of feasibility's gradients.
- Integer BO.
- Checkpoints, including pending points.
- `portable_runs` entries and Python tests.
tachsin added a commit that referenced this pull request Oct 2, 2026
## 🤖 New release

* `genoxide`: 0.12.0 -> 0.13.0 (✓ API compatible changes)
* `genoxide-python`: 0.12.0 -> 0.13.0

<details><summary><i><b>Changelog</b></i></summary><p>

## `genoxide`

<blockquote>

##
[0.13.0](v0.12.0...v0.13.0)
- 2026-10-02

### <!-- 0 -->Added

- *(bo)* Bayesian optimization with a Gaussian process model, log-EI and
a Latin hypercube design
([#382](#382))
- *(bo)* batch, asynchronous, constrained and integer Bayesian
optimization, completing batch B
([#415](#415))

### <!-- 4 -->Documentation

- correct citations found in an audit of every reference
([#412](#412))
</blockquote>



</p></details>

---
This PR was generated with
[release-plz](https://github.com/release-plz/release-plz/).

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: tachsin <tachsinachmet@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant