feat(bo): Bayesian optimization with a Gaussian process model, log-EI and a Latin hypercube design - #382
Merged
Conversation
…d the upper confidence bound
A public Gaussian process, documented as unstable for one release: a constant mean, an ARD Matérn 5/2 kernel by default or the squared exponential, learned or fixed noise, inputs scaled to the unit cube by the bounds of a Real genome and outputs standardized. The hyperparameters maximize the log marginal likelihood (Rasmussen and Williams, eq. 2.30) with the mean at its generalized least squares estimate, by genoxide's own L-BFGS-B on their logarithms with the analytic gradient of eq. 5.9, from a fixed or warm start and random starts on derived streams, the winner by value then start. The posterior mean and variance (eq. 2.25, 2.26, Algorithm 2.1) come with their gradients. Cholesky factorizations by linalg, with a jitter raised tenfold from 1e-10 of the diagonal when needed. L-BFGS-B gains a crate-private minimizer driven by hand with a supplied gradient, for such inner problems.
An ask / tell algorithm on Real genomes for expensive black-box functions: an initial design of 2(n + 1) points by default (the initial genomes, then a Latin hypercube), then one point per generation, chosen by fitting the Gaussian process of model::gp to every evaluation and maximizing an acquisition function from raw samples and the best point evaluated with L-BFGS-B and the acquisition's analytic gradient. Log-EI by default, EI, PI (maximized through its logarithm) and UCB, settable during a run. An output transform: the values standardized by default, or the logarithm of their distance above the best for objectives that span orders of magnitude. Invalid points enter the model at the worst value; a point is never asked twice. Reproducible on any platform and thread count, re-evaluation and checkpoints included. The acquisition functions gain their derivatives with respect to the mean and the standard deviation, log-EI's through an asymptotic series where the Mills ratio's difference cancels.
…ic functions genoxide's fitness functions are deterministic, and a Gaussian process that interpolates their values resolves the small differences near a minimum that a learned noise smooths over. Measured with Bo over 20 seeds and 80 evaluations to f* + 1e-4: Branin reached in 20 runs, the six-hump camel in [-3, 3] x [-2, 2] in 20 and Hartmann 3 in 20, against 14, 17 and 12 with noise learned from a least of 1e-6, which settled above that least (an absolute standard deviation of about 0.03 on Branin). Hartmann 6 reached its minimum in as many runs, far more precisely. Noise::Learned stays for noisy functions. The documented measurements of the log transform are redone with the new default, and the tests' budgets with them.
…ition per step Bo's defaults on Branin's function: 6 points of a Latin hypercube, then 24 chosen by log-EI, which end 9.6e-5 above a global minimum; the Gaussian process of the 30 evaluations, its mean minimized by L-BFGS-B from the best point, then evaluated once: 2.1e-5 above it, in 31 evaluations. Over seeds 1 to 20, the polish brings 14 runs within 1e-4 at 30 evaluations, against 4 by the search alone. Rust and Python print the same rows and write the same trace, whose frames hold the model's mean and the acquisition on a 25 x 25 grid, for a new plot kind of the site's player, `surrogate`, and a new category, `bayesian`.
gx.Bo(real, ...) with the Rust builder's settings, the acquisitions as "log-ei", "ei", gx.ProbabilityOfImprovement(xi) and gx.UpperConfidenceBound(beta), and gx.RunningBo for a control: the acquisition, which can change, the model that chose the last point and the acquisition's values. gx.model.gp.GaussianProcess fits and queries the Rust model on numpy arrays. A run in Python gives the bits of the same run in Rust, which tests on both sides check.
The genoxide program runs Bo with every builder setting: the acquisition as
"log-ei", "ei", { type = "pi", xi } or { type = "ucb", beta }, the kernel, the
noise as a number or { learned = <least> }, the output transform and the starts
of both maximizations. Checkpoints resume a run as it would have gone on.
AGENTS.md gains a section on Bo with a program that runs in CI (the search, then a polish of the model's mean), rows in both method tables and in troubleshooting, and the panics of a point of the wrong length. The README, the crate docs, the Python README, both llms.txt, the features list and the packages' descriptions name Bayesian optimization; the roadmap marks the first part of batch B done and lists the second. The plan records what was verified in Rasmussen and Williams (2006), Jones, Schonlau and Welch (1998), Loeppky, Sacks and Welch (2009), Snoek, Larochelle and Adams (2012) and Srinivas et al. (2010), and where the implementation departs from its design notes, with the measurements: no noise by default, the log transform's offset, re-evaluation.
…'t factor The final factorization of a fit can only fail if rounding defeats a jitter of the diagonal's scale; the fit then returns Error::InvalidSetting and Bo asks a random point instead. The module docs' link to Lbfgsb loses its redundant target, which the doc build rejected.
tachsin
force-pushed
the
feat/bo-foundations
branch
from
October 2, 2026 09:55
9c2d994 to
27852c1
Compare
tachsin
marked this pull request as ready for review
October 2, 2026 09:55
Merged
tachsin
added a commit
that referenced
this pull request
Oct 2, 2026
…ization, completing batch B (#415) The second part of batch B in the [optimization plan](docs/optimization-plan.md), after #382: Bayesian optimization in batches, asynchronously, with constraints, and on integer genes. **Batches:** `.batch(q)` with `.fantasy(bo::Fantasy::KrigingBeliever)`, the default, or `ConstantLiar(Lie::Min | Mean | Max)` (Ginsbourger, Le Riche and Carraro 2010, Algorithms 1 and 2, checked). - Each point of a batch is chosen after the earlier ones are added with a fantasized value, through `GaussianProcess::with_points`. Hyperparameters are not refit within a batch. - With `Engine::parallel(true)`, a batch is evaluated in parallel, with the same bits on any number of threads. - `batch(1)` reproduces single-point runs bit for bit. - The believer is the default on evidence. With batches of 4 over 20 seeds, it ties the lowest lie and beats the mean and highest lies. Under `AsyncEngine` it reached Hartmann 3 in 5 of 5 runs, against 2 of 5 for the lowest lie. **Asynchronous:** `Bo` implements `Incremental`, so `AsyncEngine` proposes a point while others are pending, with the pending points fantasized. A checkpoint holds them, and a resumed run matches an uninterrupted one. `Incremental` gains defaulted `prepare`, `wants` and `receive_evaluation`, and `AsyncEngine` evaluates the wanted extras. **Constraints** (Gardner et al. 2014, checked): - Constraint values come from `Constrained` or a problem's own. Each constraint gets its own GP. - EI is multiplied by the probability of feasibility, and log-EI adds its logarithm. Before any feasible point exists, that probability alone is maximized. - UCB with constraints is an error. **Integer genes:** `Bo<R: bo::Space = Real>`, with `Space` implemented for `Real` and `Integer`. - The model sees lattice points only (Garrido-Merchán and Hernández-Lobato 2020, eq. 7). - The acquisition is maximized on the lattice: exhaustively when the lattice is small, otherwise by raw samples and a ±1 hill climb. - A fully evaluated lattice ends the run as `Converged`. **In Python and the CLI:** - Python: `gx.Bo(..., batch=, fantasy=)`, `run(f, constraints=m)`, and integer genomes. - CLI: `batch`, `fantasy`, `asynchronous = true`, integer genomes, and constraint values from fitness programs. **Examples** (Rust and Python identical, except the asynchronous one): - **`bo_hartmann6`:** batches of 4 come within 1e-4 of the global minimum in 15 rounds (74 evaluations), against 36 rounds one point at a time. The global minimum depends on the seed: batches reach it in 13 of 20 seeds and single points in 12 of 20, the rest stopping at the local minimum −3.2032. The README says so. - **`bo_constrained`:** Gramacy et al.'s toy problem. It comes within 1e-5 of f* = 0.5997880520 (checked with mpmath and a grid scan) in 21 evaluations, and every one of 20 seeds within 39. Given only the total violation, the same search ends 0.34 above. - **`bo_asynchronous`** (Rust only): Hartmann 3 to 1e-4 with 4 workers and evaluations of 10-50 ms, in about 0.4 s, against 0.8 s for batches of 4. Its timings are masked in CI. **Tests:** - Batch and fantasy behaviour; the same bits on 1, 2 and 8 threads. - Async with one worker reproducible. - Constrained BO on the Gramacy problem and CEC 2006 G24, and the probability of feasibility's gradients. - Integer BO. - Checkpoints, including pending points. - `portable_runs` entries and Python tests.
tachsin
added a commit
that referenced
this pull request
Oct 2, 2026
## 🤖 New release * `genoxide`: 0.12.0 -> 0.13.0 (✓ API compatible changes) * `genoxide-python`: 0.12.0 -> 0.13.0 <details><summary><i><b>Changelog</b></i></summary><p> ## `genoxide` <blockquote> ## [0.13.0](v0.12.0...v0.13.0) - 2026-10-02 ### <!-- 0 -->Added - *(bo)* Bayesian optimization with a Gaussian process model, log-EI and a Latin hypercube design ([#382](#382)) - *(bo)* batch, asynchronous, constrained and integer Bayesian optimization, completing batch B ([#415](#415)) ### <!-- 4 -->Documentation - correct citations found in an audit of every reference ([#412](#412)) </blockquote> </p></details> --- This PR was generated with [release-plz](https://github.com/release-plz/release-plz/). --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: tachsin <tachsinachmet@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first part of batch B in the optimization plan: Bayesian optimization, for expensive black-box functions. It follows the decisions in #381 and #379.
Foundations:
erf,erfcand the scalederfcxinmath.Real::latin_hypercube.algorithm::bo::acquisition. Log-EI's gradient goes through the Mills ratio, with an asymptotic series below z = −64, its coefficients checked with mpmath.model::gp, public and marked unstable for one release:predictandpredict_with_gradient.Bo:Output::Standardize(default) orOutput::Log, for objectives like Goldstein-Price.Noise: none by default, which differs from the plan. The jitter is the only nugget, so the model interpolates deterministic functions. Learned noise blurred the minimum: within 80 evaluations over 20 seeds, f* + 1e-4 was reached by:
Noise::Learnedstays available for noisy functions.In Python and the CLI:
gx.Boandgx.model.gp.GaussianProcess;type = "bo".The
bayesian_optimizationexample: Branin, with the posterior mean and log-EI drawn per step in a newsurrogateplot. It ends 2.1e-5 above the global minimum after 31 evaluations, using a final L-BFGS-B polish on the GP mean and one evaluation. Rust and Python give identical output and traces.Tests:
Log, each reaching f* + 1e-3.portable_runsentries, and a Rust and Python cross-check.Coming in B2: batch BO (Kriging believer, constant liar),
AsyncEngine, constrained BO, integer genes, and their examples.