diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ca465acd..557ef2c3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -79,8 +79,8 @@ jobs: # every examples//main.rs, which Cargo finds as the example , printing what its # output.txt has. They are built once, then run as many at a time as the runner has cores: # most are single-threaded, and the ones that use rayon give the same results on any number - # of threads. The asynchronous example's numbers depend on the machine (its thread count and - # how long each evaluation takes): its output is compared with every number replaced by #. + # of threads. The asynchronous examples' numbers depend on the machine (its thread count and + # how long each evaluation takes): their output is compared with every number replaced by #. # Every example is checked, and the step fails at the end if any differs. - run: cargo build --release --examples - run: | @@ -101,7 +101,7 @@ jobs: if [ "$(cat "actual/$example.status")" != 0 ]; then echo "::error::$example exited with status $(cat "actual/$example.status")" failed="$failed $example" - elif [ "$example" = asynchronous ]; then + elif [ "$example" = asynchronous ] || [ "$example" = bo_asynchronous ]; then diff <(numbers "examples/$example/output.txt") <(numbers "actual/$example.txt") \ || failed="$failed $example" else diff --git a/AGENTS.md b/AGENTS.md index 4907ba9f..7c11f674 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -42,7 +42,7 @@ fn main() -> genoxide::Result<()> { | A neural network's weights (neuroevolution) | `nn::Mlp::new([4, 8, 1], nn::Activation::Tanh)?.representation(-1.0..=1.0)?`, `nn::Elman` (recurrent) ([template](#neuroevolution-a-networks-weights-by-cma-es)) | `Reals` | none: `Cmaes` (up to a few hundred weights), `OpenEs` (thousands and more) | none | | A smooth function of many reals, its gradient noisy (mini-batches) or a step set by a learning-rate schedule (model fitting, up to millions of parameters) | `Real::uniform(n, lo..=hi)` ([template](#first-order-methods-adam-momentum-nesterov)) | `Reals` | none: `FirstOrder` | none | | Smooth, with gradients: very many reals (up to millions), few inequality constraints | `Real::uniform(n, lo..=hi)` ([template](#many-variables-few-constraints-mma)) | `Reals` | none: `Mma` | none | -| An expensive function of reals: tens to a few hundred evaluations, up to about 10 to 20 genes | `Real::new(...)` ([template](#bayesian-optimization-expensive-functions)) | `Reals` | none: `Bo` | none | +| An expensive function: tens to a few hundred evaluations, up to about 10 to 20 genes | `Real::new(...)` or `Integer::new(...)` ([template](#bayesian-optimization-expensive-functions)) | `Reals`, `Integers` | none: `Bo` | none | | Continuous problem | Method | |---|---| @@ -50,7 +50,7 @@ fn main() -> genoxide::Result<()> { | Smooth or not, a few genes, no gradient | `NelderMead` | | Multimodal, rotated or badly conditioned, up to a few hundred genes | `Cmaes` (with `Restarts::Ipop`), `De`; then `Lbfgsb` from the best to polish it | | Smooth, gradients of the score and of each constraint, very many genes and few inequality constraints | `Mma` (`Method::Gcmma` to converge from any start) | -| Expensive: tens to a few hundred evaluations, up to about 10 to 20 genes | `Bo` (Bayesian optimization); `bo::Output::Log` for values over orders of magnitude | +| Expensive: tens to a few hundred evaluations, up to about 10 to 20 genes | `Bo` (Bayesian optimization): `.batch(q)` for q evaluations at a time, an `AsyncEngine` for uneven ones, `Constrained` values for constraints; `bo::Output::Log` for values over orders of magnitude | | Smooth, solved best in stages (a smoothing, sharpness or penalty changed step by step) | `Continuation` around `FirstOrder`, `Lbfgsb`, `Mma`, `NelderMead` or `Cmaes` ([template](#continuation-stages-of-one-problem-the-state-kept)) | | Constrained beyond the box, without gradients | `De` or `Ga` with `(score, violation)` (Deb's rules) | @@ -657,7 +657,7 @@ fn main() -> genoxide::Result<()> { ### Asynchronous evaluation for slow, uneven fitness functions -`AsyncEngine` hands each worker a new genome as soon as it's done. Build with `build_steady()` (no scheme, no memetic). +`AsyncEngine` hands each worker a new genome as soon as it's done. Build with `build_steady()` (no scheme, no memetic), or run a `Bo` ([Bayesian optimization](#bayesian-optimization-expensive-functions)). ```rust use genoxide::prelude::*; @@ -1102,7 +1102,7 @@ fn main() -> genoxide::Result<()> { ### Bayesian optimization: expensive functions -`Bo` on `Real` genomes, for functions so expensive that tens to a few hundred evaluations must do (a simulation of minutes, an experiment). A Gaussian process (`model::gp`) models the function from every evaluation, and an acquisition function of its posterior picks the next point. Generation 0 evaluates an initial design: the `.initial_genomes(...)`, and a Latin hypercube for the rest of `.initial_points(n)`, 2(n + 1) for n searched genes by default (Loeppky et al. 2009's 10n is for an accurate model of the whole box, more than a minimum needs). Each later generation evaluates one point: the model is fitted (hyperparameters by maximum likelihood, genoxide's L-BFGS-B with the analytic gradient, from the last fit's and random starts), the acquisition evaluated at 1,000 random points and maximized by L-BFGS-B from the best 10 and the best point so far. A point is never asked twice. The model's fit costs O(N³) for N evaluations: up to a few hundred evaluations, in up to about 10 to 20 genes; for a cheap function, `Cmaes` or `De`. +`Bo` on `Real` or `Integer` genomes, for functions so expensive that tens to a few hundred evaluations must do (a simulation of minutes, an experiment). A Gaussian process (`model::gp`) models the function from every evaluation, and an acquisition function of its posterior picks the next point. Generation 0 evaluates an initial design: the `.initial_genomes(...)`, and a Latin hypercube for the rest of `.initial_points(n)`, 2(n + 1) for n searched genes by default (Loeppky et al. 2009's 10n is for an accurate model of the whole box, more than a minimum needs). Each later generation evaluates `.batch(q)` points, 1 by default: the model is fitted (hyperparameters by maximum likelihood, genoxide's L-BFGS-B with the analytic gradient, from the last fit's and random starts), the acquisition evaluated at 1,000 random points and maximized by L-BFGS-B from the best 10 and the best point so far; each further point of a batch is chosen after the ones before it are added to the model with a fantasized value, the hyperparameters kept (Ginsbourger et al. 2010). A point is never asked twice. The model's fit costs O(N³) for N evaluations: up to a few hundred evaluations, in up to about 10 to 20 genes; for a cheap function, `Cmaes` or `De`. | Setting | Default | |---|---| @@ -1111,8 +1111,14 @@ fn main() -> genoxide::Result<()> { | `.noise(model::gp::Noise::...)` | `Fixed(0.0)`: the model interpolates the values, as suits a deterministic function; `Learned { min }` (e.g. 1e-6) for a noisy one | | `.output(bo::Output::...)` | `Standardize`; `Log` (`ln(v − v_best + δ)`, δ the first quartile of the distances above the best) for values over orders of magnitude (Goldstein-Price's 3 to 10⁶) | | `.raw_samples(n)`, `.acquisition_starts(k)`, `.hyperparameter_starts(k)` | 1000, 10, 5 | +| `.batch(q)` | 1: q points per generation, evaluated at once with `Engine::parallel(true)`; more evaluations than one at a time, fewer rounds (`set_batch` in `.control`) | +| `.fantasy(bo::Fantasy::...)` | `KrigingBeliever` (the model's mean at a point not yet evaluated); `ConstantLiar(bo::Lie::Min \| Mean \| Max)` (the lowest, mean or highest value so far: the higher, the farther apart the points); for batches and an `AsyncEngine`'s pending points | -The model fits the scores to minimize (negated when maximizing); an invalid fitness enters it at the worst value of the others; a violation is ignored by the search (use a penalty). `bo.model()` is the `GaussianProcess` that chose the last point (`predict(&x)`, `predict_with_gradient`, `hyperparameters()`), `bo.acquisition_at(&x)` its acquisition; `reevaluate()` asks every point again. The model can be fitted on its own: `GaussianProcess::builder(real).fit(&points, &values)?` (unstable for one release). Batch and asynchronous Bayesian optimization, constraints and integer genes come in a later release. +- **Constraints:** a fitness function that gives the constraints' values one by one (`constraint::Constrained::new(m, \|x, g\| score)`, or a test problem of `problems::cec2006` or `problems::engineering`): a Gaussian process per constraint, the acquisition weighed by the probability of feasibility `Π P(gᵢ ≤ 0)` (Gardner et al. 2014; EI, log-EI or PI, not UCB), and before a feasible point, that probability alone. With only `(score, violation)`, the search ignores the violation (use a penalty), though `best()` uses Deb's rules. +- **Integer genes:** `Bo::builder(Integer::new(...)?)`: the genes rounded inside the kernel (Garrido-Merchán and Hernández-Lobato 2020), the acquisition maximized on the lattice by a hill climb; a lattice evaluated to its last point ends the run as `StopReason::Converged`. +- **Asynchronous:** `Bo` is `Incremental`: an `AsyncEngine` gives each worker a point as soon as it's done, chosen with the points still being evaluated fantasized; a generation is `initial_points` evaluations; reproducible with one worker. A checkpoint keeps the pending points, proposed again first when the run resumes. + +The model fits the scores to minimize (negated when maximizing); an invalid fitness enters it at the worst value of the others. `bo.model()` is the `GaussianProcess` that chose the last point (`predict(&x)`, `predict_with_gradient`, `hyperparameters()`), without the fantasies; `bo.constraint_models()` the constraints'; `bo.acquisition_at(&x)` its acquisition, `bo.probability_of_feasibility_at(&x)`; `reevaluate()` asks every point again. The model can be fitted on its own: `GaussianProcess::builder(real).fit(&points, &values)?` (unstable for one release). ```rust use genoxide::model::gp::GaussianProcess; @@ -1145,7 +1151,56 @@ fn main() -> genoxide::Result<()> { } ``` -Python: `gx.Bo(real, initial_points=None, acquisition="log-ei" | "ei" | gx.ProbabilityOfImprovement(xi) | gx.UpperConfidenceBound(beta), kernel="matern52", noise=0.0 | gx.model.gp.Learned(min), output="standardize" | "log", raw_samples=1000, acquisition_starts=10, hyperparameter_starts=5, ...)`; `gx.RunningBo` in `control` (`acquisition`, `model`, `acquisition_at(points)`); `gx.model.gp.GaussianProcess.fit(real, points, values)` (`predict(points)`, `predict_with_gradient(x)`). The `genoxide` program: `type = "bo"`. See `examples/bayesian_optimization`. +Batches, constraints, integer genes and asynchronous evaluation: + +```rust +use genoxide::constraint::Constrained; +use genoxide::prelude::*; +use genoxide::problems::{Hartmann3, Problem}; + +fn main() -> genoxide::Result<()> { + // 4 points a round, evaluated in parallel + let minimum = Hartmann3.optimum().unwrap().value(); + let bo = Bo::builder(Hartmann3.representation()).batch(4).minimize().seed(1).build()?; + let outcome = Engine::new(bo, Hartmann3) + .parallel(true) + .stop_when(Stop::target(minimum + 1e-3).or(Stop::evaluations(100))) + .run()?; + assert_eq!(outcome.stop_reason(), StopReason::Target); + + // minimize x₀ + x₁ inside the unit disc, its constraint's value given: (−√½, −√½) + let disc = Constrained::new(1, |x: &Reals, g: &mut [f64]| { + g[0] = x[0] * x[0] + x[1] * x[1] - 1.0; + x[0] + x[1] + }); + let bo = Bo::builder(Real::uniform(2, -2.0..=2.0)?).minimize().seed(1).build()?; + let outcome = Engine::new(bo, disc).stop_when(Stop::evaluations(30)).run()?; + assert!(outcome.best_fitness().is_feasible()); + assert!(outcome.best_fitness().score().unwrap() < -2f64.sqrt() + 1e-3); + + // integer genes in [−10, 10]³: the minimum (3, −1, 0) of a quadratic + let quadratic = |x: &Integers| { + let (a, b, c) = (x[0] as f64 - 2.6, x[1] as f64 + 1.3, x[2] as f64); + a * a + 2.0 * b * b + c * c + }; + let bo = Bo::builder(Integer::uniform(3, -10..=10)?).minimize().seed(1).build()?; + let outcome = Engine::new(bo, quadratic) + .stop_when(Stop::target(0.5).or(Stop::evaluations(60))) + .run()?; + assert_eq!(outcome.best_genome()[..], [3, -1, 0]); + + // asynchronous: 4 workers, each given a point as soon as it's done + let bo = Bo::builder(Hartmann3.representation()).minimize().seed(1).build()?; + let outcome = AsyncEngine::new(bo, Hartmann3) + .workers(4) + .stop_when(Stop::target(minimum + 1e-2).or(Stop::evaluations(100))) + .run()?; + assert_eq!(outcome.stop_reason(), StopReason::Target); + Ok(()) +} +``` + +Python: `gx.Bo(real_or_integer, initial_points=None, acquisition="log-ei" | "ei" | gx.ProbabilityOfImprovement(xi) | gx.UpperConfidenceBound(beta), kernel="matern52", noise=0.0 | gx.model.gp.Learned(min), output="standardize" | "log", raw_samples=1000, acquisition_starts=10, hyperparameter_starts=5, batch=1, fantasy="believer" | "liar-min" | "liar-mean" | "liar-max", ...)`; `bo.run(f, constraints=m)` with `f` returning `(value, g)` (a problem with constraints gives its own); `gx.RunningBo` in `control` (`acquisition`, `batch`, `fantasy`, `constraints`, `model`, `acquisition_at(points)`, `probability_of_feasibility_at(points)`); `gx.model.gp.GaussianProcess.fit(real, points, values)` (`predict(points)`, `predict_with_gradient(x)`); no asynchronous engine. The `genoxide` program: `type = "bo"` with `batch`, `fantasy`, `asynchronous = true`, `fitness.constraints` (a line is the value, then the constraints' values) and integer genomes. See `examples/bayesian_optimization`, `bo_hartmann6`, `bo_asynchronous` and `bo_constrained`. ### Ask / tell: evaluating outside the engine @@ -1247,6 +1302,9 @@ every = 50 | `Bo` stalls above the minimum of a function whose values span orders of magnitude | `.output(bo::Output::Log)` | | `Bo` on a noisy function chases the noise | `.noise(model::gp::Noise::Learned { min: 1e-6 })` | | `Bo` takes long per step after hundreds of evaluations | The model costs O(N³) for N evaluations: for a cheap function, `Cmaes` or `De` | +| `Bo` stays at a local minimum (Hartmann 6's −3.2032 in a third of the seeds) | Another run from another seed (a new design); then `Lbfgsb` from the best | +| `Bo` under an `AsyncEngine` drifts away from where it converges | Keep `bo::Fantasy::KrigingBeliever`: a low constant lie at every point being evaluated pushes each proposal away | +| `Bo` with constraints: `Error::InvalidSetting { setting: "acquisition", .. }` | The upper confidence bound can't be weighed by the probability of feasibility: log-EI (default), EI or PI | | Slow with a cheap fitness function | `--release`; `.parallel(true)` only for expensive fitness; `.parallel_breeding(true)` (`Ga`, `De`, `Es`) when breeding takes much of a generation | ## Guarantees to rely on diff --git a/Cargo.toml b/Cargo.toml index 09cf1b0d..d54c24ae 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -84,6 +84,18 @@ name = "mma" path = "examples/mma/main.rs" required-features = ["parallel"] +# the points of a batch evaluated in parallel +[[example]] +name = "bo_hartmann6" +path = "examples/bo_hartmann6/main.rs" +required-features = ["parallel"] + +# the contrast's batches evaluated in parallel +[[example]] +name = "bo_asynchronous" +path = "examples/bo_asynchronous/main.rs" +required-features = ["parallel"] + [[bin]] name = "genoxide" required-features = ["cli"] diff --git a/README.md b/README.md index a8307861..33030346 100644 --- a/README.md +++ b/README.md @@ -48,7 +48,7 @@ fn main() -> genoxide::Result<()> { - **Evolution strategies, CMA-ES, differential evolution, particle swarms:** with IPOP and BIPOP restarts, JADE, SHADE and L-SHADE. - **Local search:** hill climbing, simulated annealing, tabu search, iterated local search, and the Nelder-Mead simplex method with random restarts. - **Gradient-based:** L-BFGS-B for smooth functions with bounds, from a few variables to millions, and gradient descent, momentum, Nesterov, Adam and AdamW with learning-rate schedules, with gradients supplied or by finite differences; MMA and GCMMA, the method of moving asymptotes, for millions of variables with few constraints, from supplied gradients and constraint Jacobians. Continuation runs any of them through stages of one problem, a smooth version first and sharper ones after, with the optimizer's state kept between stages. -- **Bayesian optimization:** for expensive functions, where tens to a few hundred evaluations must do: a Gaussian process with a Matérn or squared exponential kernel, its hyperparameters by maximum likelihood, and the log expected improvement, expected improvement, probability of improvement or confidence bound, maximized by L-BFGS-B with their gradients; a log transform for values that span orders of magnitude. The Gaussian process can be fitted and queried on its own. +- **Bayesian optimization:** for expensive functions, where tens to a few hundred evaluations must do: a Gaussian process with a Matérn or squared exponential kernel, its hyperparameters by maximum likelihood, and the log expected improvement, expected improvement, probability of improvement or confidence bound, maximized by L-BFGS-B with their gradients; a log transform for values that span orders of magnitude. Batches of points evaluated in parallel (the Kriging believer and the constant liar), asynchronous evaluation, constraints modeled one by one (the probability of feasibility) and integer genes. The Gaussian process can be fitted and queried on its own. - **Genetic programming:** strongly typed trees of your own primitives, evolved into programs and formulas, with subtree and one-point crossover, subtree, point, hoist, shrink and constant mutation, bloat control, and fast evaluation over data. - **Neuroevolution:** multilayer perceptrons and recurrent networks whose weights evolve, NEAT, which evolves networks' structure too, and pole-balancing control tasks for them to solve. - **Multi-objective:** NSGA-II, NSGA-III, SPEA2, MOEA/D and SMS-EMOA, with quality indicators. diff --git a/ROADMAP.md b/ROADMAP.md index d3704f88..699d831a 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -43,9 +43,9 @@ From a review of existing libraries (e.g. genetic_algorithm, [issues #11 to #78] - **`Genome`:** the representation: bits (bit-packed), integers, bounded reals, permutations; planned: mixed (per-gene types), trees (GP), graphs (NEAT). - **`Fitness`:** totally ordered `f64`, single or multi-objective (`[f64; M]`, with the number of objectives fixed at compile time), optional constraint violation; batch and async evaluation. - **Operators:** `Select`, `Crossover`, `Mutate`, generic over the genome; survival is each algorithm's scheme. -- **`Algorithm`:** ask / tell state machines (GA, ES, CMA-ES, DE, PSO, NSGA-II, …; Nelder-Mead, L-BFGS-B; planned: SQP, Bayesian optimization). +- **`Algorithm`:** ask / tell state machines (GA, ES, CMA-ES, DE, PSO, NSGA-II, …; Nelder-Mead, L-BFGS-B, MMA, Bayesian optimization; planned: SQP). - **Derivatives (planned):** supplied gradients, finite differences, constraint and residual Jacobians, declared by the fitness function. -- **Models (planned):** Gaussian processes for Bayesian and surrogate-assisted optimization. +- **Models:** Gaussian processes (`model::gp`) for Bayesian optimization; planned: for surrogate-assisted evolution. - **`Engine`:** termination, parallel evaluation, observers, cancellation, and a hook that changes the algorithm between generations (parameter control, re-evaluation). - **`Observer`:** statistics, hall of fame, Pareto archive, logging, checkpoints. - **Errors:** one typed error enum; no `&'static str` errors, no panics in library code. @@ -228,7 +228,7 @@ The plan: [docs/gp-neuroevolution-plan.md](docs/gp-neuroevolution-plan.md). ### 0.13: Bayesian optimization - [x] EI, log-EI, UCB and PI; Latin hypercube designs; portable `erf`, `erfc` and `erfcx` (batch B, the parts that need neither linear algebra nor L-BFGS-B) - [x] Gaussian processes (`model::gp`, unstable for one release) and Bayesian optimization (`Bo`): an initial Latin hypercube of 2(n + 1) points, log-EI by default, an output transform, in Python (`gx.Bo`, `gx.model.gp`) and the `genoxide` program, with the `bayesian_optimization` example (batch B, its first part) -- [ ] Batch Bayesian optimization (Kriging believer, constant liar), `Incremental` for the asynchronous engine, constrained Bayesian optimization and integer genes, with the `bo_hartmann6`, `bo_asynchronous` and `bo_constrained` examples (batch B, its second part) +- [x] Batch Bayesian optimization (Kriging believer, constant liar), `Incremental` for the asynchronous engine, constrained Bayesian optimization and integer genes, with the `bo_hartmann6`, `bo_asynchronous` and `bo_constrained` examples (batch B, its second part) ### 0.14: Constrained nonlinear programming - [ ] SQP and the augmented Lagrangian, on the constrained test problems (batch C) diff --git a/docs/cli.md b/docs/cli.md index 2a0b53c0..755e9467 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -103,6 +103,16 @@ for line in sys.stdin: print(sum(gene * gene for gene in x), *gradient, 1.0 - x[0], *jacobian, flush=True) ``` +**Constraints without a gradient** (`constraints = m` alone): a line is the value, then the values of the m constraints, 1 + m numbers. `bo` models each constraint from them; other algorithms read the value and the violation, the sum of the positive values. The same problem for `bo`: + +```python +import sys + +for line in sys.stdin: + x = [float(gene) for gene in line.split()] + print(sum(gene * gene for gene in x), 1.0 - x[0], flush=True) +``` + A relative program path with a directory, like `./fitness`, is relative to the run file. ## The run file @@ -131,7 +141,7 @@ TOML, or JSON for files ending in `.json`, with the same structure. Unknown sett | `nan` | `"invalid"` | a NaN makes the genome invalid (`"invalid"`), or stops the run (`"error"`) | | `timeout` | `stop.time` | the longest wait for one answer, e.g. `"10m"`; a program that takes longer stops the run with an error | | `gradient` | `false` | each answer is the value, then its gradient: see [the fitness program](#the-fitness-program). With `builtin`, the built-in's analytic gradient (`sphere`, `rastrigin`, `rosenbrock`, `ackley` and `volume`) | -| `constraints` | 0 | with `gradient = true`, the number of inequality constraints whose values and Jacobian each answer has after the gradient; 1 for the built-in `volume` | +| `constraints` | 0 | the number of inequality constraints whose values each answer has after the value; with `gradient = true`, after the gradient, followed by their Jacobian; 1 for the built-in `volume`, which needs `gradient = true` | Set exactly one of `command` and `builtin`. @@ -148,7 +158,7 @@ Set exactly one of `command` and `builtin`. | `"pso"` | real | `population_size` | `ring` (off: the whole swarm) | | `"local-search"` | any | `neighbor` (a mutation) | `neighbors` (1), `acceptance` (`not-worse`), `restart` (off) | | `"nelder-mead"` | real | | `coefficients` (`"adaptive"`), `initial_step` (0.1), `initial_step_absolute`, `tolerance` (1e-9), `restarts` (none), `speculative` (false) | -| `"bo"` | real | | `initial_points` (2(n + 1)), `acquisition` (`"log-ei"`), `kernel` (`"matern52"`), `noise` (0), `output` (`"standardize"`), `raw_samples` (1000), `acquisition_starts` (10), `hyperparameter_starts` (5) | +| `"bo"` | real, integer | | `initial_points` (2(n + 1)), `acquisition` (`"log-ei"`), `kernel` (`"matern52"`), `noise` (0), `output` (`"standardize"`), `raw_samples` (1000), `acquisition_starts` (10), `hyperparameter_starts` (5), `batch` (1), `fantasy` (`"believer"`), `asynchronous` (false) | | `"lbfgsb"` | real | | `memory` (10), `gradients` (`"auto"`), `difference_step`, `gradient_tolerance` (1e-5), `function_tolerance` (2.220446049250313e-9), `max_line_search` (20), `restarts` (none) | | `"mma"` | real | `fitness.gradient = true` | `method` (`"mma"`), `asymptote_initial` (0.5), `asymptote_decrease` (0.7), `asymptote_increase` (1.2), `move_limit` (0.5), `constraint_cost` (1000), `kkt_tolerance` (1e-9), `step_tolerance` (1e-10), `restoration` (true), `parallel_sums` (false) | | `"first-order"` | real | | `step` (`{ type = "adam", learning_rate = 0.001 }`), `gradients` (`"auto"`), `difference_step`, `gradient_tolerance` (1e-6), `step_tolerance` (1e-12), `restarts` (none) | @@ -162,7 +172,7 @@ Set exactly one of `command` and `builtin`. - `de`: SHADE's settings by default, with genoxide's restarts. `strategy` builds the mutant vectors: `"rand1"`, `"best1"`, `{ p, archive }` (current-to-pbest/1, `pbest` among the best `p` of the population, 0 < p ≤ 1, and an archive of `archive` times the population, 0 or more) or `{ max_p, archive }` (the same with a random `p` per trial up to `max_p`, as in SHADE). `control` gives F and CR: `{ f, cr }` (fixed, 0 < f ≤ 2, 0 ≤ cr ≤ 1), `{ min_f, max_f, cr }` (a random F per trial, 0 < min_f ≤ max_f ≤ 2), `{ c }` (JADE's adaptation, 0 < c ≤ 1) or `{ memory }` (SHADE's, 1 to 2^24). `restarts` is `"never"` or `{ tolerance, patience }`: all but the best are replaced when every gene's values are within `tolerance` of its range of each other and the scores within `tolerance` relative to the best (0 or more), or after `patience` generations without a better best (at least 1). With `l_shade`, they replace L-SHADE's settings, whose restarts are `"never"`. - `cmaes`: n is the number of genes whose bounds differ. `restarts` is `"never"` (a converged run goes on sampling around its point), `"ipop"`, `"bipop"` or `"stop"`: the run ends when it converges, with the stop reason `"converged"`, which saves most of a budget on smooth problems without constraints; on flat or quantized fitness and with constraints, a run can still improve after it converges. `initial_step` is the initial step size as a fraction of each gene's range, greater than 0 and at most 1. `covariance` is `"full"`, which learns the correlations between genes (O(n²) per sample: up to a few hundred genes), or `"diagonal"`, sep-CMA-ES, which learns only each gene's variance (O(n) per sample: for separable problems and thousands of genes). - `nelder-mead`: the Nelder-Mead simplex method, a local method for a few genes (up to about 10, more with the adaptive coefficients). It starts from a random point, and ends the run by itself when it has converged, with the stop reason `"converged"`; a `[stop]` condition is still needed. `coefficients` is `"adaptive"` (Gao and Han's, for the number of genes whose bounds differ), `"standard"` (Nelder and Mead's: reflection 1, expansion 2, contraction 0.5, shrink 0.5) or `{ reflection, expansion, contraction, shrink }`, with reflection greater than 0, expansion greater than 1 and than reflection, and contraction and shrink greater than 0 and less than 1. `initial_step` is the size of the first simplex as a fraction of each gene's range, greater than 0 and at most 1; `initial_step_absolute` sets it as a distance instead, the same in every gene (positive), for a wide box around an unbounded problem; give one or neither. `tolerance` is the simplex size at which a run has converged, as a fraction of the initial step, greater than 0 and less than 1. A trial point outside the bounds is mirrored back in at the bound it crossed. `restarts = ` starts again from a random point that many times (at least 1) after converging, and the result is the best of all the runs. A generation is one round of evaluations: the first simplex, one trial point, or a shrink. `speculative = true` evaluates the reflection, the expansion and both contractions in one round, so that up to four workers evaluate at the same time. -- `bo`: Bayesian optimization, for expensive functions, where tens to a few hundred evaluations must do. Generation 0 evaluates an initial design of `initial_points` (at least 1; 2(n + 1) for the n genes whose bounds differ by default) from a Latin hypercube; each later generation fits a Gaussian process to every evaluation and evaluates the one point that maximizes the acquisition function, so a single worker does: more workers only share the design. `acquisition` is `"log-ei"` (the logarithm of the expected improvement, computed so that it keeps a gradient where the expected improvement underflows), `"ei"`, `{ type = "pi", xi }` (the probability of improving by more than `xi`, 0 or more) or `{ type = "ucb", beta }` (the confidence bound with `beta` 0 or more). `kernel` is `"matern52"` (Matérn's with ν = 5/2) or `"squared-exponential"`, with a length scale per gene. `noise` is the model's noise variance as a fraction of the values' variance: a number, 0 or more (0 interpolates the values, as suits a deterministic program), or `{ learned = }`, learned with the other hyperparameters from a least variance between 0 and 1. `output = "log"` models the logarithm of the values' distance above the best, for objectives that span orders of magnitude. The acquisition is evaluated at `raw_samples` random points (at least 1), then maximized by L-BFGS-B from the best `acquisition_starts` of them (at least 1, at most `raw_samples`) and from the best point so far; the model's likelihood from `hyperparameter_starts` starts (at least 1). A point is never evaluated twice. An invalid fitness enters the model at the worst value of the others. +- `bo`: Bayesian optimization, for expensive functions, where tens to a few hundred evaluations must do. Generation 0 evaluates an initial design of `initial_points` (at least 1; 2(n + 1) for the n genes whose bounds differ by default) from a Latin hypercube; each later generation fits a Gaussian process to every evaluation and evaluates the point that maximizes the acquisition function, or `batch` points: with one point a generation, a single worker does, and more workers only share the design. `acquisition` is `"log-ei"` (the logarithm of the expected improvement, computed so that it keeps a gradient where the expected improvement underflows), `"ei"`, `{ type = "pi", xi }` (the probability of improving by more than `xi`, 0 or more) or `{ type = "ucb", beta }` (the confidence bound with `beta` 0 or more). `kernel` is `"matern52"` (Matérn's with ν = 5/2) or `"squared-exponential"`, with a length scale per gene. `noise` is the model's noise variance as a fraction of the values' variance: a number, 0 or more (0 interpolates the values, as suits a deterministic program), or `{ learned = }`, learned with the other hyperparameters from a least variance between 0 and 1. `output = "log"` models the logarithm of the values' distance above the best, for objectives that span orders of magnitude. The acquisition is evaluated at `raw_samples` random points (at least 1), then maximized by L-BFGS-B from the best `acquisition_starts` of them (at least 1, at most `raw_samples`) and from the best point so far; the model's likelihood from `hyperparameter_starts` starts (at least 1). A point is never evaluated twice. An invalid fitness enters the model at the worst value of the others. `batch` (at least 1) is the number of points of each generation after the design, evaluated by the workers at once: each is chosen after the ones before it are added to the model with a fantasized value, `fantasy`: `"believer"` (the model's mean there) or `"liar-min"`, `"liar-mean"`, `"liar-max"` (the lowest, mean or highest value so far; the higher, the farther apart the points). With `asynchronous = true`, each worker gets a new point as soon as it's done, chosen with the points still being evaluated fantasized; a generation is then `initial_points` evaluations, and the run stops at its evaluation limit exactly. With `fitness.constraints`, a Gaussian process models each constraint and the acquisition is weighed by the probability that a point is feasible (before a feasible point, that probability alone is maximized); `acquisition` is then not `ucb`. On an integer genome, the genes are rounded to the nearest integer inside the model, and the acquisition is maximized on the lattice; once every point is evaluated, the run ends with the stop reason `"converged"`. - `lbfgsb`: L-BFGS-B, the limited-memory BFGS method with bounds, a local method for smooth functions from a few genes to millions. It starts from a random point and ends the run by itself when it has converged, with the stop reason `"converged"`; a `[stop]` condition is still needed. `gradients` is `"auto"` (the program's gradient with `fitness.gradient = true`, forward differences otherwise), `"supplied"` (needs `fitness.gradient = true`), `"forward"` or `"central"` (finite differences in any case: n or 2n more points per gradient, evaluated by the workers together with the point). `difference_step` is the relative step of finite differences, above 0, with `"forward"` or `"central"` (default √ε for forward, ε^(1/3) for central). `memory` is the number of correction pairs kept, at least 1 (3 to 20 is usual). A run has converged when the largest component of the projected gradient is at most `gradient_tolerance` (absolute, 0 or more; forward differences rarely meet much less than 1e-7 of the function's scale), or a step lowers the value by at most `function_tolerance` times max(|f|, 1) (0 or more), or no step lowers it. `max_line_search` is the most trial steps of a line search, at least 1. `restarts = ` starts again from a random point that many times (at least 1) after converging. - `mma`: Svanberg's method of moving asymptotes, for smooth problems with very many genes and few inequality constraints, from the program's gradient and, with `fitness.constraints`, the constraints' values and Jacobian. Each iteration is one evaluation. It starts from a random point and ends the run by itself when it has converged (the KKT residual within `kkt_tolerance`, or a step within `step_tolerance` of each gene's range), with the stop reason `"converged"`; a `[stop]` condition is still needed. A run that converges to a point infeasible by rounding ends with a restoration step onto the feasible side of its active constraints (`restoration = false` turns it off). `method = "gcmma"` is the globally convergent form: more evaluations, convergence from any start. `asymptote_initial` (above 0) is the asymptotes' first distance from the point and `move_limit` (above 0) the largest step, as fractions of each gene's range; `asymptote_decrease` (above 0 and at most 1) and `asymptote_increase` (at least 1) move them for oscillating and steady genes. `constraint_cost` (above 0) is the cost of the artificial variable that relaxes each constraint: above the constraints' multipliers at the solution. `parallel_sums = true` sums over the genes on several threads, with the same results. One worker is enough: it evaluates one genome at a time. - `first-order`: a first-order method, for smooth functions: each generation evaluates a point and its gradient, and steps along the gradient by a step rule, without a line search; the point stays in the bounds by projection. `step` is `{ type = "gradient", learning_rate }`, `{ type = "momentum", learning_rate, momentum }` (Polyak's heavy ball), `{ type = "nesterov", learning_rate, momentum }` (Nesterov's accelerated gradient, as Sutskever et al. state it), `{ type = "adam", learning_rate, beta1, beta2, epsilon }` (Kingma and Ba; all optional, 0.001, 0.9, 0.999 and 1e-8 by default) or `{ type = "adamw", learning_rate, beta1, beta2, epsilon, weight_decay }` (Loshchilov and Hutter's decoupled weight decay, `weight_decay` required), with the learning rate greater than 0 in the units of the genes, `momentum`, `beta1` and `beta2` from 0 to less than 1, `epsilon` greater than 0 and `weight_decay` 0 or more. `gradients` and `difference_step` are as for `lbfgsb`: the program's gradient with `fitness.gradient = true` (one evaluation per generation), or finite differences, asked in the same generation as the point (`"auto"` up to 10,000 genes whose bounds differ). The run ends by itself with the stop reason `"converged"` when the gradient's largest component is within `gradient_tolerance` (0 at a bound it points out of) or the last step moved no gene by more than `step_tolerance` relative to max(1, |x|), both 0 or more; a `[stop]` condition is still needed. `restarts = ` starts again from a random point that many times (at least 1) after converging. diff --git a/docs/features.md b/docs/features.md index 0ff2e45e..5b95a299 100644 --- a/docs/features.md +++ b/docs/features.md @@ -68,6 +68,10 @@ What genoxide has on main; [docs.rs](https://docs.rs/genoxide) documents the lat - **MMA and GCMMA** (`Mma`): Svanberg's method of moving asymptotes and its globally convergent form, for smooth problems with very many variables (millions) and few inequality constraints, from the gradients of the score and of every constraint: convex separable approximations with moving asymptotes and the notes' default constants, solved through their dual in the constraints' multipliers (O(n·m) per iteration, no n × n matrix, nothing allocated after the first iteration, the sums in fixed chunks, optionally in parallel with the same bits); artificial variables that keep every subproblem feasible; GCMMA's inner iterations for convergence from any start; convergence by the KKT residual or the step (`StopReason::Converged`); a restoration step onto the feasible side of the active constraints at the end; multipliers, asymptotes and the KKT residual to read, re-evaluation and checkpoints. Reproduces the 2002 paper's table 8.1 and the 1987 paper's cantilever beam. - **Continuation** (`Continuation`, the `Continue` trait): a local method run through stages of one problem, a parameter of the fitness function changed between them by a closure (a smoothing, a p-norm's p, a penalty weight), each stage ending when the method has converged or after its budget of generations, and the run only after the last (`StopReason::Converged`). Each stage starts by evaluating the point again on the changed function, with the method's state kept (`Keep::State`: Adam's averages and step count, MMA's asymptotes, a Nelder-Mead simplex, a CMA-ES distribution; L-BFGS-B's pairs optionally) or only the point (`Keep::Point`). Each stage's generations, evaluations, best value and end reported by a callback and after the run; checkpoints that resume in their stage; nothing allocated per step (tested at a million genes); in Python (`gx.Continuation` around `FirstOrder`, `Lbfgsb` or `Mma`, with Python stage callbacks). - **Bayesian optimization** (`Bo`): for expensive black-box functions, tens to a few hundred evaluations. An initial design (the initial genomes, then a Latin hypercube; 2(n + 1) points by default), then a point per generation: a Gaussian process fitted to every evaluation, and the acquisition function (log-EI by default, EI, PI through its logarithm, or the confidence bound, settable during a run) maximized from 1,000 raw samples by L-BFGS-B with its analytic gradient from the best 10 and the best point so far, on rayon with the winner by value then start. A point is never asked twice; invalid points enter the model at the worst value; an output transform (`bo::Output::Log`) for values over orders of magnitude. Reproducible to the bit on any platform and thread count, re-evaluation and checkpoints; the model that chose each point and its acquisition readable during a run; in Python (`gx.Bo`, `gx.RunningBo`) and the `genoxide` program (`type = "bo"`). + - **Batches** (`.batch(q)`): q points per generation, each chosen after the ones before it are added to the model with a fantasized value, the Kriging believer or the constant liar with the lowest, mean or highest value (Ginsbourger, Le Riche and Carraro 2010), evaluated at once by `Engine::parallel`. + - **Asynchronous evaluation:** `Bo` is `Incremental`, so an `AsyncEngine` gives each worker a point as soon as it's done, chosen with the points still being evaluated fantasized; reproducible with one worker; a checkpoint keeps the pending points. `Incremental` and the `AsyncEngine` gained the extras (`prepare`, `wants`, `receive_evaluation`). + - **Constraints:** the values of `Constrained` fitness functions and of the constrained test problems, a Gaussian process per constraint, and the acquisition weighed by the probability of feasibility (Gardner et al. 2014); before a feasible point, that probability alone. + - **Integer genomes:** the genes rounded inside the kernel (Garrido-Merchán and Hernández-Lobato 2020), the acquisition maximized on the lattice by a hill climb (every point of a small lattice at once), the run finished once the lattice is evaluated. - **Gaussian processes** (`model::gp`, unstable for one release): regression with a constant mean, an ARD Matérn 5/2 or squared exponential kernel, no noise by default (the model interpolates a deterministic function) or learned noise, inputs scaled to the unit cube and outputs standardized; hyperparameters by maximum marginal likelihood (Rasmussen and Williams, eq. 2.30 and 5.9) with genoxide's L-BFGS-B from fixed and seeded random starts; the posterior mean and variance with their gradients; Cholesky factorizations with a growing jitter. In Python as `gx.model.gp`. - **Test problems** (`problems`): Sphere, the axis-parallel ellipsoid, Schwefel 1.2 and 2.26, Rastrigin, Rosenbrock, Ackley, Griewank, Levy, Zakharov, Styblinski-Tang, Michalewicz, Himmelblau, Branin, Goldstein-Price, the six-hump camel, Hartmann's functions in 3 and 6 dimensions, Shekel's with 5, 7 and 10 wells, Easom, the eggholder, Schaffer's F6, Schwefel 2.21 and 2.22, Dixon-Price, Trid, Powell's singular function, Beale, Booth, Matyas, Bohachevsky's three functions, the three-hump camel, Langermann, Shekel's foxholes and Kowalik, each with its bounds, its analytic gradient (all but the eggholder and Schwefel 2.21 and 2.22, which aren't differentiable), known optimum (or best known, for those found numerically) and reference, in Rust and Python. - **Constrained test problems:** CEC 2006's g01-g24 (`problems::cec2006`), and the engineering design problems (`problems::engineering`): the welded beam in two forms, the pressure vessel, the tension/compression spring, the speed reducer, the gear train (integer), the three-bar truss, the cantilever beam and the car side impact, each with its optimum or best known solution and references, in Rust and Python. Those with inequalities only give their constraints' values one by one to the algorithms that use them. @@ -130,6 +134,9 @@ cargo run --release --example adam # 100,000 values smoothed to t cargo run --release --example mma # a million variables and one constraint, by MMA, to the closed-form minimum cargo run --release --example continuation # a tilted Rastrigin through 6 Gaussian smoothings, L-BFGS-B, to the global minimum cargo run --release --example bayesian_optimization # Branin in 31 evaluations, a Gaussian process and log-EI, then a polish of its mean +cargo run --release --example bo_hartmann6 # Hartmann 6-D in batches of 4 evaluated in parallel: 15 rounds where one point a round takes 36 +cargo run --release --example bo_asynchronous # 4 workers on evaluations of uneven duration, each new point chosen with the pending ones fantasized +cargo run --release --example bo_constrained # Gramacy et al.'s toy problem to within 1e-5 by the probability of feasibility cargo run --release --example pressure_vessel # constrained mixed discrete-continuous design, SHADE cargo run --release --example welded_beam # constrained design in two forms, SHADE cargo run --release --example gear_train # integer genome, genetic algorithm diff --git a/docs/optimization-plan.md b/docs/optimization-plan.md index a522b0dd..18b01c61 100644 --- a/docs/optimization-plan.md +++ b/docs/optimization-plan.md @@ -20,8 +20,11 @@ stage's closure (a fitness program would have to be told its stage), with its ex done. Of batch B, the first part (B1): the Gaussian process (`model::gp`, public and documented as unstable for one release), `Bo` with EI, log-EI, PI and UCB, the Latin hypercube design and the output transform, in Python and the `genoxide` program, with the `bayesian_optimization` example; -batch BO, `Incremental` for `AsyncEngine`, constrained BO and integer genes (B2) aren't yet, nor -the later batches. genoxide has +and the second part (B2): batch BO (Kriging believer, constant liar), `Incremental` for +`AsyncEngine` (with the extras path: `prepare`, `wants`, `receive_evaluation`), constrained BO by +the probability of feasibility and integer genes, in Python (not asynchronous: the package has no +`AsyncEngine`) and the `genoxide` program, with the `bo_hartmann6`, `bo_asynchronous` and +`bo_constrained` examples. Batch B is done; the later batches aren't yet. genoxide has evolutionary and population-based methods (GA, ES, CMA-ES, DE, PSO, local search, NSGA-II and the other multi-objective algorithms). This plan adds the other families of a general optimization library: local derivative-free methods, gradient-based methods, constrained @@ -391,6 +394,64 @@ is left for later: with the noise-free default, the best evaluated point is the evaluations, the six-hump camel in [−3, 3] × [−2, 2] 50, Hartmann 3 26, all within 80; Hartmann 6 reached in 6 runs of 10 within 100 (48), the others in its local minimum −3.2032. +**Batch B's second part (B2), read on 2026-10-02.** Ginsbourger, Le Riche and Carraro (2010), read +in the authors' preprint (HAL emse-00436126, the folder of papers has none): **verified**, section +4.2: the Kriging believer (Algorithm 1) adds each chosen point with the kriging mean, the constant +liar (Algorithm 2) with a fixed lie L, min, mean or max of the observations, "the larger L is, the +more explorative"; both update the model "without hyperparameter re-estimation", the point and +its value added to X and Y as if observed, so the best value includes the fantasies; on Branin +(Table 1), CL[min] gave the best improvement and the believer clustered its points. As +implemented: the kernel's hyperparameters and the standardization kept, the constant mean +estimated again by GLS, as the ordinary kriging of the paper does (the believer then leaves the +mean unchanged, which a unit test checks); the lies from the values the model fits (after the +output transform). Gardner, Kusner, Xu, Weinberger and Cunningham (2014), read: **verified**, +section 3.1: the expected constrained improvement EIC = PF × EI, with the best the lowest feasible +value; section 3.2: independent constraints, PF the product of the univariate normal +probabilities. The paper doesn't say what to do before a feasible point: maximizing PF alone is +the plan's rule (Gelbart et al. 2014, not in the folder, not read). Garrido-Merchán and +Hernández-Lobato (2020), read: **verified**, section 3.2, eq. 7: k′(x, x′) = k(T(x), T(x′)), T +rounding the integer genes to the nearest integer; the paper doesn't say how the acquisition is +maximized. Schonlau, Welch and Jones (1998): not in the folder, not read. Gramacy et al. (2016), +read: **verified**, section 1: the toy problem, f = x₁ + x₂ on [0, 1]², c₁ = 3/2 − x₁ − 2x₂ − +½ sin(2π(x₁² − 2x₂)), c₂ = x₁² + x₂² − 3/2, minima x_A ≈ (0.1954, 0.4044), f ≈ 0.5998, c₁ active, +x_B ≈ (0.7197, 0.1411), x_C = (0, 0.75). The extracted text loses the π: c₁(x_A) ≈ 0, as the paper +says, holds with it and not without. `tests/reference/gramacy_toy.py` solves c₁ = 0 with the +Lagrange condition in mpmath: f* = 0.59978805201006756 at (0.19512268347, 0.40466536854), and a +grid scan confirms it's global. + +As implemented, and where it differs from the design notes above: **the fantasy** is a setting +(`bo::Fantasy`, `bo::Lie`), the Kriging believer by default: over 20 seeds, batches of 4 to +f* + 1e-3 reached Branin in a median of 34 evaluations with the believer and the lowest lie (42 the +mean, 54 the highest), Hartmann 3 in 30 (30, 40, 40), Hartmann 6 within 200 in 13 runs of 20 +(13, 12, 11); but under an `AsyncEngine` with 4 workers, where every proposal has 3 points +fantasized, the lowest lie reached Hartmann 3 to f* + 1e-4 in 2 runs of 5 within 120 evaluations, +the believer in 5 of 5 (39 to 76). The constraints' models take their mean at a fantasized point +whatever the fantasy, and a fantasized point counts toward the best if its fantasized constraints +are met. The random streams of the first point of a batch are those of a single point, so a batch +of 1 gives B1's runs to the bit. **Asynchronous proposals**: the design first, then random points +until a result has a valid score, then the model with the pending points fantasized; a proposal +derives its streams from its index; a generation is `initial_points` evaluations. A checkpoint +holds the pending points, and a new run of the engine (`Incremental::prepare`) proposes them +again first, without counting them as new proposals, so a resumed run draws what an uninterrupted +one does (tested). **Constraints**: from `Provided::inequalities` (`Constrained`, the CEC 2006 and +engineering problems), a GP per constraint on its standardized values (an invalid evaluation's at +the worst), EI × PF, log-EI + ln PF, ln PI + ln PF; UCB with constraints is an error (a bound can +be negative); without values, the search ignores the violation as in B1. Equalities aren't +supported, as `Constrained` has none yet: two inequalities with a tolerance do. **Integer genes**: +`Bo` is generic over a sealed `bo::Space`, `Real` (the default type) and `Integer`, rather than a +second type, as `Ga` is generic over its representation; the model sees only lattice points, which +is the rounding inside the kernel, and the acquisition, constant between integers, is maximized +on the lattice: every point of a lattice of at most `raw_samples` points, else the raw samples and +a hill climb of ±1 steps in one gene from the best of them and the best point, not CMA-ES as the +notes say, since the acquisition is exactly a function of the lattice. A lattice evaluated to its +last point finishes the run. **The `genoxide` program**: `asynchronous = true` runs the +`AsyncEngine`, rather than `workers` > 1 as 2.11 says, since batches use the workers too; a fitness +program with `constraints = m` and no gradient writes the value and the m values. The B2 examples' +results: `bo_hartmann6`, 4 points a round to within 1e-4 in 15 rounds and 74 evaluations, one a +round in 36 and 50 (13 and 12 runs of 20, the others at −3.2032); `bo_constrained`, within 1e-5 in +21 evaluations (all 20 seeds within 39); `bo_asynchronous`, every run of 25 within 1e-4 in 38 to 61 +evaluations, in about half the time of batches of 4. + ### 1.6 Surrogate-assisted evolution, multi-fidelity, hybrids Brief and later: each needs the GP (batch B) and a design settled on real use. @@ -647,7 +708,8 @@ returns the largest relative error per gene, for users' tests. - **Bayesian optimization** asks the initial design as generation 0, then q points per generation (`batch(q)`, default 1). It implements `Incremental` as well: `propose` chooses a point with the pending ones fantasized (Kriging believer), so `AsyncEngine` keeps every worker - busy on slow evaluations; reproducible with one worker, like the steady-state GA. + busy on slow evaluations; reproducible with one worker, like the steady-state GA. (Implemented + in batch B; see 1.5.) Usage, as the rest of genoxide: @@ -1046,7 +1108,7 @@ a batch isn't done until every example reaches its optimum on the three platform | A1 | `linalg` (Cholesky, triangular solves, QR, the eigendecomposition moved from CMA-ES) with the dependency check of 2.8 (done, in-crate, #373: products, Cholesky with jitter, triangular solves and the eigendecomposition; QR, LDLᵀ and Bunch-Kaufman come with their first users); `Algorithm::is_finished` and `StopReason::Converged`; `Restarts` for local methods; Nelder-Mead (Gao-Han, 1965 option, speculative asks) | | `nelder_mead`, `nelder_mead_himmelblau` | | A2 | The extras of 2.3 (`Provided`, `Wanted`, `Extras`, `Evaluations`, `prepare`, `tell_evaluations`) in `Engine`; `Differentiable`, `gradient::Gradients`, finite differences, `gradient::check`; analytic gradients for the smooth problems; Moré-Thuente; L-BFGS-B | A1 | `lbfgsb`, `lbfgsb_bounds`, `polish` | | A3 | Momentum, Nesterov, Adam and AdamW (moved from D1); MMA and GCMMA, with supplied constraint values and Jacobians in `Extras` (the supplied half of batch C's constraint Jacobians; finite differences of constraints stay in C); `Continuation` and the `Continue` trait (2.12); the scale requirements of 2.13 for A2's and A3's methods, with the allocation test and the benchmarks | A2 | `adam`, `mma`, `continuation` | -| B | B1 (done): `model::gp` (kernels, hyperparameters by L-BFGS-B), portable `erf`/`erfc`/`erfcx` in `math`; `Bo` with EI, log-EI, UCB, PI; Latin hypercube; the output transform. B2: batch BO (Kriging believer, constant liar); `Incremental` for `AsyncEngine`; `Constrained` values (`constraint::Constraints`) and constrained BO; integer genes | A2 | B1: `bayesian_optimization`; B2: `bo_hartmann6`, `bo_asynchronous`, `bo_constrained` | +| B | B1 (done): `model::gp` (kernels, hyperparameters by L-BFGS-B), portable `erf`/`erfc`/`erfcx` in `math`; `Bo` with EI, log-EI, UCB, PI; Latin hypercube; the output transform. B2 (done): batch BO (Kriging believer, constant liar); `Incremental` for `AsyncEngine`; constrained BO from the values of `Constrained` and the test problems (`problems::Constraints` not moved to `constraint`, which nothing needed yet); integer genes | A2 | B1: `bayesian_optimization`; B2: `bo_hartmann6`, `bo_asynchronous`, `bo_constrained` | | C | Constraint Jacobians in `Extras` (supplied or by finite differences); the dense QP (Goldfarb-Idnani); SQP; the augmented Lagrangian (L-BFGS-B inner); `provides()` for CEC 2006 and the engineering problems; the Hock-Schittkowski selection | A2, B's `Constrained` | `sqp`, `sqp_welded_beam`, `augmented_lagrangian` | | D1 | BFGS, nonlinear CG with Hager-Zhang, trust-region Newton (Steihaug-CG and exact), Levenberg-Marquardt with `LeastSquares`; optional `dual` feature; MGH test set | A2 | `conjugate_gradient`, `trust_region`, `levenberg_marquardt`, `dual_numbers` | | D2 | BOBYQA, COBYLA, compass search / GPS, MADS with the progressive barrier, Powell's method (optional); basin hopping | A1, C's constraint values | `bobyqa`, `cobyla`, `mads` | diff --git a/docs/site/llms.txt b/docs/site/llms.txt index a3de1d2f..cb3e19a9 100644 --- a/docs/site/llms.txt +++ b/docs/site/llms.txt @@ -1,6 +1,6 @@ # genoxide -> Optimization for Rust and Python, in one library: genetic algorithms, evolution strategies, CMA-ES, differential evolution, particle swarms, local search, Nelder-Mead, genetic programming, neuroevolution, multi-objective optimization (NSGA-II, NSGA-III, MOEA/D, SPEA2, SMS-EMOA) gradient-based methods (L-BFGS-B, Adam and momentum, MMA and GCMMA, and continuation in stages) and Bayesian optimization with Gaussian processes, for expensive functions. A seed gives the same results, to the bit, on every platform and thread count. Invalid settings are errors, not panics. Each method is checked against its source paper. +> Optimization for Rust and Python, in one library: genetic algorithms, evolution strategies, CMA-ES, differential evolution, particle swarms, local search, Nelder-Mead, genetic programming, neuroevolution, multi-objective optimization (NSGA-II, NSGA-III, MOEA/D, SPEA2, SMS-EMOA) gradient-based methods (L-BFGS-B, Adam and momentum, MMA and GCMMA, and continuation in stages) and Bayesian optimization with Gaussian processes, for expensive functions (in batches or asynchronously, with constraints, on real or integer genes). A seed gives the same results, to the bit, on every platform and thread count. Invalid settings are errors, not panics. Each method is checked against its source paper. Install: `cargo add genoxide` (Rust), `pip install genoxide` (Python), or `cargo install genoxide --features cli` for a program that runs an optimization described in TOML with any program as the fitness function. genoxide is pre-1.0: check the version and use the guide below, which matches the latest release. diff --git a/examples/README.md b/examples/README.md index fc16fc62..d5f1509b 100644 --- a/examples/README.md +++ b/examples/README.md @@ -212,6 +212,9 @@ python examples/tsp_berlin52/main.py | [MMA on a million variables](mma/) | local | Rust, Python | [tachsin.gr](https://tachsin.gr/projects/genoxide/examples/mma) | | [Continuation by Gaussian smoothing](continuation/) | local | Rust, Python | [tachsin.gr](https://tachsin.gr/projects/genoxide/examples/continuation) | | [Bayesian optimization of Branin](bayesian_optimization/) | bayesian | Rust, Python | [tachsin.gr](https://tachsin.gr/projects/genoxide/examples/bayesian-optimization) | +| [Bayesian optimization in batches on Hartmann 6-D](bo_hartmann6/) | bayesian | Rust, Python | [tachsin.gr](https://tachsin.gr/projects/genoxide/examples/bo-hartmann6) | +| [Asynchronous Bayesian optimization](bo_asynchronous/) | bayesian | Rust | [tachsin.gr](https://tachsin.gr/projects/genoxide/examples/bo-asynchronous) | +| [Constrained Bayesian optimization](bo_constrained/) | bayesian | Rust, Python | [tachsin.gr](https://tachsin.gr/projects/genoxide/examples/bo-constrained) | The GPU example is a crate of its own, with wgpu as a dependency: `cargo run --release --manifest-path examples/gpu/Cargo.toml`. diff --git a/examples/bo_asynchronous/README.md b/examples/bo_asynchronous/README.md new file mode 100644 index 00000000..da4521e3 --- /dev/null +++ b/examples/bo_asynchronous/README.md @@ -0,0 +1,85 @@ +--- +title: Asynchronous Bayesian optimization +category: bayesian +summary: Keep 4 workers busy on evaluations of uneven duration with an AsyncEngine, each new point chosen with the ones still being evaluated fantasized, to the global minimum of Hartmann's 3-D function. +reference: "Ginsbourger, D., Le Riche, R. and Carraro, L. (2010). Kriging is well-suited to parallelize optimization. In Computational Intelligence in Expensive Optimization Problems, Springer: 131-162." +reference_url: "https://doi.org/10.1007/978-3-642-10701-6_6" +optimum: "−3.86278 at (0.11461, 0.55565, 0.85255) (best known)" +languages: [rust] +order: 292 +trace_note: "Recorded from another run like the one below: its times and values differ." +--- + +# Asynchronous Bayesian optimization + +## The problem + +Hartmann's function in 3 dimensions, whose global minimum is −3.86278; the [Hartmann +3-D](../hartmann3/) example gives its formula and constants. It stands for an expensive function +whose evaluations take different times, as simulations often do: each evaluation here sleeps for 10 +to 50 ms, a time drawn from a hash of the point's bits and a seed, so a point always takes as long. + +Four workers evaluate at a time. The example measures how long the search takes to come within +1e-4 of the minimum, and in how many evaluations. + +There's no Python version: the Python package has no asynchronous engine. + +## What makes it hard + +The waiting. In [batches](../bo_hartmann6/), a round of 4 points is evaluated together, and the +next round can't be chosen before the slowest of the 4 is done: workers that finish early sit idle, +here for up to 40 ms of every round. Choosing a point as soon as a worker is free means choosing it +while 3 others are still being evaluated, with their values unknown. + +## Representation + +A `Real` genome of 3 genes in [0, 1]: genoxide's `problems::Hartmann3`, evaluated in Rust, after +the sleep. + +## Algorithm + +`Bo` with its defaults, run by an `AsyncEngine` with 4 workers. `Bo` implements `Incremental`: it +proposes the initial design of 2(n + 1) = 8 points first, one at a time, then each proposal is a +point the model chooses. The model is fitted to every result so far, then the points still being +evaluated are added to it with a fantasized value, as the points of a batch are (Ginsbourger, Le +Riche and Carraro, 2010): the Kriging believer, the model's own mean there (`Fantasy`), which +removes the model's uncertainty at those points, so that the log expected improvement that +chooses the next point looks elsewhere. A result that arrives replaces its fantasy. + +With one worker, a seed gives the same run every time. With more, the order of the results depends +on the timing, so runs differ, as with any `AsyncEngine`. + +As a contrast, the same search in batches of 4 points (`.batch(4)`), each round evaluated at once +by `Engine::parallel(true)` and waiting for its slowest evaluation. Both stop within 1e-4 of the +minimum, or after 120 evaluations. + +## Output + +The first line gives the function, how long an evaluation takes, how many run at a time and the +global minimum. Then a line per run: its wall-clock time, its evaluations, its evaluations per +second, its best value and how far that is above the minimum. + +The times depend on the machine, and the asynchronous run's evaluations and best value on the order +of the results, so they change from run to run. + +[The project page](https://tachsin.gr/projects/genoxide/examples/bo-asynchronous) plays back +another run, recorded the same way: when each worker evaluated, and the best value as it fell. + +## Good results + +The asynchronous run reaches the minimum: in the run of `output.txt`, 7.1e-7 above it after 46 +evaluations, in 0.38 s. Over 25 runs, every one came within 1e-4, after 38 to 61 evaluations, +half of them within 44. + +The time is what the example compares. The batches came within 1e-4 too, after 60 evaluations +(every run, as they don't depend on the timing), in 0.78 s: twice as long, since every round waits +for its slowest evaluation, and more evaluations, since a batch's points are chosen 4 at a time, +and an asynchronous proposal knows every result that has arrived. The times vary with the machine; +the gap stays. + +The fantasy matters more here than in batches: every proposal has 3 points fantasized, never 0. +With the constant liar's lowest value instead of the believer, in 5 runs without the sleeps, only +2 came within 1e-4 in 120 evaluations: telling the model that the points being evaluated are as +good as the best keeps their region at the best value, with no improvement left there, and pushes +every proposal away from where the search is converging. With the believer, all 5 did, in 39 to +76 evaluations. diff --git a/examples/bo_asynchronous/main.rs b/examples/bo_asynchronous/main.rs new file mode 100644 index 00000000..db701002 --- /dev/null +++ b/examples/bo_asynchronous/main.rs @@ -0,0 +1,93 @@ +//! Asynchronous Bayesian optimization for an expensive function whose evaluations take different +//! times: an `AsyncEngine` gives each of 4 workers a new point as soon as it's done, chosen by the +//! model of the results so far with the points still being evaluated added at a lie. It reaches +//! the global minimum of Hartmann's 3-D function to within 1e-4. Then, as a contrast in time, +//! batches of 4 points, each evaluated together, which wait for their slowest evaluation. +//! +//! With `GENOXIDE_TRACE=`, it also writes a trace of the asynchronous run for the plot on +//! the example's page, with `trace.rs`. +//! +//! ```text +//! cargo run --release --example bo_asynchronous +//! ``` + +mod trace; + +use genoxide::prelude::*; +use genoxide::problems::{Hartmann3, Problem}; +use std::time::{Duration, Instant}; + +// the workers, how close to the global minimum, and the evaluations a run may take at most +const WORKERS: usize = 4; +const TOLERANCE: f64 = 1e-4; +const BUDGET: u64 = 120; +const SEED: u64 = 1; + +// Hartmann 3, taking 10 to 50 ms: a time drawn from the point's bits and a seed, so the same point +// always takes as long +fn simulation(x: &Reals) -> f64 { + let mut hash = 0x9e37_79b9_7f4a_7c15_u64 ^ SEED; + for gene in x.iter() { + hash = splitmix64(hash ^ gene.to_bits()); + } + std::thread::sleep(Duration::from_millis(10 + hash % 41)); + Hartmann3.evaluate(x) +} + +// a step of SplitMix64 (Steele, Lea and Flood, 2014): a well-mixed 64-bit hash +fn splitmix64(state: u64) -> u64 { + let mut z = state.wrapping_add(0x9e37_79b9_7f4a_7c15); + z = (z ^ (z >> 30)).wrapping_mul(0xbf58_476d_1ce4_e5b9); + z = (z ^ (z >> 27)).wrapping_mul(0x94d0_49bb_1331_11eb); + z ^ (z >> 31) +} + +fn main() -> Result<()> { + let minimum = Hartmann3.optimum().expect("known").value(); + println!( + "Hartmann's 3-D function, evaluations of 10 to 50 ms, {WORKERS} at a time: global \ + minimum {minimum:.6}" + ); + let bo = || { + Bo::builder(Hartmann3.representation()) + .batch(WORKERS) + .minimize() + .seed(SEED) + .build() + }; + let stop = || Stop::target(minimum + TOLERANCE).or(Stop::evaluations(BUDGET)); + + // with GENOXIDE_TRACE=, a trace of the run for the plot on the example's page + let mut trace = trace::Trace::from_env(WORKERS); + let start = Instant::now(); + let asynchronous = AsyncEngine::new(bo()?, trace.timed(simulation)) + .workers(WORKERS) + .stop_when(stop()) + .observe(&mut trace) + .run()?; + let elapsed = start.elapsed(); + report("asynchronous", &asynchronous, elapsed, minimum); + assert!(asynchronous.best_fitness().score().expect("valid") - minimum <= TOLERANCE); + trace.write(); + + // the contrast: batches of 4, a round waiting for its slowest evaluation + let start = Instant::now(); + let batches = Engine::new(bo()?, simulation) + .parallel(true) + .stop_when(stop()) + .run()?; + report("in batches", &batches, start.elapsed(), minimum); + Ok(()) +} + +fn report(name: &str, outcome: &Outcome, elapsed: Duration, minimum: f64) { + let best = outcome.best_fitness().score().expect("valid"); + println!( + "{name:>12}: {:.2} s, {} evaluations, {:.1} evaluations/s, best {best:.6}, {:.1e} above \ + the minimum", + elapsed.as_secs_f64(), + outcome.evaluations(), + outcome.evaluations() as f64 / elapsed.as_secs_f64(), + best - minimum + ); +} diff --git a/examples/bo_asynchronous/output.txt b/examples/bo_asynchronous/output.txt new file mode 100644 index 00000000..2b8efa21 --- /dev/null +++ b/examples/bo_asynchronous/output.txt @@ -0,0 +1,3 @@ +Hartmann's 3-D function, evaluations of 10 to 50 ms, 4 at a time: global minimum -3.862782 +asynchronous: 0.38 s, 46 evaluations, 120.1 evaluations/s, best -3.862781, 7.1e-7 above the minimum + in batches: 0.78 s, 60 evaluations, 76.7 evaluations/s, best -3.862782, 4.1e-7 above the minimum diff --git a/examples/bo_asynchronous/trace.json b/examples/bo_asynchronous/trace.json new file mode 100644 index 00000000..ab401671 --- /dev/null +++ b/examples/bo_asynchronous/trace.json @@ -0,0 +1,8 @@ +{"example":"bo_asynchronous","format":1,"log_y":true,"objective":"minimize","optimum":0.0,"plot":"timeline","problem":{"workers":4},"x_label":"evaluations","y_label":"best value's distance above the global minimum","frames":[ +{"best":1.27906,"evaluations":8,"generation":0,"seconds":0.0729372,"state":{"events":[[0,0.0002504,0.0116664],[1,0.0002246,0.0176221],[0,0.0117185,0.028797],[2,0.000198,0.0303993],[1,0.0176526,0.0390447],[3,0.0001757,0.0454433],[2,0.0304315,0.0588669],[0,0.0288275,0.0729019]]}}, +{"best":0.664915,"evaluations":16,"generation":1,"seconds":0.140918,"state":{"events":[[0,0.0002504,0.0116664],[1,0.0002246,0.0176221],[0,0.0117185,0.028797],[2,0.000198,0.0303993],[1,0.0176526,0.0390447],[3,0.0001757,0.0454433],[2,0.0304315,0.0588669],[0,0.0288275,0.0729019],[1,0.0447882,0.0739364],[3,0.0469975,0.081517],[2,0.0615079,0.0859659],[1,0.0782173,0.0884816],[3,0.0829946,0.111517],[2,0.0887309,0.115046],[0,0.0757197,0.119906],[1,0.0917494,0.14089]]}}, +{"best":0.0107636,"evaluations":24,"generation":2,"seconds":0.19978,"state":{"events":[[0,0.0002504,0.0116664],[1,0.0002246,0.0176221],[0,0.0117185,0.028797],[2,0.000198,0.0303993],[1,0.0176526,0.0390447],[3,0.0001757,0.0454433],[2,0.0304315,0.0588669],[0,0.0288275,0.0729019],[1,0.0447882,0.0739364],[3,0.0469975,0.081517],[2,0.0615079,0.0859659],[1,0.0782173,0.0884816],[3,0.0829946,0.111517],[2,0.0887309,0.115046],[0,0.0757197,0.119906],[1,0.0917494,0.14089],[2,0.118696,0.14895],[3,0.115135,0.151457],[0,0.122206,0.151462],[1,0.144218,0.168645],[0,0.158973,0.171264],[2,0.153093,0.192691],[3,0.156227,0.192691],[1,0.171296,0.197757]]}}, +{"best":0.000351714,"evaluations":32,"generation":3,"seconds":0.253528,"state":{"events":[[0,0.0002504,0.0116664],[1,0.0002246,0.0176221],[0,0.0117185,0.028797],[2,0.000198,0.0303993],[1,0.0176526,0.0390447],[3,0.0001757,0.0454433],[2,0.0304315,0.0588669],[0,0.0288275,0.0729019],[1,0.0447882,0.0739364],[3,0.0469975,0.081517],[2,0.0615079,0.0859659],[1,0.0782173,0.0884816],[3,0.0829946,0.111517],[2,0.0887309,0.115046],[0,0.0757197,0.119906],[1,0.0917494,0.14089],[2,0.118696,0.14895],[3,0.115135,0.151457],[0,0.122206,0.151462],[1,0.144218,0.168645],[0,0.158973,0.171264],[2,0.153093,0.192691],[3,0.156227,0.192691],[1,0.171296,0.197757],[0,0.172882,0.206174],[2,0.196262,0.216617],[0,0.209719,0.227246],[2,0.220414,0.230921],[1,0.202726,0.235064],[0,0.231788,0.241924],[3,0.19978,0.249311],[2,0.235204,0.25352]]}}, +{"best":0.000283752,"evaluations":40,"generation":4,"seconds":0.31971,"state":{"events":[[0,0.0002504,0.0116664],[1,0.0002246,0.0176221],[0,0.0117185,0.028797],[2,0.000198,0.0303993],[1,0.0176526,0.0390447],[3,0.0001757,0.0454433],[2,0.0304315,0.0588669],[0,0.0288275,0.0729019],[1,0.0447882,0.0739364],[3,0.0469975,0.081517],[2,0.0615079,0.0859659],[1,0.0782173,0.0884816],[3,0.0829946,0.111517],[2,0.0887309,0.115046],[0,0.0757197,0.119906],[1,0.0917494,0.14089],[2,0.118696,0.14895],[3,0.115135,0.151457],[0,0.122206,0.151462],[1,0.144218,0.168645],[0,0.158973,0.171264],[2,0.153093,0.192691],[3,0.156227,0.192691],[1,0.171296,0.197757],[0,0.172882,0.206174],[2,0.196262,0.216617],[0,0.209719,0.227246],[2,0.220414,0.230921],[1,0.202726,0.235064],[0,0.231788,0.241924],[3,0.19978,0.249311],[2,0.235204,0.25352],[1,0.239899,0.255161],[2,0.256846,0.271896],[1,0.261073,0.273477],[0,0.245086,0.276233],[3,0.253332,0.282425],[1,0.279839,0.297089],[3,0.287684,0.310163],[1,0.303486,0.319684]]}}, +{"best":1.07528e-06,"evaluations":44,"generation":4,"seconds":0.357043,"state":{"events":[[0,0.0002504,0.0116664],[1,0.0002246,0.0176221],[0,0.0117185,0.028797],[2,0.000198,0.0303993],[1,0.0176526,0.0390447],[3,0.0001757,0.0454433],[2,0.0304315,0.0588669],[0,0.0288275,0.0729019],[1,0.0447882,0.0739364],[3,0.0469975,0.081517],[2,0.0615079,0.0859659],[1,0.0782173,0.0884816],[3,0.0829946,0.111517],[2,0.0887309,0.115046],[0,0.0757197,0.119906],[1,0.0917494,0.14089],[2,0.118696,0.14895],[3,0.115135,0.151457],[0,0.122206,0.151462],[1,0.144218,0.168645],[0,0.158973,0.171264],[2,0.153093,0.192691],[3,0.156227,0.192691],[1,0.171296,0.197757],[0,0.172882,0.206174],[2,0.196262,0.216617],[0,0.209719,0.227246],[2,0.220414,0.230921],[1,0.202726,0.235064],[0,0.231788,0.241924],[3,0.19978,0.249311],[2,0.235204,0.25352],[1,0.239899,0.255161],[2,0.256846,0.271896],[1,0.261073,0.273477],[0,0.245086,0.276233],[3,0.253332,0.282425],[1,0.279839,0.297089],[3,0.287684,0.310163],[1,0.303486,0.319684],[0,0.283499,0.320702],[2,0.274948,0.324363],[3,0.316077,0.347279],[1,0.326304,0.356848]]}} +]} diff --git a/examples/bo_asynchronous/trace.rs b/examples/bo_asynchronous/trace.rs new file mode 100644 index 00000000..4aa5061c --- /dev/null +++ b/examples/bo_asynchronous/trace.rs @@ -0,0 +1,154 @@ +//! The trace of the asynchronous run for the plot on the example's page, written to the file that +//! `GENOXIDE_TRACE` names: when each worker evaluated what, a frame per generation (the initial +//! design's size in evaluations). + +use genoxide::observer::{Observer, Snapshot}; +use genoxide::prelude::*; +use genoxide::problems::{Hartmann3, Problem}; +use serde_json::{Value, json}; +use std::sync::{Arc, Mutex}; +use std::thread::{self, ThreadId}; +use std::time::Instant; + +pub struct Trace { + path: Option, + workers: usize, + start: Instant, + timeline: Arc>, + frames: Vec, +} + +impl Trace { + // a trace of a run on `workers`, for the file that GENOXIDE_TRACE names, or nothing to record + // if it isn't set + pub fn from_env(workers: usize) -> Self { + Self { + path: std::env::var("GENOXIDE_TRACE").ok(), + workers, + start: Instant::now(), + timeline: Arc::default(), + frames: Vec::new(), + } + } + + // `fitness`, recording when each evaluation starts and ends, and on which worker + pub fn timed(&self, fitness: F) -> impl Fn(&Reals) -> f64 + Send + Sync + use + where + F: Fn(&Reals) -> f64 + Send + Sync, + { + let (tracing, start, timeline) = (self.path.is_some(), self.start, self.timeline.clone()); + move |x: &Reals| { + let begin = start.elapsed().as_secs_f64(); + let value = fitness(x); + if tracing { + let end = start.elapsed().as_secs_f64(); + timeline.lock().expect("the timeline").record(begin, end); + } + value + } + } + + // writes the trace, if there's one + pub fn write(self) { + let Some(path) = self.path else { return }; + let settings = json!({ + "format": 1, + "example": "bo_asynchronous", + "objective": "minimize", + "x_label": "evaluations", + "y_label": "best value's distance above the global minimum", + "log_y": true, + "optimum": 0.0, + "plot": "timeline", + "problem": { "workers": self.workers }, + }); + write(&path, settings, self.frames); + } +} + +impl Observer for Trace { + // records a generation: its progress, the seconds since the start and every evaluation so far + fn observe(&mut self, snapshot: &Snapshot<'_, Reals>) { + if self.path.is_none() { + return; + } + let timeline = self.timeline.lock().expect("the timeline"); + let progress = snapshot.progress(); + let minimum = Hartmann3.optimum().expect("known").value(); + self.frames.push(json!({ + "generation": progress.generation(), + "evaluations": progress.evaluations(), + "seconds": self.start.elapsed().as_secs_f64(), + "best": progress.best().and_then(Fitness::score).map(|best| best - minimum), + "state": { "events": timeline.events }, + })); + } +} + +// the evaluations so far: [worker, start, end], in seconds since the run started, with the workers +// numbered in the order they first evaluated +#[derive(Default)] +struct Timeline { + events: Vec<(usize, f64, f64)>, + workers: Vec, +} + +impl Timeline { + // the evaluation that the current thread did + fn record(&mut self, start: f64, end: f64) { + let thread = thread::current().id(); + let worker = match self.workers.iter().position(|&worker| worker == thread) { + Some(worker) => worker, + None => { + self.workers.push(thread); + self.workers.len() - 1 + } + }; + self.events.push((worker, start, end)); + } +} + +// ---- the same in every example's trace --------------------------------------------------------- + +// writes the settings and the frames to `path`, a frame per line +fn write(path: &str, settings: Value, frames: Vec) { + let frames: Vec = frames.iter().map(to_json).collect(); + let settings = to_json(&settings); + let head = &settings[..settings.len() - 1]; + let text = format!("{head},\"frames\":[\n{}\n]}}\n", frames.join(",\n")); + std::fs::write(path, text).expect("the trace is written"); +} + +// compact JSON with sorted keys, and numbers rounded to 6 significant digits and written as +// Python writes them (7542.0, 1e-08) +fn to_json(value: &Value) -> String { + let join = |items: Vec| items.join(","); + match value { + Value::Number(number) if number.is_f64() => python_float(number.as_f64().expect("f64")), + Value::Array(items) => format!("[{}]", join(items.iter().map(to_json).collect())), + Value::Object(map) => { + let entry = + |(key, item): (&String, &Value)| format!("{}:{}", json!(key), to_json(item)); + format!("{{{}}}", join(map.iter().map(entry).collect())) + } + other => other.to_string(), + } +} + +fn python_float(value: f64) -> String { + let rounded: f64 = format!("{value:.5e}").parse().expect("a number"); + let shortest = format!("{rounded:e}"); + let (mantissa, exponent) = shortest.split_once('e').expect("an exponent"); + let exponent: i32 = exponent.parse().expect("an exponent"); + if (-4..16).contains(&exponent) { + let text = rounded.to_string(); + if text.contains('.') { + text + } else { + text + ".0" + } + } else { + let sign = if exponent < 0 { '-' } else { '+' }; + format!("{mantissa}e{sign}{:02}", exponent.abs()) + } +} diff --git a/examples/bo_constrained/README.md b/examples/bo_constrained/README.md new file mode 100644 index 00000000..da5d7ad9 --- /dev/null +++ b/examples/bo_constrained/README.md @@ -0,0 +1,90 @@ +--- +title: Constrained Bayesian optimization +category: bayesian +summary: Find the minimum of Gramacy et al.'s toy problem, on the boundary of a wavy constraint, to within 1e-5 in 21 evaluations, with a Gaussian process per constraint and the probability of feasibility. +reference: "Gramacy, R. B., Gray, G. A., Le Digabel, S., Lee, H. K. H., Ranjan, P., Wells, G. and Wild, S. M. (2016). Modeling an augmented Lagrangian for blackbox constrained optimization. Technometrics 58(1): 1-11." +reference_url: "https://doi.org/10.1080/00401706.2015.1014065" +optimum: "0.599788 at (0.195123, 0.404665)" +languages: [rust, python] +order: 293 +--- + +# Constrained Bayesian optimization + +## The problem + +The toy problem of Gramacy et al. (2016, section 1): minimize a linear function of two variables +in the unit square, subject to two nonlinear constraints, + +```text +minimize f(x) = x₁ + x₂, x in [0, 1]² +subject to c₁(x) = 3/2 − x₁ − 2x₂ − ½ sin(2π(x₁² − 2x₂)) ≤ 0 + c₂(x) = x₁² + x₂² − 3/2 ≤ 0 +``` + +The paper gives its global minimizer as about (0.1954, 0.4044), where f is about 0.5998, c₁ is +active and c₂ isn't, and two local minimizers, about (0.7197, 0.1411), f ≈ 0.8609, and (0, 0.75) +on the bound. `tests/reference/gramacy_toy.py` solves c₁ = 0 with the Lagrange condition, which +says the two partial derivatives of c₁ are equal there, with mpmath from the paper's points: +the minimum is 0.5997880520 at (0.1951226835, 0.4046653685), c₂ = −1.298 there, and a scan of the +box on a grid of 1/2000 confirms it's the global one. The paper's coordinates differ from these in +the fourth digit, and its f agrees: along the boundary, f changes slowly. + +The objective is known and cheap here, but the method treats it as a black box, as for a +simulation whose output and constraints all come from one expensive run. + +## What makes it hard + +The minimum lies on the boundary of c₁, whose sine makes the feasible region's edge a wave: the +objective falls toward the infeasible corner (0, 0), so the best points are exactly where the +constraint stops them, and a search that ignores the constraint goes the wrong way. Bayesian +optimization has only the evaluations to learn where that boundary is. + +## Representation + +A `Real` genome of 2 genes in [0, 1]. The fitness function is a `constraint::Constrained` with 2 +constraints: it writes c₁ and c₂ into a slice and returns x₁ + x₂, and genoxide makes the fitness +`(score, violation)` of it, the violation the sum of the positive values, while `Bo` reads the +values one by one. In Python, the function returns `(value, g)` and `run` takes `constraints=2`. +Both use genoxide's portable sine, so both versions give the same run. + +## Algorithm + +`Bo` with its defaults. With the constraints' values, it fits a Gaussian process to the objective +and one to each constraint, and maximizes the log expected improvement over the best feasible +point plus the logarithm of the probability that a point is feasible, `P(c₁ ≤ 0) P(c₂ ≤ 0)` under +the constraints' models: the expected constrained improvement of Gardner et al. (2014), the +product of the expected improvement and the probability of feasibility, through its logarithm. +Until a feasible point is evaluated, the search maximizes the probability of feasibility alone. +The run stops within 1e-5 of the minimum, or after 60 evaluations. + +As a contrast, the same search with a fitness function that returns only `(score, violation)`: +without the values, `Bo` models the score alone, and only the best point found is chosen by Deb's +rules. + +## Output + +The first lines give the problem and the method. Then a row per evaluation: its number, the point, +x₁ + x₂, both constraints' values (feasible at 0 or below) and the best feasible value's distance +above the minimum so far. Evaluations 1 to 6 are the initial design; the rest are the points the +models chose. The last lines give the best feasible point found and how far it is from the +minimum, and the contrast's feasible points and best. + +[The project page](https://tachsin.gr/projects/genoxide/examples/bo-constrained) plays the run back: +at each step, the probability of feasibility over the square with the points so far, and the +acquisition that chose the next point. + +## Good results + +A good result is feasible and within 1e-5 of 0.599788. Three points of the design are feasible, +the best 0.96; the models then probe the line x₁ = 0, where the objective is lowest, and from +evaluation 11 on, they walk the boundary c₁ = 0 to the minimum: 1.3e-2 above it at evaluation 11, +3.9e-4 at 15, 8.4e-6 at 21, (0.194733, 0.405064), with c₁ just below 0. That point is 5.6e-4 from +the minimizer: along the boundary, f changes slowly. + +Over seeds 1 to 20, every search came within 1e-5, after 18 to 39 evaluations, half of them within +26; within 1e-3, after 13 to 35, half within 19. + +The contrast doesn't come close: with only the violation, the search models x₁ + x₂ alone and +heads for the infeasible corner, where the function is lowest. In its 21 evaluations, 6 points are +feasible, and the best of them is 0.944530, 3.4e-1 above the minimum. diff --git a/examples/bo_constrained/main.py b/examples/bo_constrained/main.py new file mode 100644 index 00000000..db021642 --- /dev/null +++ b/examples/bo_constrained/main.py @@ -0,0 +1,128 @@ +"""Constrained Bayesian optimization of the toy problem of Gramacy et al. (2016): a linear objective +on [0, 1]^2 with two constraints whose values the fitness function gives one by one, each modeled +by a Gaussian process. The search maximizes the log expected improvement over the best feasible +point plus the logarithm of the probability of feasibility, and reaches the global minimum, on the +boundary of a wavy constraint, to within 1e-5. Then, as a contrast, the same search told only the +total violation, which it ignores. + +With ``GENOXIDE_TRACE=``, it also writes a trace of its run for the plot on the example's +page, with trace.py. + + python examples/bo_constrained/main.py +""" + +import math + +import numpy as np + +import genoxide as gx + +from trace import Trace + +# the global minimum and its point (tests/reference/gramacy_toy.py, with mpmath) +MINIMUM = 0.5997880520100676 +MINIMIZER = [0.19512268347207176, 0.4046653685379958] +# how close to the minimum, and the evaluations the search may take at most +TOLERANCE = 1e-5 +BUDGET = 60 + + +def scientific(value): + """Two significant digits, e.g. 1.2e-7.""" + mantissa, exponent = f"{value:.1e}".split("e") + return f"{mantissa}e{int(exponent)}" + + +def constraints(x): + """The values of the two constraints g(x) <= 0, with genoxide's portable sine.""" + wave = gx.math.sin(2.0 * math.pi * (x[0] * x[0] - 2.0 * x[1])) + return np.array([1.5 - x[0] - 2.0 * x[1] - 0.5 * wave, x[0] * x[0] + x[1] * x[1] - 1.5]) + + +def toy(x): + """x1 + x2, and the constraints' values.""" + return x[0] + x[1], constraints(x) + + +def feasible(g): + return bool(np.all(g <= 0.0)) + + +print("Gramacy et al.'s toy problem: minimize x1 + x2 on [0, 1]^2 subject to") +print(" c1 = 1.5 - x1 - 2 x2 - sin(2 pi (x1^2 - 2 x2)) / 2 <= 0, c2 = x1^2 + x2^2 - 1.5 <= 0") +print( + f"global minimum {MINIMUM:.6f} at ({MINIMIZER[0]:.6f}, {MINIMIZER[1]:.6f}), on the boundary " + "c1 = 0" +) +print("6 points of a Latin hypercube, then a point per step by log-EI x P(feasible)") +print("evaluation x1 x2 f c1 c2 best feasible f - f*") +# with GENOXIDE_TRACE=, a trace of the run for the plot on the example's page +trace = Trace(MINIMIZER, MINIMUM, constraints) +shown = {"printed": 0, "best": math.inf} + + +def on_generation(progress): + # the points evaluated in this generation + for index in range(shown["printed"], len(progress.population)): + x = progress.population[index] + value, g = toy(x) + if feasible(g): + shown["best"] = min(shown["best"], float(value)) + gap = scientific(shown["best"] - MINIMUM) if math.isfinite(shown["best"]) else "none yet" + print( + f"{index + 1:>10} {x[0]:>9.6f} {x[1]:>9.6f} {value:>9.6f} {g[0]:>10.6f} " + f"{g[1]:>10.6f} {gap:>20}" + ) + shown["printed"] = len(progress.population) + + +space = gx.Real((0.0, 1.0), length=2) +bo = gx.Bo(space, objective="minimize", seed=1) +result = bo.run( + toy, + constraints=2, + target=MINIMUM + TOLERANCE, + evaluations=BUDGET, + on_generation=on_generation, + control=trace.record, +) +x = result.best_genome +value = float(x[0] + x[1]) +assert feasible(constraints(x)) +distance = math.hypot(x[0] - MINIMIZER[0], x[1] - MINIMIZER[1]) +print( + f"{result.evaluations} evaluations: the best feasible point ({x[0]:.6f}, {x[1]:.6f}), " + f"{value:.6f}, {scientific(value - MINIMUM)} above the minimum, {scientific(distance)} from " + "its point" +) +assert value - MINIMUM <= TOLERANCE +trace.write() + + +# the contrast: the same problem as (score, violation), without the constraints' values +def violation(x): + value, g = toy(x) + return value, max(g[0], 0.0) + max(g[1], 0.0) + + +evaluated = {} + + +def keep(progress): + evaluated["population"] = progress.population + + +blind = gx.Bo(space, objective="minimize", seed=1) +contrast = blind.run(violation, evaluations=result.evaluations, on_generation=keep) +count = sum(feasible(constraints(x)) for x in evaluated["population"]) +best = contrast.best_genome +g = constraints(best) +if feasible(g): + score = float(best[0] + best[1]) + summary = f"{score:.6f}, {scientific(score - MINIMUM)} above the minimum" +else: + summary = f"infeasible, by {scientific(max(g[0], 0.0) + max(g[1], 0.0))}" +print( + f"without the constraints' values: {count} feasible points of {contrast.evaluations}, the " + f"best {summary}" +) diff --git a/examples/bo_constrained/main.rs b/examples/bo_constrained/main.rs new file mode 100644 index 00000000..be583e11 --- /dev/null +++ b/examples/bo_constrained/main.rs @@ -0,0 +1,142 @@ +//! Constrained Bayesian optimization of the toy problem of Gramacy et al. (2016): a linear +//! objective on [0, 1]² with two constraints whose values the fitness function gives one by one, +//! each modeled by a Gaussian process. The search maximizes the log expected improvement over the +//! best feasible point plus the logarithm of the probability of feasibility, and reaches the +//! global minimum, on the boundary of a wavy constraint, to within 1e-5. Then, as a contrast, the +//! same search told only the total violation, which it ignores. +//! +//! With `GENOXIDE_TRACE=`, it also writes a trace of its run for the plot on the example's +//! page, with `trace.rs`. +//! +//! ```text +//! cargo run --release --example bo_constrained +//! ``` + +mod trace; + +use genoxide::constraint::Constrained; +use genoxide::math::sin; +use genoxide::prelude::*; +use std::cell::RefCell; +use std::f64::consts::PI; + +// the global minimum and its point (tests/reference/gramacy_toy.py, with mpmath) +const MINIMUM: f64 = 0.599_788_052_010_067_6; +const MINIMIZER: [f64; 2] = [0.195_122_683_472_071_76, 0.404_665_368_537_995_8]; +// how close to the minimum, and the evaluations the search may take at most +const TOLERANCE: f64 = 1e-5; +const BUDGET: u64 = 60; + +// x₁ + x₂, and the values of the two constraints g(x) <= 0 into `g` +fn toy(x: &Reals, g: &mut [f64]) -> f64 { + g[0] = 1.5 - x[0] - 2.0 * x[1] - 0.5 * sin(2.0 * PI * (x[0] * x[0] - 2.0 * x[1])); + g[1] = x[0] * x[0] + x[1] * x[1] - 1.5; + x[0] + x[1] +} + +fn main() -> Result<()> { + println!("Gramacy et al.'s toy problem: minimize x1 + x2 on [0, 1]^2 subject to"); + println!( + " c1 = 1.5 - x1 - 2 x2 - sin(2 pi (x1^2 - 2 x2)) / 2 <= 0, c2 = x1^2 + x2^2 - 1.5 <= 0" + ); + println!( + "global minimum {MINIMUM:.6} at ({:.6}, {:.6}), on the boundary c1 = 0", + MINIMIZER[0], MINIMIZER[1] + ); + println!("6 points of a Latin hypercube, then a point per step by log-EI x P(feasible)"); + println!( + "evaluation x1 x2 f c1 c2 best feasible f - f*" + ); + let bo = Bo::builder(Real::uniform(2, 0.0..=1.0)?) + .minimize() + .seed(1) + .build()?; + // with GENOXIDE_TRACE=, a trace of the run for the plot on the example's page + let trace = RefCell::new(trace::Trace::from_env()); + let mut printed = 0; + let mut best = f64::INFINITY; + let mut engine = Engine::new(bo, Constrained::new(2, toy)) + .stop_when(Stop::target(MINIMUM + TOLERANCE).or(Stop::evaluations(BUDGET))) + .control(|bo, progress| { + // the points evaluated in this generation + for index in printed..bo.population().len() { + let individual = &bo.population().as_slice()[index]; + let (x, fitness) = ( + individual.genome(), + individual.fitness().expect("evaluated"), + ); + let g = bo.constraint_values(index); + if fitness.is_feasible() { + best = best.min(fitness.score().expect("valid")); + } + let gap = if best.is_finite() { + format!("{:.1e}", best - MINIMUM) + } else { + "none yet".to_string() + }; + println!( + "{:>10} {:>9.6} {:>9.6} {:>9.6} {:>10.6} {:>10.6} {gap:>20}", + index + 1, + x[0], + x[1], + fitness.score().expect("valid"), + g[0], + g[1] + ); + } + printed = bo.population().len(); + trace.borrow_mut().record(bo, progress); + Ok(()) + }); + let outcome = engine.run()?; + drop(engine); + let x = outcome.best_genome(); + let value = outcome.best_fitness().score().expect("valid"); + assert!(outcome.best_fitness().is_feasible()); + let distance = (x[0] - MINIMIZER[0]).hypot(x[1] - MINIMIZER[1]); + println!( + "{} evaluations: the best feasible point ({:.6}, {:.6}), {value:.6}, {:.1e} above the \ + minimum, {distance:.1e} from its point", + outcome.evaluations(), + x[0], + x[1], + value - MINIMUM + ); + assert!(value - MINIMUM <= TOLERANCE); + trace.into_inner().write(); + + // the contrast: the same problem as (score, violation), without the constraints' values + let violation = |x: &Reals| { + let mut g = [0.0; 2]; + let score = toy(x, &mut g); + (score, g[0].max(0.0) + g[1].max(0.0)) + }; + let blind = Bo::builder(Real::uniform(2, 0.0..=1.0)?) + .minimize() + .seed(1) + .build()?; + let mut engine = + Engine::new(blind, violation).stop_when(Stop::evaluations(outcome.evaluations())); + let contrast = engine.run()?; + let feasible = engine + .algorithm() + .population() + .iter() + .filter(|individual| individual.fitness().is_some_and(Fitness::is_feasible)) + .count(); + let fitness = contrast.best_fitness(); + println!( + "without the constraints' values: {feasible} feasible points of {}, the best {}", + contrast.evaluations(), + if fitness.is_feasible() { + format!( + "{:.6}, {:.1e} above the minimum", + fitness.score().expect("valid"), + fitness.score().expect("valid") - MINIMUM + ) + } else { + format!("infeasible, by {:.1e}", fitness.violation()) + } + ); + Ok(()) +} diff --git a/examples/bo_constrained/output.txt b/examples/bo_constrained/output.txt new file mode 100644 index 00000000..d299bdf4 --- /dev/null +++ b/examples/bo_constrained/output.txt @@ -0,0 +1,28 @@ +Gramacy et al.'s toy problem: minimize x1 + x2 on [0, 1]^2 subject to + c1 = 1.5 - x1 - 2 x2 - sin(2 pi (x1^2 - 2 x2)) / 2 <= 0, c2 = x1^2 + x2^2 - 1.5 <= 0 +global minimum 0.599788 at (0.195123, 0.404665), on the boundary c1 = 0 +6 points of a Latin hypercube, then a point per step by log-EI x P(feasible) +evaluation x1 x2 f c1 c2 best feasible f - f* + 1 0.193732 0.765545 0.959277 -0.204590 -0.876409 3.6e-1 + 2 0.971102 0.545241 1.516343 -0.161850 -0.259672 3.6e-1 + 3 0.451976 0.061188 0.513164 0.679542 -1.291974 3.6e-1 + 4 0.541108 0.210487 0.751595 0.898457 -1.162897 3.6e-1 + 5 0.039182 0.427164 0.466345 0.207200 -1.315996 3.6e-1 + 6 0.674825 0.892300 1.567125 -0.520084 -0.248411 3.6e-1 + 7 0.000000 0.508147 0.508147 0.534808 -1.241786 3.6e-1 + 8 0.000000 0.000000 0.000000 1.500000 -1.500000 3.6e-1 + 9 0.000000 0.341614 0.341614 0.360133 -1.383300 3.6e-1 + 10 0.000000 0.767732 0.767732 -0.145956 -0.910588 1.7e-1 + 11 0.212447 0.400488 0.612935 -0.013086 -1.294476 1.3e-2 + 12 0.190897 0.387819 0.578716 0.034616 -1.313155 1.3e-2 + 13 0.205325 0.398267 0.603592 -0.001670 -1.299225 3.8e-3 + 14 0.191724 0.406987 0.598711 0.001595 -1.297603 3.8e-3 + 15 0.193398 0.406780 0.600178 -0.000220 -1.297127 3.9e-4 + 16 0.560359 0.000000 0.560359 0.479528 -1.185998 3.9e-4 + 17 0.000000 0.201622 0.201622 1.382342 -1.459348 3.9e-4 + 18 0.172931 0.425271 0.598202 0.024969 -1.289239 3.9e-4 + 19 0.194758 0.405116 0.599874 -0.000089 -1.297951 8.6e-5 + 20 0.194735 0.405068 0.599803 -0.000009 -1.297998 1.5e-5 + 21 0.194733 0.405064 0.599796 -0.000001 -1.298003 8.4e-6 +21 evaluations: the best feasible point (0.194733, 0.405064), 0.599796, 8.4e-6 above the minimum, 5.6e-4 from its point +without the constraints' values: 6 feasible points of 21, the best 0.944530, 3.4e-1 above the minimum diff --git a/examples/bo_constrained/trace.json b/examples/bo_constrained/trace.json new file mode 100644 index 00000000..893cecdb --- /dev/null +++ b/examples/bo_constrained/trace.json @@ -0,0 +1,18 @@ +{"example":"bo_constrained","format":1,"log_y":true,"objective":"minimize","optimum":0.0,"plot":"surrogate","problem":{"bounds":[[0.0,1.0],[0.0,1.0]],"grid":25,"minima":[[0.195123,0.404665]],"minima_label":"the global minimum","panels":[{"shading":"shading: more likely feasible","title":"The probability of feasibility"},{"shading":"shading: more worth evaluating","title":"Log-EI + log P(feasible)"}]},"x_label":"evaluations","y_label":"error of the best feasible point","frames":[ +{"best":0.359489,"evaluations":6,"generation":0,"state":{"best":[0.193732,0.765545],"newest":[0.674825,0.8923],"population":[[0.193732,0.765545],[0.971102,0.545241],[0.451976,0.0611882],[0.541108,0.210487],[0.0391818,0.427164],[0.674825,0.8923]]}}, +{"best":0.359489,"evaluations":7,"generation":1,"state":{"acquisition":[[558,556,554,552,550,548,546,543,541,538,535,532,528,525,520,516,510,504,497,488,477,461,433,298,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[102,99,97,93,90,86,82,77,72,66,59,50,39,23,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[786,782,778,774,769,764,758,751,742,731,715,687,533,0,0,0,0,0,0,0,0,0,0,0,0],[1000,996,992,987,982,976,969,960,948,932,905,751,0,0,0,0,0,0,0,0,0,0,0,0,0],[998,994,989,984,978,971,962,950,934,907,753,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[994,989,984,978,971,962,950,934,907,753,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[989,983,977,970,961,950,934,906,753,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[982,976,969,960,949,932,905,752,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[977,970,961,950,934,906,753,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[971,962,950,934,907,754,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[962,950,934,907,755,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[950,934,907,756,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[934,907,758,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[907,763,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[769,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]],"best":[0.193732,0.765545],"mean":[[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,5,5,5],[949,949,950,950,950,950,950,950,950,950,950,950,951,951,951,951,951,951,951,951,951,951,951,951,950],[1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000],[1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000],[984,984,984,984,984,984,984,984,984,984,984,984,984,984,984,984,984,984,984,984,984,984,984,984,984],[956,956,956,956,956,956,956,956,956,956,956,956,956,956,956,956,956,956,956,956,956,956,956,956,945],[985,985,985,985,985,985,985,985,985,985,985,985,985,985,985,985,985,985,985,985,985,985,985,970,836],[1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,987,866,616],[1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,993,896,656,427],[1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,999,937,710,467,307],[1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,975,779,519,340,237],[1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,993,850,585,383,264,197],[1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,995,896,656,440,303,223,177],[998,998,998,998,998,998,998,998,998,998,998,998,998,998,998,998,997,984,902,712,506,355,260,203,168]],"newest":[0.0,0.508147],"population":[[0.193732,0.765545],[0.971102,0.545241],[0.451976,0.0611882],[0.541108,0.210487],[0.0391818,0.427164],[0.674825,0.8923],[0.0,0.508147]]}}, +{"best":0.359489,"evaluations":8,"generation":2,"state":{"acquisition":[[1000,994,987,979,969,958,944,928,909,891,877,870,871,879,890,899,906,911,912,910,905,894,871,735,0],[990,981,970,957,939,913,875,813,699,464,0,0,165,531,703,787,832,857,869,873,869,851,714,0,0],[981,971,958,941,919,888,840,763,624,343,0,0,0,297,562,693,764,803,824,831,820,686,0,0,0],[977,966,953,936,916,889,853,804,739,653,553,468,443,489,570,648,708,749,773,775,648,0,0,0,0],[976,965,952,936,916,890,855,808,740,643,501,310,125,83,222,408,552,643,687,585,0,0,0,0,0],[979,969,956,941,921,896,861,810,731,593,308,0,0,0,0,0,267,511,489,0,0,0,0,0,0],[985,977,966,953,938,918,893,860,815,753,665,548,426,364,410,509,589,516,0,0,0,0,0,0,0],[993,986,977,968,957,945,931,915,897,878,859,840,824,810,798,779,636,0,0,0,0,0,0,0,0],[997,991,984,977,970,962,954,945,936,927,917,906,893,877,850,698,0,0,0,0,0,0,0,0,0],[989,982,977,973,970,966,962,956,950,943,934,922,906,879,725,0,0,0,0,0,0,0,0,0,0],[801,382,798,914,944,953,955,953,949,942,932,917,891,737,0,0,0,0,0,0,0,0,0,0,0],[474,445,722,851,902,925,934,937,935,929,917,892,740,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,507,759,848,888,907,915,915,907,887,736,0,0,0,0,0,0,0,0,0,0,0,0,0],[428,546,697,796,851,881,897,902,898,880,732,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[868,872,881,892,901,906,907,901,881,733,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[939,936,934,932,929,923,913,891,741,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[963,959,954,948,940,928,904,753,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[976,971,965,956,943,919,767,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[983,978,973,963,940,787,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[980,973,961,937,788,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[968,955,930,781,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[950,925,777,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[920,774,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[773,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]],"best":[0.193732,0.765545],"mean":[[198,178,156,134,110,87,65,46,31,21,16,15,17,22,32,45,62,81,100,120,140,159,176,193,209],[159,134,108,81,54,31,13,3,0,0,0,0,0,0,0,3,11,25,43,64,85,107,128,148,168],[136,110,84,58,35,17,6,1,0,0,0,0,0,0,0,0,2,8,18,33,50,69,90,111,132],[128,103,78,55,34,19,8,3,1,0,0,0,0,0,0,0,1,3,8,16,28,44,63,83,105],[132,106,81,57,37,21,9,3,1,0,0,0,0,0,0,0,0,0,2,6,15,28,45,66,88],[150,123,95,69,46,26,12,4,1,0,0,0,0,0,0,0,0,0,1,4,11,24,42,64,87],[186,158,130,101,74,49,28,14,5,1,0,0,0,0,0,0,0,1,5,12,25,43,64,87,111],[236,211,184,156,129,103,80,60,43,31,22,17,14,13,14,18,24,32,45,60,78,98,119,141,163],[281,259,237,215,194,175,158,143,131,120,112,107,104,103,106,110,118,127,139,153,169,185,202,219,236],[244,224,213,211,213,216,217,218,217,216,215,215,215,217,221,226,232,240,249,260,271,283,296,308,320],[2,0,3,53,123,175,210,234,251,264,274,284,292,301,309,318,328,338,349,360,371,382,392,402,410],[0,0,0,12,49,98,146,188,223,253,279,303,324,344,363,381,399,418,436,454,471,487,501,512,520],[0,0,0,1,15,46,89,135,180,222,261,298,331,363,394,425,457,489,523,560,599,639,675,700,706],[0,0,0,4,18,46,85,130,176,222,266,308,348,387,426,466,507,553,605,669,752,863,980,1000,1000],[19,23,33,50,75,108,146,187,230,272,314,354,393,432,470,508,547,589,633,681,732,784,829,854,853],[122,130,145,165,191,222,256,292,329,365,400,434,466,497,527,555,582,608,633,656,677,693,705,710,707],[259,271,289,311,338,369,401,433,464,492,518,542,563,582,599,614,628,639,650,658,663,666,666,664,655],[411,437,468,506,547,586,618,641,656,667,674,680,684,688,691,693,694,694,693,690,686,680,673,661,541],[589,658,758,888,982,994,976,938,900,870,848,831,818,808,799,791,783,774,764,753,741,727,710,605,325],[687,759,844,927,977,991,990,982,968,954,941,930,921,913,906,898,888,876,862,845,825,802,708,408,170],[677,720,766,811,851,885,910,929,943,954,963,971,977,982,985,985,983,978,967,949,925,849,522,221,91],[642,674,707,741,775,809,842,875,906,934,960,981,994,999,1000,1000,1000,1000,1000,998,968,650,281,114,53],[604,629,656,684,714,745,778,812,847,883,918,950,976,993,999,1000,1000,1000,1000,990,733,330,134,61,33],[563,583,604,627,651,676,703,731,760,790,820,849,876,900,918,930,935,932,915,718,349,146,67,36,22],[523,538,553,570,588,606,625,644,664,683,702,720,736,750,761,767,769,750,606,340,155,73,39,24,17]],"newest":[0.0,0.0],"population":[[0.193732,0.765545],[0.971102,0.545241],[0.451976,0.0611882],[0.541108,0.210487],[0.0391818,0.427164],[0.674825,0.8923],[0.0,0.508147],[0.0,0.0]]}}, +{"best":0.359489,"evaluations":9,"generation":3,"state":{"acquisition":[[0,0,0,0,345,545,656,723,768,803,834,864,892,915,931,940,945,946,944,939,930,916,890,742,0],[99,127,251,385,487,551,579,567,499,328,9,45,537,777,867,905,921,927,926,921,908,884,735,0,0],[762,756,757,761,765,762,750,719,653,512,213,0,338,674,809,865,891,901,901,893,872,725,0,0,0],[894,888,882,877,871,862,849,830,802,763,714,672,670,715,774,820,848,862,864,850,707,0,0,0,0],[943,937,930,922,912,900,883,860,826,775,697,580,453,432,550,676,756,796,804,674,0,0,0,0,0],[969,962,955,946,935,921,901,874,832,761,617,231,0,0,0,282,614,714,623,0,0,0,0,0,0],[984,978,971,963,953,940,924,903,875,833,773,685,581,524,570,655,710,612,0,0,0,0,0,0,0],[995,989,983,976,968,959,948,936,921,905,888,870,854,840,828,810,668,0,0,0,0,0,0,0,0],[1000,995,990,984,978,971,964,956,948,939,928,917,904,888,861,710,0,0,0,0,0,0,0,0,0],[994,989,985,982,980,976,971,965,959,951,941,929,913,885,732,0,0,0,0,0,0,0,0,0,0],[886,558,887,954,968,971,969,965,958,950,939,924,897,744,0,0,0,0,0,0,0,0,0,0,0],[668,646,850,922,947,956,957,955,950,941,926,901,748,0,0,0,0,0,0,0,0,0,0,0,0],[0,159,759,878,918,935,941,941,935,924,900,748,0,0,0,0,0,0,0,0,0,0,0,0,0],[588,702,818,880,911,925,930,928,919,897,747,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[899,903,912,922,928,930,928,919,897,747,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[950,948,947,945,942,936,925,902,751,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[968,965,960,955,947,934,910,759,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[977,972,967,959,947,922,771,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[981,976,971,962,943,792,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[977,970,959,938,790,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[966,952,928,781,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[947,922,775,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[917,771,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[767,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]],"best":[0.193732,0.765545],"mean":[[0,0,0,0,0,0,0,0,1,2,5,11,25,48,79,114,146,175,199,218,233,245,254,262,268],[0,0,0,0,0,0,0,0,0,0,0,0,0,2,18,53,93,129,159,183,202,218,230,240,249],[0,0,0,1,1,1,1,0,0,0,0,0,0,0,5,23,52,84,113,139,162,181,197,211,223],[14,13,12,11,10,9,7,4,2,1,0,0,0,0,2,9,23,43,67,92,116,139,160,178,195],[52,46,41,36,30,24,17,10,5,1,0,0,0,0,0,0,3,12,29,52,77,103,128,151,172],[102,92,81,69,56,43,29,16,6,1,0,0,0,0,0,0,0,3,14,35,60,87,114,139,162],[160,146,131,114,95,76,55,36,19,8,2,0,0,0,0,0,3,11,27,49,74,101,127,151,173],[220,206,189,171,151,131,110,90,70,54,41,31,26,25,27,34,45,60,79,100,122,144,166,186,205],[269,255,241,226,212,197,183,169,156,144,134,127,123,121,123,129,138,149,162,177,193,209,225,240,254],[249,234,230,235,241,245,245,242,238,234,229,226,224,224,226,230,236,243,252,262,273,284,295,305,314],[18,0,22,127,201,242,264,277,284,289,292,296,299,303,308,314,320,328,337,346,356,365,373,380,386],[0,0,9,64,133,189,231,261,284,303,318,331,343,355,366,379,391,404,418,432,445,458,468,475,478],[0,0,1,24,75,132,184,228,265,295,322,346,368,389,409,430,452,475,500,528,559,592,622,641,639],[0,0,5,29,72,124,175,222,263,300,333,363,391,419,445,473,502,535,573,621,685,780,924,1000,996],[36,45,64,95,133,176,220,261,300,336,369,399,428,455,482,510,538,569,603,642,686,734,778,803,790],[145,157,176,202,234,268,304,339,372,403,431,457,481,504,525,546,567,587,606,625,642,655,662,662,653],[261,276,297,323,354,387,420,451,478,502,522,540,555,568,580,591,601,610,617,622,624,624,620,613,601],[377,403,436,478,524,568,600,621,633,640,645,648,650,652,653,653,653,651,648,643,636,627,615,602,476],[495,552,639,778,941,978,936,879,835,805,784,770,759,751,743,736,727,718,706,692,676,659,639,547,195],[566,632,720,831,929,967,962,939,914,893,877,867,859,854,848,840,830,815,796,773,746,717,641,274,55],[563,608,658,710,760,802,834,857,875,890,904,918,931,944,953,958,955,943,920,885,842,771,376,80,15],[533,565,599,634,671,707,743,778,814,850,887,924,960,986,998,1000,1000,1000,998,975,914,498,109,21,5],[497,522,547,575,604,635,667,702,739,778,821,867,914,957,987,999,1000,999,992,956,582,133,25,6,2],[461,479,499,521,544,568,594,621,651,682,714,748,782,814,841,860,867,860,834,549,144,29,7,2,1],[426,440,455,471,488,506,524,544,564,584,604,624,642,658,670,678,679,656,445,148,35,9,3,1,1]],"newest":[0.0,0.341614],"population":[[0.193732,0.765545],[0.971102,0.545241],[0.451976,0.0611882],[0.541108,0.210487],[0.0391818,0.427164],[0.674825,0.8923],[0.0,0.508147],[0.0,0.0],[0.0,0.341614]]}}, +{"best":0.167944,"evaluations":10,"generation":4,"state":{"acquisition":[[0,0,0,0,139,412,562,654,717,766,810,852,890,920,941,954,961,964,962,958,949,936,910,758,0],[0,0,4,179,315,400,436,418,321,80,0,0,435,747,863,912,933,942,943,938,927,903,752,0,0],[679,673,677,686,692,691,676,636,550,364,0,0,106,595,785,862,897,912,915,909,889,741,0,0,0],[845,839,834,830,825,816,802,780,746,695,630,571,570,642,734,803,845,867,874,862,720,0,0,0,0],[899,892,884,876,867,854,835,808,767,702,600,444,270,249,431,617,730,788,805,681,0,0,0,0,0],[913,905,896,887,876,861,839,805,750,653,452,0,0,0,0,105,552,691,616,0,0,0,0,0,0],[896,888,884,880,876,867,854,832,799,748,672,562,438,387,470,596,679,598,0,0,0,0,0,0,0],[788,802,837,868,887,897,899,896,888,877,864,850,838,828,820,806,669,0,0,0,0,0,0,0,0],[0,401,808,900,933,946,950,950,946,941,933,924,913,898,872,722,0,0,0,0,0,0,0,0,0],[701,753,910,960,976,981,981,978,973,966,957,946,930,903,750,0,0,0,0,0,0,0,0,0,0],[774,0,869,966,986,990,989,986,980,971,961,945,918,765,0,0,0,0,0,0,0,0,0,0,0],[540,383,820,933,966,977,980,978,973,964,950,924,772,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,725,881,932,954,962,963,958,946,923,773,0,0,0,0,0,0,0,0,0,0,0,0,0],[375,600,786,876,919,939,948,948,940,919,770,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[870,882,903,922,936,943,944,937,917,769,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[949,949,951,953,954,951,942,921,773,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[978,975,972,969,963,952,930,781,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[992,988,983,977,966,943,794,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[1000,996,992,984,965,814,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[999,993,982,961,813,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[989,976,952,804,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[971,946,798,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[941,793,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[789,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]],"best":[0.0,0.767732],"mean":[[0,0,0,0,0,0,0,0,0,0,2,5,14,32,60,93,127,157,183,204,220,233,243,251,257],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,10,37,74,110,141,167,187,204,217,228,236],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,12,36,65,94,121,144,164,181,195,208],[2,2,2,2,2,2,1,1,0,0,0,0,0,0,1,3,12,28,49,73,97,120,141,160,176],[10,9,8,7,6,4,3,2,1,0,0,0,0,0,0,0,1,6,18,36,59,83,107,130,150],[15,13,11,9,8,6,3,2,0,0,0,0,0,0,0,0,0,1,7,21,42,67,92,116,139],[10,9,9,8,8,7,6,4,2,1,0,0,0,0,0,0,1,5,15,31,53,77,102,126,149],[1,1,3,7,12,16,19,19,18,16,13,11,10,11,13,18,27,39,55,75,96,118,140,161,181],[0,0,1,16,40,61,75,84,88,89,88,88,88,90,94,101,110,122,136,151,167,184,201,217,232],[0,0,21,78,129,163,183,194,199,201,201,201,201,202,205,209,215,223,232,242,253,264,275,286,296],[1,0,8,99,181,229,257,272,281,286,290,293,296,299,303,308,313,320,327,336,344,352,360,367,374],[0,0,3,48,125,190,237,271,295,314,329,342,353,364,375,386,397,409,421,433,445,455,464,470,473],[0,0,0,15,62,122,180,229,270,305,334,360,383,405,426,447,468,491,516,544,574,606,635,653,649],[0,0,1,15,51,102,158,211,259,302,340,375,407,437,466,496,528,562,603,653,721,818,951,1000,999],[10,16,30,56,94,141,192,242,289,333,373,410,444,477,508,540,572,606,643,685,732,782,827,850,836],[83,95,115,145,183,227,274,320,363,404,441,474,504,533,559,584,608,631,653,674,692,706,713,712,702],[193,209,233,266,307,353,400,443,482,515,544,568,589,607,623,637,649,660,668,674,678,677,673,664,651],[321,348,386,438,499,559,607,640,661,675,684,691,696,700,704,706,707,706,703,699,692,682,670,655,510],[468,531,629,789,959,989,960,916,880,853,835,822,812,804,797,790,783,773,762,748,733,714,694,615,115],[572,647,748,866,958,987,985,972,955,938,925,915,908,902,896,889,879,865,847,825,799,770,715,202,11],[586,638,695,754,808,853,885,908,923,936,946,956,965,973,978,980,977,968,951,923,885,833,305,20,1],[561,598,637,677,717,756,794,830,864,897,929,958,981,996,1000,1000,1000,1000,1000,988,948,402,28,1,0],[526,553,583,614,646,679,713,749,786,824,864,905,944,976,995,1000,1000,1000,997,976,442,31,2,0,0],[486,508,531,554,580,606,634,662,692,724,756,789,821,851,877,894,900,894,870,378,28,2,0,0,0],[448,464,481,499,518,537,557,578,599,620,641,661,679,695,706,714,715,678,271,26,2,0,0,0,0]],"newest":[0.0,0.767732],"population":[[0.193732,0.765545],[0.971102,0.545241],[0.451976,0.0611882],[0.541108,0.210487],[0.0391818,0.427164],[0.674825,0.8923],[0.0,0.508147],[0.0,0.0],[0.0,0.341614],[0.0,0.767732]]}}, +{"best":0.0131468,"evaluations":11,"generation":5,"state":{"acquisition":[[0,0,0,0,0,0,166,404,563,676,759,821,868,902,926,940,944,938,900,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,264,600,769,850,883,866,0,0,0,0,0,0,0],[761,746,731,714,692,661,616,546,435,257,0,0,3,392,646,767,795,0,0,0,0,0,0,0,0],[905,895,885,872,858,841,821,796,765,731,697,673,667,685,714,720,0,0,0,0,0,0,0,0,0],[942,930,916,899,879,853,822,780,726,656,565,461,369,334,357,0,0,0,0,0,0,0,0,0,0],[945,928,909,885,855,815,761,681,556,338,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[924,904,882,857,828,794,753,703,641,567,486,407,336,0,0,0,0,0,0,0,0,0,0,0,0],[823,801,795,802,815,827,836,842,843,839,827,788,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,590,803,884,921,937,943,940,926,883,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[770,800,902,958,983,992,992,984,968,922,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[644,0,775,949,989,998,995,980,935,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[516,459,731,882,941,960,958,922,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,377,728,845,887,874,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[511,593,706,792,838,834,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[923,921,919,910,875,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[984,974,956,912,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[1000,981,935,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[999,953,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[981,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]],"best":[0.212447,0.400488],"mean":[[0,0,0,0,0,0,0,0,0,0,0,1,3,9,21,38,60,86,113,140,165,187,206,223,237],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,6,22,49,81,113,143,169,192,210,225],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,8,25,49,76,104,129,151,170,187],[4,3,2,2,1,1,1,0,0,0,0,0,0,0,0,1,4,10,21,36,54,73,93,112,130],[10,8,6,4,3,2,1,0,0,0,0,0,0,0,0,0,0,0,2,7,15,27,42,59,77],[11,8,5,3,2,1,0,0,0,0,0,0,0,0,0,0,0,0,0,1,4,11,22,36,52],[7,5,3,2,1,0,0,0,0,0,0,0,0,0,0,0,0,1,2,6,12,22,34,49,64],[1,0,0,0,1,1,2,3,3,4,5,7,9,11,14,18,24,31,39,49,61,73,87,100,115],[0,0,0,1,5,15,28,41,53,64,74,83,91,99,107,114,122,130,139,148,157,167,177,186,196],[0,0,7,32,70,107,139,164,183,198,209,219,226,233,239,245,250,256,261,266,271,277,282,288,293],[0,0,0,30,100,164,211,246,272,291,307,319,329,338,346,353,359,365,371,376,380,385,388,392,394],[0,0,0,7,39,90,144,193,236,272,303,330,354,375,394,412,428,443,457,470,481,491,498,504,507],[0,0,0,0,5,24,59,102,148,194,238,280,320,357,393,429,464,499,534,569,603,635,661,679,686],[0,0,0,1,7,22,47,81,121,163,207,252,296,341,387,434,485,541,605,682,780,899,992,1000,1000],[20,24,32,44,61,83,111,142,176,213,252,291,332,373,415,458,503,550,598,649,699,747,785,810,818],[116,125,138,154,173,195,219,247,276,306,337,369,402,434,466,497,527,556,582,607,628,645,659,667,671],[241,254,269,285,304,325,347,370,395,420,445,469,493,515,536,555,572,586,599,609,618,623,627,628,627],[402,423,445,467,490,513,535,557,577,595,610,622,632,639,645,649,651,653,653,652,650,646,642,637,432],[863,903,934,958,976,985,985,974,953,924,893,865,840,818,800,783,769,755,742,730,718,706,693,616,20],[959,969,977,983,988,992,994,994,992,988,981,971,958,943,926,909,891,872,853,834,814,794,755,65,0],[812,836,858,878,898,917,934,949,962,972,980,985,988,990,989,987,983,975,963,946,925,895,133,0,0],[736,761,787,812,838,864,891,916,941,963,980,993,998,1000,1000,1000,1000,1000,1000,999,993,184,1,0,0],[671,695,719,744,771,798,826,856,885,915,942,966,984,995,999,1000,1000,1000,1000,997,165,0,0,0,0],[602,621,641,661,682,703,726,748,771,794,816,837,856,873,886,894,897,895,881,95,0,0,0,0,0],[532,546,560,573,588,602,616,630,643,656,668,679,689,696,702,705,706,621,43,0,0,0,0,0,0]],"newest":[0.212447,0.400488],"population":[[0.193732,0.765545],[0.971102,0.545241],[0.451976,0.0611882],[0.541108,0.210487],[0.0391818,0.427164],[0.674825,0.8923],[0.0,0.508147],[0.0,0.0],[0.0,0.341614],[0.0,0.767732],[0.212447,0.400488]]}}, +{"best":0.0131468,"evaluations":12,"generation":6,"state":{"acquisition":[[0,0,0,0,0,0,97,373,554,679,767,829,872,896,894,0,0,0,0,0,0,0,0,0,0],[38,0,0,0,0,0,0,0,0,0,0,0,246,545,0,0,0,0,0,0,0,0,0,0,0],[813,797,779,758,731,695,643,568,455,283,50,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[943,932,919,905,889,869,846,818,785,748,707,659,0,0,0,0,0,0,0,0,0,0,0,0,0],[967,952,935,915,890,860,822,773,709,625,510,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[959,939,916,886,849,799,731,630,469,180,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[933,913,891,864,834,796,749,688,602,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[831,824,826,834,844,851,851,832,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,422,748,874,925,929,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[829,861,911,959,995,1000,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[296,0,67,749,970,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[202,473,750,872,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,223,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[582,615,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[923,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]],"best":[0.212447,0.400488],"mean":[[0,0,0,0,0,0,0,0,0,0,0,1,2,7,15,28,47,69,93,118,143,166,186,205,221],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,14,36,64,95,126,154,178,198,216],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,5,17,37,62,89,115,139,159,178],[3,3,2,2,1,1,1,0,0,0,0,0,0,0,0,1,3,7,15,27,43,60,78,97,115],[6,5,3,2,1,1,0,0,0,0,0,0,0,0,0,0,0,0,1,4,9,18,30,44,61],[6,4,2,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,7,15,27,41],[3,2,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,1,3,7,14,24,37,51,67],[0,0,0,1,1,1,2,3,5,7,9,12,17,22,29,38,48,60,73,87,102,116,131,145,159],[0,0,0,0,2,13,36,66,100,133,164,190,212,231,246,259,269,277,284,289,294,298,302,304,307],[0,1,4,20,77,212,394,533,606,635,642,637,626,612,596,581,566,551,536,523,510,498,487,476,466],[0,0,0,0,101,516,763,843,862,860,848,831,812,792,772,752,732,713,694,676,658,640,623,607,591],[0,0,0,9,58,159,279,392,486,560,615,656,686,707,722,731,736,738,737,734,727,719,709,696,682],[0,0,0,0,4,27,77,147,224,301,372,437,495,547,593,635,673,708,739,765,787,802,810,810,800],[0,0,0,2,8,25,56,98,148,201,256,311,365,417,470,522,577,636,701,777,865,954,999,1000,1000],[31,38,48,63,83,107,136,169,204,240,278,316,354,392,429,467,506,544,583,623,662,697,727,747,754],[143,154,168,185,204,225,248,272,298,324,351,378,404,430,455,480,503,524,544,562,578,590,600,607,610],[265,279,294,310,327,346,365,385,406,426,446,465,484,501,517,531,544,555,564,571,577,580,582,583,582],[414,435,456,477,498,519,540,559,576,591,604,613,621,626,629,631,631,631,629,626,623,618,613,608,384],[856,898,930,956,974,983,985,979,963,939,911,882,854,829,807,787,769,752,737,722,708,695,681,628,2],[959,969,977,983,988,992,994,994,994,991,986,978,967,953,936,918,899,879,858,838,817,796,771,15,0],[823,846,868,888,907,925,941,955,966,976,982,987,990,991,990,988,984,976,965,950,931,908,48,0,0],[756,781,807,832,858,883,908,932,954,973,987,996,999,1000,1000,1000,1000,1000,1000,1000,996,74,0,0,0],[692,716,741,766,792,819,847,875,903,929,954,974,988,996,999,1000,1000,1000,1000,998,55,0,0,0,0],[618,636,655,675,695,715,736,757,778,798,818,836,853,867,878,884,887,885,875,20,0,0,0,0,0],[540,553,565,578,591,604,616,628,640,651,661,670,678,684,688,691,691,585,5,0,0,0,0,0,0]],"newest":[0.190897,0.387819],"population":[[0.193732,0.765545],[0.971102,0.545241],[0.451976,0.0611882],[0.541108,0.210487],[0.0391818,0.427164],[0.674825,0.8923],[0.0,0.508147],[0.0,0.0],[0.0,0.341614],[0.0,0.767732],[0.212447,0.400488],[0.190897,0.387819]]}}, +{"best":0.00380407,"evaluations":13,"generation":7,"state":{"acquisition":[[0,0,0,0,0,0,0,326,540,688,794,869,920,951,953,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,160,549,0,0,0,0,0,0,0,0,0,0,0],[776,758,737,713,682,639,576,481,330,87,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[943,930,916,899,879,854,823,784,736,678,613,543,0,0,0,0,0,0,0,0,0,0,0,0,0],[978,961,941,918,889,852,805,742,658,543,379,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[968,946,919,886,844,787,708,592,406,72,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[923,902,878,850,816,773,716,640,530,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[741,754,778,801,820,831,833,814,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,130,594,789,883,916,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[520,374,147,0,0,844,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,675,1000,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,112,603,848,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[351,468,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[925,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]],"best":[0.205325,0.398267],"mean":[[0,0,0,0,0,0,0,0,0,0,0,0,2,6,15,30,51,77,105,134,161,187,209,228,245],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,14,36,67,101,134,163,189,211,229],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,11,29,54,82,109,134,157,177],[1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,3,8,17,31,49,68,87,107],[2,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,6,13,24,38,55],[2,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,5,12,23,37],[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,3,9,17,29,43,59],[0,0,0,0,0,0,0,0,1,1,2,2,4,6,10,16,25,35,48,62,78,94,110,127,142],[0,0,0,0,0,1,6,17,37,63,93,123,151,175,197,215,230,243,253,263,270,277,283,288,292],[0,0,0,0,0,1,189,457,581,630,645,645,637,624,609,593,576,560,545,530,516,502,489,477,466],[0,0,0,0,47,862,999,998,993,981,962,939,913,885,857,829,802,775,749,725,700,677,655,633,613],[0,0,0,1,29,153,357,553,693,778,826,850,862,866,864,859,852,842,830,816,801,784,765,744,722],[0,0,0,0,2,20,76,168,274,379,474,554,621,677,723,761,793,819,841,858,871,876,875,865,846],[0,0,0,0,4,17,46,93,152,217,284,350,414,474,532,589,646,704,767,836,911,976,1000,1000,1000],[7,11,17,28,45,68,99,136,177,221,266,311,356,400,443,485,526,568,609,650,690,727,757,776,780],[76,88,102,120,142,167,195,225,258,291,325,358,391,424,454,484,511,536,559,580,597,611,622,628,631],[191,207,225,245,268,292,318,345,373,401,429,455,480,503,524,543,560,574,585,595,601,606,609,609,608],[357,383,410,437,466,495,524,551,576,598,616,631,642,650,656,659,661,661,660,658,655,651,646,640,374],[875,920,951,973,987,993,994,990,978,959,935,909,884,861,840,821,803,787,772,757,743,729,716,682,0],[982,988,991,994,996,998,999,999,999,998,995,991,983,973,960,944,927,908,888,868,848,827,806,1,0],[870,890,908,925,940,954,966,976,984,990,993,996,997,997,997,996,993,988,979,966,949,928,10,0,0],[800,823,846,869,891,912,933,952,969,983,993,998,1000,1000,1000,1000,1000,1000,1000,1000,998,17,0,0,0],[735,758,781,804,828,853,877,902,926,948,968,984,994,998,1000,1000,1000,1000,1000,1000,8,0,0,0,0],[663,682,701,720,740,760,781,801,822,842,862,880,896,909,920,927,930,929,922,1,0,0,0,0,0],[588,601,615,629,642,656,669,682,695,707,718,728,737,743,748,751,752,603,0,0,0,0,0,0,0]],"newest":[0.205325,0.398267],"population":[[0.193732,0.765545],[0.971102,0.545241],[0.451976,0.0611882],[0.541108,0.210487],[0.0391818,0.427164],[0.674825,0.8923],[0.0,0.508147],[0.0,0.0],[0.0,0.341614],[0.0,0.767732],[0.212447,0.400488],[0.190897,0.387819],[0.205325,0.398267]]}}, +{"best":0.00380407,"evaluations":14,"generation":8,"state":{"acquisition":[[0,0,0,0,0,0,0,270,525,699,822,908,967,1000,996,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,98,542,0,0,0,0,0,0,0,0,0,0,0],[755,734,710,682,645,593,518,402,218,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[943,928,911,891,866,836,797,747,683,603,509,400,0,0,0,0,0,0,0,0,0,0,0,0,0],[980,961,937,909,874,829,770,692,584,432,207,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[964,939,907,868,818,751,656,515,289,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[905,881,854,822,782,730,660,562,413,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[661,685,718,751,775,789,791,764,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,361,658,809,873,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[154,0,0,0,0,343,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,299,999,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,438,827,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[7,219,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[891,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]],"best":[0.205325,0.398267],"mean":[[0,0,0,0,0,0,0,0,0,0,0,0,1,5,14,30,53,82,114,147,178,206,231,252,270],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,12,34,67,103,140,173,201,226,246],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,7,21,45,73,102,130,156,178],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,3,10,21,37,56,77,98],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,3,8,17,30,46],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,3,9,19,32],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,7,15,26,41,58],[0,0,0,0,0,0,0,0,0,0,0,0,1,2,5,11,19,31,46,63,82,101,120,139,156],[0,0,0,0,0,0,1,4,16,42,79,124,168,207,240,266,286,302,314,323,330,335,339,342,345],[0,0,0,0,0,0,191,716,853,882,881,868,847,823,796,769,741,714,688,663,639,616,595,575,557],[0,0,0,0,15,940,1000,1000,1000,1000,1000,999,995,986,973,956,935,911,885,857,829,800,771,743,716],[0,0,0,0,21,171,455,709,856,924,953,965,970,970,967,962,955,946,935,922,905,887,865,841,815],[0,0,0,0,1,13,67,171,300,429,544,639,715,774,821,857,885,907,924,936,943,944,940,929,910],[0,0,0,0,1,7,26,65,121,189,263,338,411,481,547,611,674,738,803,872,939,988,1000,1000,1000],[1,2,4,8,17,32,55,86,124,167,214,262,310,358,405,452,497,541,585,629,672,712,745,766,773],[33,40,51,64,82,104,130,160,193,228,265,302,339,375,410,443,474,503,530,553,573,590,603,612,617],[123,137,154,174,197,223,251,281,313,345,378,410,440,468,494,517,538,555,570,582,591,597,602,604,604],[286,313,343,375,409,444,479,514,546,574,599,619,634,646,654,660,663,665,665,663,661,657,653,647,369],[874,924,958,980,991,996,997,995,988,974,955,932,908,885,863,843,825,808,791,776,761,747,733,708,0],[993,995,997,998,999,1000,1000,1000,1000,1000,999,997,993,986,976,962,946,929,909,889,869,848,827,0,0],[912,928,942,954,965,974,982,988,993,996,998,999,999,999,999,998,997,993,987,976,961,942,3,0,0],[847,867,887,906,924,941,957,971,983,991,997,999,1000,1000,1000,1000,1000,1000,1000,1000,999,5,0,0,0],[784,804,825,846,867,888,909,929,948,965,980,990,997,999,1000,1000,1000,1000,1000,1000,2,0,0,0,0],[712,729,747,765,783,801,820,839,857,875,892,907,921,933,942,948,950,949,944,0,0,0,0,0,0],[635,648,661,674,687,700,712,725,736,748,758,767,775,781,786,788,788,607,0,0,0,0,0,0,0]],"newest":[0.191724,0.406987],"population":[[0.193732,0.765545],[0.971102,0.545241],[0.451976,0.0611882],[0.541108,0.210487],[0.0391818,0.427164],[0.674825,0.8923],[0.0,0.508147],[0.0,0.0],[0.0,0.341614],[0.0,0.767732],[0.212447,0.400488],[0.190897,0.387819],[0.205325,0.398267],[0.191724,0.406987]]}}, +{"best":0.000389932,"evaluations":15,"generation":9,"state":{"acquisition":[[0,0,0,0,0,0,0,146,449,654,795,894,961,1000,1000,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,485,0,0,0,0,0,0,0,0,0,0,0],[734,710,683,649,605,544,456,325,119,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[928,911,892,869,841,807,763,707,637,548,443,321,0,0,0,0,0,0,0,0,0,0,0,0,0],[956,934,907,874,833,781,713,623,499,327,75,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[925,894,856,808,745,662,543,368,86,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[842,812,778,737,686,620,532,413,237,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[530,562,604,646,680,703,712,694,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,125,534,739,831,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,59,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,148,683,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,26,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[867,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]],"best":[0.193398,0.40678],"mean":[[0,0,0,0,0,0,0,0,0,0,0,0,1,3,10,23,43,70,101,134,167,197,225,250,272],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,7,25,54,91,129,165,198,226,250],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,13,33,60,90,120,149,174],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,5,14,27,44,64,85],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,4,10,20,33],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,4,10,19],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,3,7,14,25,38],[0,0,0,0,0,0,0,0,0,0,0,0,0,1,2,4,8,15,25,39,54,71,90,108,126],[0,0,0,0,0,0,0,1,6,19,42,74,111,148,184,214,240,262,279,293,304,312,319,325,330],[0,0,0,0,0,0,76,744,928,955,953,940,920,894,865,833,801,769,738,709,681,655,630,607,586],[0,0,0,0,0,1000,1000,1000,1000,1000,1000,1000,1000,998,995,987,976,960,939,916,889,861,831,801,772],[0,0,0,0,5,136,524,821,936,972,984,988,989,989,988,985,981,976,969,959,947,932,914,893,869],[0,0,0,0,0,3,36,135,286,446,585,693,773,832,874,906,928,945,957,964,969,970,967,959,945],[0,0,0,0,0,2,10,35,82,148,226,309,393,474,552,626,698,769,838,906,964,996,1000,1000,1000],[0,1,2,4,9,18,34,58,90,129,173,221,271,322,372,422,472,520,569,616,662,704,739,763,773],[21,27,35,46,61,79,102,129,160,193,229,266,304,342,379,414,448,479,508,534,557,576,591,602,609],[95,109,125,145,167,192,220,250,283,316,350,384,416,447,475,501,523,543,559,573,584,592,598,601,603],[250,279,311,345,382,420,459,497,533,565,594,617,635,649,659,666,670,672,673,672,669,666,661,656,373],[874,928,963,984,994,998,998,998,995,987,974,957,936,915,893,872,853,834,816,799,783,768,753,730,0],[996,998,999,999,1000,1000,1000,1000,1000,1000,1000,999,997,994,987,978,965,949,931,912,892,871,850,0,0],[937,950,961,970,978,985,990,994,997,998,999,1000,1000,1000,1000,999,999,996,992,984,973,957,2,0,0],[881,899,916,932,947,961,973,983,991,996,999,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,3,0,0,0],[819,839,858,877,896,914,932,949,965,978,988,995,998,1000,1000,1000,1000,1000,1000,1000,1,0,0,0,0],[745,761,778,795,812,829,846,862,879,894,909,922,934,944,951,956,958,957,953,0,0,0,0,0,0],[664,676,688,700,712,724,735,746,756,766,775,783,790,795,799,801,801,616,0,0,0,0,0,0,0]],"newest":[0.193398,0.40678],"population":[[0.193732,0.765545],[0.971102,0.545241],[0.451976,0.0611882],[0.541108,0.210487],[0.0391818,0.427164],[0.674825,0.8923],[0.0,0.508147],[0.0,0.0],[0.0,0.341614],[0.0,0.767732],[0.212447,0.400488],[0.190897,0.387819],[0.205325,0.398267],[0.191724,0.406987],[0.193398,0.40678]]}}, +{"best":0.000389932,"evaluations":16,"generation":10,"state":{"acquisition":[[0,0,0,0,0,0,0,271,520,693,817,906,966,1000,992,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,50,535,0,0,0,0,0,0,0,0,0,0,0],[687,668,647,622,589,541,466,348,150,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[905,891,875,856,832,801,761,707,634,539,420,278,0,0,0,0,0,0,0,0,0,0,0,0,0],[955,937,915,889,856,813,755,676,563,398,137,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[944,921,894,859,814,753,666,536,325,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[877,858,835,808,772,722,651,548,380,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[595,630,673,710,734,743,733,688,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,241,563,720,778,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[87,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,248,719,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,29,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[825,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]],"best":[0.193398,0.40678],"mean":[[0,0,0,0,0,0,0,0,0,0,0,0,1,6,16,35,62,94,129,162,193,219,243,262,278],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,15,40,74,111,146,177,204,226,244],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,8,24,48,76,104,130,154,175],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,4,11,23,39,58,79,99],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,4,10,20,33,49],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,4,10,20,32],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,4,10,19,31,45],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,2,4,9,16,26,38,53,68,84,101],[0,0,0,0,0,0,0,0,2,5,12,22,37,56,76,96,116,134,151,166,180,192,203,212,221],[0,0,0,0,0,0,14,391,675,739,731,701,664,628,593,562,533,508,486,467,449,434,420,408,397],[0,0,0,0,0,1000,1000,1000,1000,1000,1000,997,987,968,941,906,867,827,786,746,707,671,637,606,577],[0,0,0,0,11,213,636,875,949,971,977,977,974,968,960,950,936,920,901,878,853,824,792,758,722],[0,0,0,0,0,15,94,245,416,563,673,751,805,842,869,889,904,916,925,930,933,930,920,901,870],[0,0,0,0,1,8,34,89,167,259,352,440,518,588,650,706,757,807,856,907,955,991,1000,1000,1000],[0,1,2,5,13,29,56,96,145,202,262,322,381,437,490,539,585,629,671,711,749,782,807,819,813],[19,24,34,47,66,91,122,159,201,245,292,338,383,425,465,502,535,565,591,613,630,643,652,655,654],[92,106,125,147,173,204,238,276,316,357,397,436,471,504,533,558,579,596,610,620,628,632,634,633,629],[253,283,317,354,393,434,477,518,556,590,617,639,656,668,677,682,686,687,687,685,681,677,671,664,375],[880,931,964,984,994,998,998,995,987,971,950,927,904,883,863,844,828,812,797,782,767,753,739,718,0],[995,997,998,999,999,1000,1000,1000,1000,1000,999,997,992,984,974,960,945,927,908,888,867,846,825,0,0],[922,936,949,960,969,977,985,990,994,997,998,999,999,999,999,999,997,994,987,975,959,938,1,0,0],[855,874,892,910,926,942,957,971,982,991,997,999,1000,1000,1000,1000,1000,1000,1000,1000,998,2,0,0,0],[790,810,830,850,870,890,910,929,948,965,979,990,997,999,1000,1000,1000,1000,1000,1000,0,0,0,0,0],[720,738,756,774,792,810,829,848,866,884,902,918,932,944,954,960,962,961,956,0,0,0,0,0,0],[647,661,675,688,702,716,729,742,755,767,778,788,797,804,810,812,812,629,0,0,0,0,0,0,0]],"newest":[0.560359,0.0],"population":[[0.193732,0.765545],[0.971102,0.545241],[0.451976,0.0611882],[0.541108,0.210487],[0.0391818,0.427164],[0.674825,0.8923],[0.0,0.508147],[0.0,0.0],[0.0,0.341614],[0.0,0.767732],[0.212447,0.400488],[0.190897,0.387819],[0.205325,0.398267],[0.191724,0.406987],[0.193398,0.40678],[0.560359,0.0]]}}, +{"best":0.000389932,"evaluations":17,"generation":11,"state":{"acquisition":[[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[448,386,322,249,155,18,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[853,820,783,738,683,611,513,380,198,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[976,949,918,880,832,769,683,563,388,123,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[1000,975,945,908,860,794,700,558,328,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[944,928,910,887,855,809,741,637,463,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[627,681,742,792,824,836,827,778,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,265,623,800,870,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[41,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,218,791,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[895,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]],"best":[0.193398,0.40678],"mean":[[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,28,82,143,197,241,275,300],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,11,38,79,124,166,201,230,254],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,6,20,42,69,99,127,153,177],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,3,10,21,37,57,78,100],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,3,10,20,34,51],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,4,11,21,35],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,4,10,19,32,47],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,2,6,13,22,35,49,66,83,100],[0,0,0,0,0,0,0,0,1,3,8,16,29,46,64,83,102,120,137,152,166,179,190,201,211],[0,0,0,0,0,0,12,389,669,725,710,674,634,595,559,527,498,474,452,433,417,403,390,380,371],[0,0,0,0,0,1000,1000,1000,1000,1000,999,995,982,958,924,884,840,796,752,710,670,633,599,568,539],[0,0,0,0,8,218,683,906,964,978,981,979,974,967,956,944,928,909,887,861,832,800,764,727,689],[0,0,0,0,0,17,108,281,466,614,718,786,831,862,883,898,909,918,924,929,930,926,915,892,855],[0,0,0,0,1,8,40,105,198,301,402,494,572,639,696,746,791,833,875,918,960,992,1000,1000,1000],[0,0,1,3,10,26,57,102,161,227,297,365,430,489,544,594,639,682,722,760,796,827,850,858,848],[10,15,23,35,54,81,117,160,209,262,316,370,421,468,511,550,584,614,639,661,677,688,694,695,690],[70,84,103,127,156,191,232,277,325,373,421,465,505,541,571,597,619,636,649,659,665,668,668,665,659],[229,262,299,341,387,435,485,534,578,615,645,668,685,697,706,711,714,716,715,713,709,703,696,688,443],[896,944,973,989,997,999,999,996,988,973,953,932,911,891,873,857,841,827,813,799,785,771,756,739,0],[998,999,999,999,1000,1000,1000,1000,1000,1000,999,997,993,986,976,964,950,934,916,897,877,856,834,0,0],[935,948,958,967,975,982,988,993,996,998,999,999,1000,1000,1000,999,998,995,989,978,962,941,1,0,0],[870,888,904,920,934,949,962,974,984,992,997,999,1000,1000,1000,1000,1000,1000,1000,1000,998,2,0,0,0],[808,826,845,863,881,900,918,936,953,968,982,991,997,1000,1000,1000,1000,1000,1000,1000,0,0,0,0,0],[741,758,776,793,810,828,846,864,882,899,916,932,946,958,966,972,975,974,969,0,0,0,0,0,0],[672,686,700,714,728,742,755,769,782,795,807,818,827,835,841,844,844,649,0,0,0,0,0,0,0]],"newest":[0.0,0.201622],"population":[[0.193732,0.765545],[0.971102,0.545241],[0.451976,0.0611882],[0.541108,0.210487],[0.0391818,0.427164],[0.674825,0.8923],[0.0,0.508147],[0.0,0.0],[0.0,0.341614],[0.0,0.767732],[0.212447,0.400488],[0.190897,0.387819],[0.205325,0.398267],[0.191724,0.406987],[0.193398,0.40678],[0.560359,0.0],[0.0,0.201622]]}}, +{"best":0.000389932,"evaluations":18,"generation":12,"state":{"acquisition":[[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,41,308,487,603,671,686,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,129,567,789,891,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[97,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,269,856,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,95,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[1000,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]],"best":[0.193398,0.40678],"mean":[[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,27,85,153,214,264,302,330],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,16,55,110,167,217,258,291,317],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,11,34,69,111,152,189,222,249],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,6,18,39,66,97,128,158],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,6,18,36,60,87],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,7,19,37,60],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,7,17,34,54,78],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,5,12,23,40,60,83,106,130,152],[0,0,0,0,0,0,0,0,1,3,8,20,39,64,94,124,153,179,203,223,241,256,269,281,290],[0,0,0,0,0,0,6,372,704,785,788,765,735,702,670,640,613,587,564,544,525,508,492,478,466],[0,0,0,0,0,1000,1000,1000,1000,1000,1000,999,994,982,963,937,906,871,835,798,761,726,692,660,630],[0,0,0,0,6,198,664,905,968,983,987,987,985,981,975,967,957,944,928,908,885,858,828,794,759],[0,0,0,0,0,13,92,256,443,600,712,789,839,873,897,914,927,937,944,949,950,948,939,921,891],[0,0,0,0,0,6,32,88,172,270,370,462,544,615,677,732,782,828,875,921,964,993,1000,1000,1000],[0,0,1,3,10,24,51,90,142,201,265,329,390,448,502,552,598,641,682,721,757,789,813,823,815],[13,17,25,38,55,80,112,150,193,240,289,338,385,429,470,507,541,570,596,617,634,646,653,656,653],[76,90,108,130,157,189,225,266,308,352,395,436,473,507,537,562,584,601,614,625,631,635,636,635,631],[232,264,299,338,381,426,472,517,558,594,623,646,663,676,684,690,693,693,693,690,686,681,675,667,434],[886,938,969,987,996,998,999,996,989,976,956,934,912,890,870,852,834,818,803,788,773,758,743,728,0],[997,998,999,999,1000,1000,1000,1000,1000,1000,999,998,994,987,978,965,949,932,913,893,872,850,829,0,0],[935,948,958,968,976,983,988,993,996,998,999,999,1000,1000,1000,999,998,995,988,977,961,941,0,0,0],[872,890,906,922,937,951,964,976,985,993,997,999,1000,1000,1000,1000,1000,1000,1000,1000,998,1,0,0,0],[811,829,848,866,884,903,921,938,955,970,983,992,997,1000,1000,1000,1000,1000,1000,1000,0,0,0,0,0],[744,761,778,795,812,829,846,864,881,898,914,929,943,954,962,968,970,970,965,0,0,0,0,0,0],[674,687,700,713,727,740,753,765,778,789,800,810,819,826,831,834,834,651,0,0,0,0,0,0,0]],"newest":[0.172931,0.425271],"population":[[0.193732,0.765545],[0.971102,0.545241],[0.451976,0.0611882],[0.541108,0.210487],[0.0391818,0.427164],[0.674825,0.8923],[0.0,0.508147],[0.0,0.0],[0.0,0.341614],[0.0,0.767732],[0.212447,0.400488],[0.190897,0.387819],[0.205325,0.398267],[0.191724,0.406987],[0.193398,0.40678],[0.560359,0.0],[0.0,0.201622],[0.172931,0.425271]]}}, +{"best":8.57343e-05,"evaluations":19,"generation":13,"state":{"acquisition":[[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,180,403,545,629,653,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,479,749,880,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[1000,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]],"best":[0.194758,0.405116],"mean":[[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,23,75,139,198,247,286,314],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,12,46,96,150,199,240,272,298],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,8,27,59,97,136,173,205,233],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,4,14,33,58,87,117,146],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,5,15,32,54,80],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,6,17,33,55],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,5,14,29,48,71],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,3,8,17,31,49,71,93,116,139],[0,0,0,0,0,0,0,0,0,1,4,12,27,48,74,102,130,156,179,200,218,234,248,260,270],[0,0,0,0,0,0,3,355,707,789,789,764,730,694,659,627,597,570,546,524,504,487,471,457,444],[0,0,0,0,0,1000,1000,1000,1000,1000,1000,999,995,985,967,941,909,872,833,793,754,716,679,645,613],[0,0,0,0,0,94,752,956,986,992,993,992,989,986,980,972,962,949,933,913,888,859,826,790,751],[0,0,0,0,0,5,73,256,472,642,755,825,869,898,916,930,940,947,953,956,957,954,944,925,892],[0,0,0,0,0,4,27,84,175,283,393,492,578,651,712,764,810,853,894,935,971,995,1000,1000,1000],[0,0,0,2,6,19,44,85,139,204,274,344,411,473,530,582,629,672,713,751,786,816,838,846,835],[6,10,16,26,43,66,99,139,186,238,292,346,397,445,489,529,564,594,620,641,657,669,675,675,670],[55,68,85,108,136,170,209,254,301,350,398,444,485,522,553,580,602,620,633,643,649,652,652,650,644],[203,236,274,317,364,415,467,518,564,604,635,660,678,691,699,705,708,708,707,704,700,694,687,679,441],[894,945,975,991,997,999,999,998,992,979,962,941,919,899,880,861,845,828,813,798,783,768,752,736,0],[999,999,1000,1000,1000,1000,1000,1000,1000,1000,1000,998,995,990,981,969,955,938,919,899,878,857,835,0,0],[949,960,968,976,982,987,992,995,997,999,999,1000,1000,1000,1000,999,998,996,990,980,964,943,0,0,0],[891,906,921,934,947,959,971,980,989,995,998,1000,1000,1000,1000,1000,1000,1000,1000,1000,998,1,0,0,0],[831,849,865,882,899,915,931,947,962,975,986,994,998,1000,1000,1000,1000,1000,1000,1000,0,0,0,0,0],[766,782,799,815,831,847,863,880,896,912,927,941,953,963,971,976,978,978,974,0,0,0,0,0,0],[698,711,724,737,750,763,775,788,800,811,822,832,840,847,852,855,855,671,0,0,0,0,0,0,0]],"newest":[0.194758,0.405116],"population":[[0.193732,0.765545],[0.971102,0.545241],[0.451976,0.0611882],[0.541108,0.210487],[0.0391818,0.427164],[0.674825,0.8923],[0.0,0.508147],[0.0,0.0],[0.0,0.341614],[0.0,0.767732],[0.212447,0.400488],[0.190897,0.387819],[0.205325,0.398267],[0.191724,0.406987],[0.193398,0.40678],[0.560359,0.0],[0.0,0.201622],[0.172931,0.425271],[0.194758,0.405116]]}}, +{"best":1.50745e-05,"evaluations":20,"generation":14,"state":{"acquisition":[[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,55,319,484,581,612,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,385,697,857,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[1000,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]],"best":[0.194735,0.405068],"mean":[[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,19,66,126,183,231,269,298],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,9,38,84,134,181,221,253,279],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,6,22,50,85,122,157,188,215],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,11,27,50,77,105,133],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,4,13,28,49,73],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,5,15,30,50],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,4,12,25,43,64],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,5,12,24,41,60,82,104,126],[0,0,0,0,0,0,0,0,0,0,2,8,19,37,59,85,111,135,158,178,197,213,227,239,250],[0,0,0,0,0,0,1,346,714,792,788,758,721,682,644,609,577,549,523,500,480,462,447,433,420],[0,0,0,0,0,1000,1000,1000,1000,1000,1000,999,996,987,969,942,908,869,827,784,742,701,662,626,592],[0,0,0,0,0,72,784,972,992,995,996,995,993,989,984,977,967,954,936,915,888,856,820,780,738],[0,0,0,0,0,3,64,257,492,671,783,850,890,915,931,941,949,955,959,962,962,958,948,926,890],[0,0,0,0,0,3,23,80,176,293,410,516,605,678,738,788,832,871,909,945,977,996,1000,1000,1000],[0,0,0,1,4,14,38,78,136,205,281,357,429,495,554,608,656,699,739,776,811,840,860,867,854],[3,6,10,18,33,55,87,129,179,236,294,353,409,461,508,550,586,617,643,664,680,691,696,695,688],[41,52,68,89,117,153,195,243,295,349,402,452,497,536,569,597,620,638,652,661,667,669,669,665,659],[178,211,251,297,348,404,462,518,569,613,647,673,691,705,714,719,722,723,721,718,713,707,699,691,452],[899,951,979,993,998,1000,1000,998,993,982,966,946,926,906,888,870,854,838,822,807,792,777,761,745,0],[999,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,999,996,992,984,973,959,943,924,905,884,862,840,0,0],[960,969,976,982,987,991,994,997,998,999,1000,1000,1000,1000,1000,1000,999,997,991,981,966,946,0,0,0],[906,920,933,945,956,966,976,984,991,996,999,1000,1000,1000,1000,1000,1000,1000,1000,1000,999,0,0,0,0],[849,865,881,896,911,926,940,954,967,979,988,995,999,1000,1000,1000,1000,1000,1000,1000,0,0,0,0,0],[787,802,817,832,848,863,878,893,908,923,937,950,961,970,977,981,984,983,980,0,0,0,0,0,0],[720,733,746,758,771,783,795,807,819,830,841,850,858,865,870,873,873,690,0,0,0,0,0,0,0]],"newest":[0.194735,0.405068],"population":[[0.193732,0.765545],[0.971102,0.545241],[0.451976,0.0611882],[0.541108,0.210487],[0.0391818,0.427164],[0.674825,0.8923],[0.0,0.508147],[0.0,0.0],[0.0,0.341614],[0.0,0.767732],[0.212447,0.400488],[0.190897,0.387819],[0.205325,0.398267],[0.191724,0.406987],[0.193398,0.40678],[0.560359,0.0],[0.0,0.201622],[0.172931,0.425271],[0.194758,0.405116],[0.194735,0.405068]]}}, +{"best":8.3593e-06,"evaluations":21,"generation":15,"state":{"acquisition":[[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,160,359,477,523,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,165,547,762,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[1000,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]],"best":[0.194733,0.405064],"mean":[[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,16,59,116,171,217,255,284],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,8,33,74,121,166,205,237,263],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,18,43,75,110,144,174,201],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,9,24,44,70,97,124],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,4,11,26,45,68],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,5,13,28,47],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,3,10,23,40,60],[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,4,10,20,35,54,75,96,118],[0,0,0,0,0,0,0,0,0,0,1,4,13,28,49,72,97,122,144,165,183,199,213,226,237],[0,0,0,0,0,0,0,299,702,790,789,761,723,683,644,607,573,543,515,491,470,451,434,419,406],[0,0,0,0,0,1000,1000,1000,1000,1000,1000,1000,998,992,978,953,920,880,836,790,744,699,657,617,581],[0,0,0,0,0,74,853,990,998,999,999,998,997,995,992,986,978,966,949,927,899,865,825,780,734],[0,0,0,0,0,3,71,294,554,735,839,895,926,944,955,962,966,970,972,973,972,967,955,932,891],[0,0,0,0,0,3,24,87,196,328,456,567,657,727,783,828,866,899,930,959,983,998,1000,1000,1000],[0,0,0,0,3,12,36,80,143,221,305,387,464,533,593,646,692,733,770,803,834,859,876,880,863],[2,3,7,14,27,49,82,127,182,244,308,372,432,486,534,576,612,642,666,685,698,706,709,705,695],[31,41,56,77,105,142,187,240,297,356,413,466,513,554,587,615,637,654,666,674,678,679,677,671,663],[158,193,234,283,338,398,462,523,578,624,659,686,704,717,725,730,732,732,730,726,720,713,704,694,450],[905,956,982,995,999,1000,1000,999,994,984,969,950,930,911,893,875,859,842,827,811,795,779,763,743,0],[1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,1000,999,997,993,985,975,961,945,927,907,885,863,840,0,0],[968,975,981,986,990,993,996,998,999,999,1000,1000,1000,1000,1000,1000,999,997,992,982,967,946,2,0,0],[918,931,942,953,962,971,980,987,993,997,999,1000,1000,1000,1000,1000,1000,1000,1000,1000,998,4,0,0,0],[865,879,894,907,921,934,947,960,971,982,990,996,999,1000,1000,1000,1000,1000,1000,1000,1,0,0,0,0],[805,819,834,848,862,876,890,904,918,932,945,957,967,975,982,986,987,987,984,0,0,0,0,0,0],[740,753,765,777,789,801,813,824,836,846,857,866,874,881,885,888,888,689,0,0,0,0,0,0,0]],"newest":[0.194733,0.405064],"population":[[0.193732,0.765545],[0.971102,0.545241],[0.451976,0.0611882],[0.541108,0.210487],[0.0391818,0.427164],[0.674825,0.8923],[0.0,0.508147],[0.0,0.0],[0.0,0.341614],[0.0,0.767732],[0.212447,0.400488],[0.190897,0.387819],[0.205325,0.398267],[0.191724,0.406987],[0.193398,0.40678],[0.560359,0.0],[0.0,0.201622],[0.172931,0.425271],[0.194758,0.405116],[0.194735,0.405068],[0.194733,0.405064]]}} +]} diff --git a/examples/bo_constrained/trace.py b/examples/bo_constrained/trace.py new file mode 100644 index 00000000..d139a693 --- /dev/null +++ b/examples/bo_constrained/trace.py @@ -0,0 +1,136 @@ +"""The trace of the run for the plot on the example's page, written to the file that +``GENOXIDE_TRACE`` names: a frame per step, with the points evaluated so far and, from the first +point the models chose, the models that chose it on a grid of 25 x 25 points: the probability that +a point is feasible under the constraints' models, and the acquisition, the log expected +improvement plus the logarithm of that probability (before a feasible point, the logarithm alone), +the 25 nats below its highest value shaded. Both are rounded to thousandths of their range, which +is all the page draws. The Rust example writes the same file.""" + +import json +import math +import os + +import numpy as np + +# the points per side of the grid, and the span of the acquisition that is shaded, in nats +GRID = 25 +SPAN = 25.0 + + +class Trace: + """Records the run through ``control`` when ``GENOXIDE_TRACE`` is set.""" + + def __init__(self, minimizer, minimum, constraints): + self.path = os.environ.get("GENOXIDE_TRACE") + self.minimizer = minimizer + self.minimum = minimum + self.constraints = constraints + self.frames = [] + + def record(self, algorithm, progress): + """Records a step: the points so far, the newest one, the best, and the models that + chose the newest on the grid.""" + if not self.path: + return + points = progress.population.tolist() + best = progress.best_genome + state = { + "population": points, + "newest": points[-1], + "best": best.tolist(), + } + if algorithm.model is not None: + grid = points_of_grid() + feasible = algorithm.probability_of_feasibility_at(grid).reshape(GRID, GRID) + acquisition = algorithm.acquisition_at(grid).reshape(GRID, GRID) + state["mean"] = shaded(feasible, lambda lo, hi, v: v) + state["acquisition"] = shaded(acquisition, acquisition_shade) + # the best feasible value's distance above the minimum, none before a feasible point + error = None + if np.all(self.constraints(best) <= 0.0): + error = float(best[0] + best[1]) - self.minimum + self.frames.append( + { + "generation": progress.generation, + "evaluations": progress.evaluations, + "best": error, + "state": state, + } + ) + + def write(self): + """Writes the trace, if there's one.""" + if not self.path: + return + settings = { + "format": 1, + "example": "bo_constrained", + "objective": "minimize", + "x_label": "evaluations", + "y_label": "error of the best feasible point", + "log_y": True, + "optimum": 0.0, + "plot": "surrogate", + "problem": { + "bounds": [[0.0, 1.0], [0.0, 1.0]], + "minima": [self.minimizer], + "minima_label": "the global minimum", + "grid": GRID, + "panels": [ + { + "title": "The probability of feasibility", + "shading": "shading: more likely feasible", + }, + { + "title": "Log-EI + log P(feasible)", + "shading": "shading: more worth evaluating", + }, + ], + }, + } + write(self.path, settings, self.frames) + + +def points_of_grid(): + """The grid over the box, a point per row: x2 from 0 to 1 by row of the grid, x1 by column, + as the Rust example orders them.""" + return np.array( + [[column / (GRID - 1), row / (GRID - 1)] for row in range(GRID) for column in range(GRID)] + ) + + +def acquisition_shade(lo, hi, v): + lo = max(lo, hi - SPAN) + return max((v - lo) / (hi - lo), 0.0) + + +def shaded(grid, shade): + """The grid's values as thousandths: ``shade(lo, hi, v)`` in [0, 1], rounded.""" + lo, hi = float(grid.min()), float(grid.max()) + return [[math.floor(1000.0 * shade(lo, hi, float(v)) + 0.5) for v in row] for row in grid] + + +# ---- the same in every example's trace ---------------------------------------------------------- + + +def write(path, settings, frames): + """Writes the settings and the frames to ``path``, a frame per line.""" + lines = ",\n".join(map(to_json, frames)) + with open(path, "w", encoding="utf-8", newline="\n") as file: + file.write(f'{to_json(settings)[:-1]},"frames":[\n{lines}\n]}}\n') + + +def to_json(value): + """Compact JSON with sorted keys, and numbers rounded to 6 significant digits, as the Rust + example writes it.""" + return json.dumps(rounded(value), sort_keys=True, separators=(",", ":"), ensure_ascii=False) + + +def rounded(value): + if isinstance(value, dict): + return {key: rounded(item) for key, item in value.items()} + if isinstance(value, (list, tuple)): + return [rounded(item) for item in value] + if isinstance(value, float): + return float(f"{value:.5e}") if math.isfinite(value) else None + return value diff --git a/examples/bo_constrained/trace.rs b/examples/bo_constrained/trace.rs new file mode 100644 index 00000000..d7e27cba --- /dev/null +++ b/examples/bo_constrained/trace.rs @@ -0,0 +1,168 @@ +//! The trace of the run for the plot on the example's page, written to the file that +//! `GENOXIDE_TRACE` names: a frame per step, with the points evaluated so far and, from the first +//! point the models chose, the models that chose it on a grid of 25 × 25 points: the probability +//! that a point is feasible under the constraints' models, and the acquisition, the log expected +//! improvement plus the logarithm of that probability (before a feasible point, the logarithm +//! alone), the 25 nats below its highest value shaded. Both are rounded to thousandths of their +//! range, which is all the page draws. The Python example writes the same file. + +use genoxide::engine::Progress; +use genoxide::prelude::*; +use serde_json::{Value, json}; + +// the points per side of the grid, and the span of the acquisition that is shaded, in nats +const GRID: usize = 25; +const SPAN: f64 = 25.0; + +pub struct Trace { + path: Option, + frames: Vec, +} + +impl Trace { + // a trace for the file that GENOXIDE_TRACE names, or nothing to record if it isn't set + pub fn from_env() -> Self { + Self { + path: std::env::var("GENOXIDE_TRACE").ok(), + frames: Vec::new(), + } + } + + // records a step: the points so far, the newest one, the best, and the models that chose the + // newest on the grid + pub fn record(&mut self, bo: &Bo, progress: &Progress) { + if self.path.is_none() { + return; + } + let points: Vec<[f64; 2]> = bo + .population() + .iter() + .map(|individual| [individual.genome()[0], individual.genome()[1]]) + .collect(); + let best = bo.best().expect("evaluated"); + let mut state = json!({ + "population": points, + "newest": points[points.len() - 1], + "best": [best.genome()[0], best.genome()[1]], + }); + if bo.model().is_some() { + let feasible = grid(|x| bo.probability_of_feasibility_at(x).expect("a model")); + let acquisition = grid(|x| bo.acquisition_at(x).expect("a model")); + state["mean"] = json!(shaded(&feasible, |_, _, v| v)); + state["acquisition"] = json!(shaded(&acquisition, |lo, hi, v| { + let lo = lo.max(hi - SPAN); + ((v - lo) / (hi - lo)).max(0.0) + })); + } + // the best feasible value's distance above the minimum, none before a feasible point + let fitness = best.fitness().expect("evaluated"); + let error = fitness + .is_feasible() + .then(|| fitness.score().expect("valid") - super::MINIMUM); + self.frames.push(json!({ + "generation": progress.generation(), + "evaluations": progress.evaluations(), + "best": error, + "state": state, + })); + } + + // writes the trace, if there's one + pub fn write(self) { + let Some(path) = self.path else { return }; + let settings = json!({ + "format": 1, + "example": "bo_constrained", + "objective": "minimize", + "x_label": "evaluations", + "y_label": "error of the best feasible point", + "log_y": true, + "optimum": 0.0, + "plot": "surrogate", + "problem": { + "bounds": [[0.0, 1.0], [0.0, 1.0]], + "minima": [super::MINIMIZER], + "minima_label": "the global minimum", + "grid": GRID, + "panels": [ + { + "title": "The probability of feasibility", + "shading": "shading: more likely feasible", + }, + { + "title": "Log-EI + log P(feasible)", + "shading": "shading: more worth evaluating", + }, + ], + }, + }); + write(&path, settings, self.frames); + } +} + +// `f` on the grid over the box: a row per value of x₂, from 0 to 1, and a column per value of x₁ +fn grid(f: impl Fn(&[f64]) -> f64) -> Vec> { + let at = |i: usize| i as f64 / (GRID - 1) as f64; + (0..GRID) + .map(|row| (0..GRID).map(|column| f(&[at(column), at(row)])).collect()) + .collect() +} + +// the grid's values as thousandths: `shade(lo, hi, v)` in [0, 1], rounded +fn shaded(grid: &[Vec], shade: impl Fn(f64, f64, f64) -> f64) -> Vec> { + let values = grid.iter().flatten(); + let lo = values.clone().fold(f64::INFINITY, |a, &b| a.min(b)); + let hi = values.fold(f64::NEG_INFINITY, |a, &b| a.max(b)); + grid.iter() + .map(|row| { + row.iter() + .map(|&v| (1000.0 * shade(lo, hi, v) + 0.5).floor() as u64) + .collect() + }) + .collect() +} + +// ---- the same in every example's trace --------------------------------------------------------- + +// writes the settings and the frames to `path`, a frame per line +fn write(path: &str, settings: Value, frames: Vec) { + let frames: Vec = frames.iter().map(to_json).collect(); + let settings = to_json(&settings); + let head = &settings[..settings.len() - 1]; + let text = format!("{head},\"frames\":[\n{}\n]}}\n", frames.join(",\n")); + std::fs::write(path, text).expect("the trace is written"); +} + +// compact JSON with sorted keys, and numbers rounded to 6 significant digits and written as +// Python writes them (7542.0, 1e-08): the Python example writes the same file +fn to_json(value: &Value) -> String { + let join = |items: Vec| items.join(","); + match value { + Value::Number(number) if number.is_f64() => python_float(number.as_f64().expect("f64")), + Value::Array(items) => format!("[{}]", join(items.iter().map(to_json).collect())), + Value::Object(map) => { + let entry = + |(key, item): (&String, &Value)| format!("{}:{}", json!(key), to_json(item)); + format!("{{{}}}", join(map.iter().map(entry).collect())) + } + other => other.to_string(), + } +} + +fn python_float(value: f64) -> String { + let rounded: f64 = format!("{value:.5e}").parse().expect("a number"); + let shortest = format!("{rounded:e}"); + let (mantissa, exponent) = shortest.split_once('e').expect("an exponent"); + let exponent: i32 = exponent.parse().expect("an exponent"); + if (-4..16).contains(&exponent) { + let text = rounded.to_string(); + if text.contains('.') { + text + } else { + text + ".0" + } + } else { + let sign = if exponent < 0 { '-' } else { '+' }; + format!("{mantissa}e{sign}{:02}", exponent.abs()) + } +} diff --git a/examples/bo_hartmann6/README.md b/examples/bo_hartmann6/README.md new file mode 100644 index 00000000..310721ee --- /dev/null +++ b/examples/bo_hartmann6/README.md @@ -0,0 +1,85 @@ +--- +title: Bayesian optimization in batches on Hartmann 6-D +category: bayesian +summary: Reach the global minimum of Hartmann's 6-D function with 4 points a round, chosen by the Kriging believer and evaluated in parallel, in 15 rounds where one point a round takes 36. +reference: "Ginsbourger, D., Le Riche, R. and Carraro, L. (2010). Kriging is well-suited to parallelize optimization. In Computational Intelligence in Expensive Optimization Problems, Springer: 131-162." +reference_url: "https://doi.org/10.1007/978-3-642-10701-6_6" +optimum: "−3.32237 at (0.20169, 0.15001, 0.47687, 0.27533, 0.31165, 0.65730) (best known)" +languages: [rust, python] +order: 291 +--- + +# Bayesian optimization in batches on Hartmann 6-D + +## The problem + +Hartmann's function in 6 dimensions, a sum of four Gaussian wells on [0, 1]⁶, whose global minimum +is −3.32237; the [Hartmann 6-D](../hartmann6/) example gives its formula and constants. Here it +stands for an expensive function that can be evaluated several times at once: a simulation that +runs for an hour, on 4 machines. What counts is how many rounds of evaluations the search takes, +each as long as one evaluation when its points run side by side, and how many evaluations in all. + +## What makes it hard + +Its second deepest minimum, −3.2032, is nearly as deep as the global one and far from it, with a +basin as wide: a search that finds it first can stay there. The [Bayesian +optimization](../bayesian_optimization/) of one point at a time already has to choose between +exploring and refining; a batch has to choose 4 points before it knows the value of any. + +## Representation + +A `Real` genome of 6 genes in [0, 1]: genoxide's `problems::Hartmann6`, evaluated in Rust in both +languages. + +## Algorithm + +`Bo` with batches of 4 (`.batch(4)`), its other settings the defaults: a Latin hypercube of +2(n + 1) = 14 points, then a Gaussian process with Matérn's 5/2 kernel fitted to every evaluation, +and the log expected improvement maximized for each point. + +The 4 points of a round are chosen one after the other, as Ginsbourger, Le Riche and Carraro +(2010) propose: after each, the model is told the point with a fantasized value, without fitting +its hyperparameters again, and the next point maximizes the acquisition of that model. The +fantasy here is the Kriging believer (`bo::Fantasy::KrigingBeliever`, the default): the model's +own mean at the point. It leaves the model's mean where it was and removes its uncertainty at the +point, so the expected improvement there vanishes and the next point goes elsewhere. The constant +liar (`bo::Fantasy::ConstantLiar`) tells the model a fixed value instead, the lowest, mean or +highest value so far: the higher the lie, the farther the next points go. + +`Engine::parallel(true)` evaluates the 4 points of a round at once; a seed gives the same points +on any number of threads. The run stops within 1e-4 of the minimum, or after 200 evaluations. + +As a contrast, the same search one point a round, with the same seed. + +## Output + +The first lines give the problem and the method. Then a row per round: its number (0 is the +initial design), the evaluations so far, the best value and its distance above the global +minimum. The last lines give the evaluations and rounds each search took to come within 1e-4 of +the minimum, and the distance of the batches' best point from the minimum's. Both versions print +the same rows: the problem is evaluated in Rust. + +[The project page](https://tachsin.gr/projects/genoxide/examples/bo-hartmann6) plots the best value's +distance above the minimum after each round, for both searches. + +## Good results + +A good result is within 1e-4 of −3.32237. The batches get there in round 15, after 74 evaluations, +3.9e-5 above it; one point a round needs 50 evaluations, but 36 rounds: with an evaluation of an +hour and 4 machines, 15 hours against 36. A batch spends more evaluations, since each of its points +is chosen knowing less than a point chosen after the others' results. + +Not every seed finds the global minimum. Over seeds 1 to 20, within 200 evaluations: 13 batch +searches came within 1e-4, after 62 to 90 evaluations (a median of 70, 14 rounds); the other 7 +stayed at the second minimum, −3.2032. One point a round: 12 of 20 (a median of 50 evaluations +and 36 rounds), the others at −3.2032 too. The seed of this example, 3, is the first to reach the +global minimum both ways. With the constant liar's lowest value and a larger initial design (30 or +60 points), or the log transform of the values, the share stayed at 10 to 13 of 20 (to 1e-3); +another run from another design is the way out, as the [Hartmann 6-D](../hartmann6/) example +shows for CMA-ES with restarts. + +The constant liars do about as well on this problem, to 1e-3 within 200 evaluations: 13 of 20 +with the lowest value, 12 with the mean, 11 with the highest, which explores most and needed a +median of 86 evaluations, against the believer's 13. On Branin and Hartmann 3, every seed of 20 +reached 1e-3 within 80 evaluations with each fantasy: the believer and the lowest lie in a median +of 34 evaluations on Branin and 30 on Hartmann 3, the higher lies in 42 to 54 and 40. diff --git a/examples/bo_hartmann6/main.py b/examples/bo_hartmann6/main.py new file mode 100644 index 00000000..5c89f329 --- /dev/null +++ b/examples/bo_hartmann6/main.py @@ -0,0 +1,76 @@ +"""Bayesian optimization of Hartmann's 6-D function in batches: 4 points a round, chosen one after +the other with the Kriging believer and evaluated in parallel, to within 1e-4 of the global +minimum. Then, as a contrast, the same search one point a round: fewer evaluations, more rounds. + +With ``GENOXIDE_TRACE=``, it also writes a trace of both runs for the plot on the example's +page, with trace.py. + + python examples/bo_hartmann6/main.py +""" + +import numpy as np + +import genoxide as gx + +import trace + +# how close to the global minimum, and the evaluations each run may take at most +TOLERANCE = 1e-4 +BUDGET = 200 +SEED = 3 + + +def scientific(value): + """Two significant digits, e.g. 1.2e-7.""" + mantissa, exponent = f"{value:.1e}".split("e") + return f"{mantissa}e{int(exponent)}" + + +problem = gx.problems.Hartmann6() +minimum = problem.optimum.value + + +def search(batch, show): + """A search in batches of ``batch`` points evaluated in parallel, to within the tolerance of + the minimum or the budget, printing a row per round if ``show``; its result, and its best + value's distance above the minimum after each round.""" + rounds = [] + + def on_generation(progress): + rounds.append(progress.best_fitness - minimum) + if show: + print( + f"{progress.generation:>5} {progress.evaluations:>12} " + f"{progress.best_fitness:>13.6f} {scientific(progress.best_fitness - minimum):>10}" + ) + + bo = gx.Bo(problem.genome, batch=batch, objective="minimize", seed=SEED) + result = bo.run( + problem, + target=minimum + TOLERANCE, + evaluations=BUDGET, + parallel=True, + on_generation=on_generation, + ) + return result, rounds + + +print(f"Hartmann's 6-D function in [0, 1]^6: global minimum {minimum:.6f}") +print("14 points of a Latin hypercube, then 4 points a round by log-EI and the Kriging believer") +print("round evaluations best f - f*") +batched, rounds = search(4, True) +distance = float(np.linalg.norm(batched.best_genome - problem.optimum.solutions[0])) +print( + f"4 points a round: within 1e-4 of the minimum after {batched.evaluations} evaluations in " + f"{batched.generations} rounds, {scientific(distance)} from its point" +) +assert batched.best_fitness - minimum <= TOLERANCE + +# the contrast: one point a round, the same seed +single, single_rounds = search(1, False) +reached = single.best_fitness - minimum <= TOLERANCE +print( + f"1 point a round: {'within 1e-4 of the minimum' if reached else 'not within the tolerance'}" + f" after {single.evaluations} evaluations in {single.generations} rounds" +) +trace.write(rounds, single_rounds) diff --git a/examples/bo_hartmann6/main.rs b/examples/bo_hartmann6/main.rs new file mode 100644 index 00000000..97076719 --- /dev/null +++ b/examples/bo_hartmann6/main.rs @@ -0,0 +1,93 @@ +//! Bayesian optimization of Hartmann's 6-D function in batches: 4 points a round, chosen one after +//! the other with the Kriging believer and evaluated in parallel, to within 1e-4 of the global +//! minimum. Then, as a contrast, the same search one point a round: fewer evaluations, more rounds. +//! +//! With `GENOXIDE_TRACE=`, it also writes a trace of both runs for the plot on the +//! example's page, with `trace.rs`. +//! +//! ```text +//! cargo run --release --example bo_hartmann6 +//! ``` + +mod trace; + +use genoxide::prelude::*; +use genoxide::problems::{Hartmann6, Problem}; + +// how close to the global minimum, and the evaluations each run may take at most +const TOLERANCE: f64 = 1e-4; +const BUDGET: u64 = 200; +const SEED: u64 = 3; + +fn main() -> Result<()> { + let optimum = Hartmann6.optimum().expect("known"); + let minimum = optimum.value(); + println!("Hartmann's 6-D function in [0, 1]^6: global minimum {minimum:.6}"); + println!( + "14 points of a Latin hypercube, then 4 points a round by log-EI and the Kriging believer" + ); + println!("round evaluations best f - f*"); + let (batched, rounds) = search(4, minimum, true)?; + let best = batched.best_fitness().score().expect("valid"); + let x = batched.best_genome(); + let distance = x + .iter() + .zip(&optimum.solutions()[0][..]) + .map(|(a, b)| (a - b) * (a - b)) + .sum::() + .sqrt(); + println!( + "4 points a round: within {TOLERANCE:.0e} of the minimum after {} evaluations in {} \ + rounds, {distance:.1e} from its point", + batched.evaluations(), + batched.generations() + ); + assert!(best - minimum <= TOLERANCE); + + // the contrast: one point a round, the same seed + let (single, single_rounds) = search(1, minimum, false)?; + let reached = single.best_fitness().score().expect("valid") - minimum <= TOLERANCE; + println!( + "1 point a round: {} after {} evaluations in {} rounds", + if reached { + format!("within {TOLERANCE:.0e} of the minimum") + } else { + "not within the tolerance".to_string() + }, + single.evaluations(), + single.generations() + ); + trace::write(&rounds, &single_rounds); + Ok(()) +} + +// a search in batches of `batch` points evaluated in parallel, to within the tolerance of +// `minimum` or the budget, printing a row per round if `print`; its outcome, and its best value's +// distance above the minimum after each round +fn search(batch: usize, minimum: f64, print: bool) -> Result<(Outcome, Vec)> { + let bo = Bo::builder(Hartmann6.representation()) + .batch(batch) + .minimize() + .seed(SEED) + .build()?; + let mut rounds = Vec::new(); + let outcome = Engine::new(bo, Hartmann6) + .parallel(true) + .stop_when(Stop::target(minimum + TOLERANCE).or(Stop::evaluations(BUDGET))) + .on_generation(|snapshot| { + let progress = snapshot.progress(); + let best = progress.best().and_then(Fitness::score).expect("valid"); + rounds.push(best - minimum); + if print { + println!( + "{:>5} {:>12} {:>13.6} {:>10}", + progress.generation(), + progress.evaluations(), + best, + format!("{:.1e}", best - minimum) + ); + } + }) + .run()?; + Ok((outcome, rounds)) +} diff --git a/examples/bo_hartmann6/output.txt b/examples/bo_hartmann6/output.txt new file mode 100644 index 00000000..df89d240 --- /dev/null +++ b/examples/bo_hartmann6/output.txt @@ -0,0 +1,21 @@ +Hartmann's 6-D function in [0, 1]^6: global minimum -3.322368 +14 points of a Latin hypercube, then 4 points a round by log-EI and the Kriging believer +round evaluations best f - f* + 0 14 -0.885455 2.4e0 + 1 18 -0.942238 2.4e0 + 2 22 -0.942238 2.4e0 + 3 26 -1.154749 2.2e0 + 4 30 -1.170522 2.2e0 + 5 34 -1.381891 1.9e0 + 6 38 -1.381891 1.9e0 + 7 42 -1.875810 1.4e0 + 8 46 -2.431634 8.9e-1 + 9 50 -2.756340 5.7e-1 + 10 54 -2.756340 5.7e-1 + 11 58 -2.969910 3.5e-1 + 12 62 -3.241253 8.1e-2 + 13 66 -3.317829 4.5e-3 + 14 70 -3.321314 1.1e-3 + 15 74 -3.322329 3.9e-5 +4 points a round: within 1e-4 of the minimum after 74 evaluations in 15 rounds, 1.4e-3 from its point +1 point a round: within 1e-4 of the minimum after 50 evaluations in 36 rounds diff --git a/examples/bo_hartmann6/trace.json b/examples/bo_hartmann6/trace.json new file mode 100644 index 00000000..305ed9f3 --- /dev/null +++ b/examples/bo_hartmann6/trace.json @@ -0,0 +1,39 @@ +{"example":"bo_hartmann6","format":1,"log_y":true,"objective":"minimize","optimum":0.0,"plot":"multi-curve","problem":{"series":["4 points a round","1 point a round"]},"x_label":"rounds","y_label":"best value's distance above the global minimum","frames":[ +{"best":2.43691,"generation":0,"state":{"values":{"1 point a round":2.43691,"4 points a round":2.43691}}}, +{"best":2.38013,"generation":1,"state":{"values":{"1 point a round":2.38013,"4 points a round":2.38013}}}, +{"best":2.38013,"generation":2,"state":{"values":{"1 point a round":2.38013,"4 points a round":2.38013}}}, +{"best":2.16762,"generation":3,"state":{"values":{"1 point a round":2.23575,"4 points a round":2.16762}}}, +{"best":2.15185,"generation":4,"state":{"values":{"1 point a round":2.11126,"4 points a round":2.15185}}}, +{"best":1.94048,"generation":5,"state":{"values":{"1 point a round":2.11126,"4 points a round":1.94048}}}, +{"best":1.94048,"generation":6,"state":{"values":{"1 point a round":1.90827,"4 points a round":1.94048}}}, +{"best":1.44656,"generation":7,"state":{"values":{"1 point a round":1.90827,"4 points a round":1.44656}}}, +{"best":0.890734,"generation":8,"state":{"values":{"1 point a round":1.90827,"4 points a round":0.890734}}}, +{"best":0.566028,"generation":9,"state":{"values":{"1 point a round":1.90827,"4 points a round":0.566028}}}, +{"best":0.566028,"generation":10,"state":{"values":{"1 point a round":1.90827,"4 points a round":0.566028}}}, +{"best":0.352458,"generation":11,"state":{"values":{"1 point a round":1.88621,"4 points a round":0.352458}}}, +{"best":0.081115,"generation":12,"state":{"values":{"1 point a round":1.88621,"4 points a round":0.081115}}}, +{"best":0.00453888,"generation":13,"state":{"values":{"1 point a round":1.49937,"4 points a round":0.00453888}}}, +{"best":0.00105441,"generation":14,"state":{"values":{"1 point a round":1.26101,"4 points a round":0.00105441}}}, +{"best":3.85755e-05,"generation":15,"state":{"values":{"1 point a round":1.10945,"4 points a round":3.85755e-05}}}, +{"best":3.85755e-05,"generation":16,"state":{"values":{"1 point a round":1.10945,"4 points a round":3.85755e-05}}}, +{"best":3.85755e-05,"generation":17,"state":{"values":{"1 point a round":0.788134,"4 points a round":3.85755e-05}}}, +{"best":3.85755e-05,"generation":18,"state":{"values":{"1 point a round":0.486839,"4 points a round":3.85755e-05}}}, +{"best":3.85755e-05,"generation":19,"state":{"values":{"1 point a round":0.425581,"4 points a round":3.85755e-05}}}, +{"best":3.85755e-05,"generation":20,"state":{"values":{"1 point a round":0.425581,"4 points a round":3.85755e-05}}}, +{"best":3.85755e-05,"generation":21,"state":{"values":{"1 point a round":0.253761,"4 points a round":3.85755e-05}}}, +{"best":3.85755e-05,"generation":22,"state":{"values":{"1 point a round":0.253761,"4 points a round":3.85755e-05}}}, +{"best":3.85755e-05,"generation":23,"state":{"values":{"1 point a round":0.0910801,"4 points a round":3.85755e-05}}}, +{"best":3.85755e-05,"generation":24,"state":{"values":{"1 point a round":0.0910801,"4 points a round":3.85755e-05}}}, +{"best":3.85755e-05,"generation":25,"state":{"values":{"1 point a round":0.0910801,"4 points a round":3.85755e-05}}}, +{"best":3.85755e-05,"generation":26,"state":{"values":{"1 point a round":0.0505678,"4 points a round":3.85755e-05}}}, +{"best":3.85755e-05,"generation":27,"state":{"values":{"1 point a round":0.0094041,"4 points a round":3.85755e-05}}}, +{"best":3.85755e-05,"generation":28,"state":{"values":{"1 point a round":0.00618008,"4 points a round":3.85755e-05}}}, +{"best":3.85755e-05,"generation":29,"state":{"values":{"1 point a round":0.00618008,"4 points a round":3.85755e-05}}}, +{"best":3.85755e-05,"generation":30,"state":{"values":{"1 point a round":0.00618008,"4 points a round":3.85755e-05}}}, +{"best":3.85755e-05,"generation":31,"state":{"values":{"1 point a round":0.0023628,"4 points a round":3.85755e-05}}}, +{"best":3.85755e-05,"generation":32,"state":{"values":{"1 point a round":0.0023628,"4 points a round":3.85755e-05}}}, +{"best":3.85755e-05,"generation":33,"state":{"values":{"1 point a round":0.0023628,"4 points a round":3.85755e-05}}}, +{"best":3.85755e-05,"generation":34,"state":{"values":{"1 point a round":0.000444854,"4 points a round":3.85755e-05}}}, +{"best":3.85755e-05,"generation":35,"state":{"values":{"1 point a round":0.000232423,"4 points a round":3.85755e-05}}}, +{"best":3.85755e-05,"generation":36,"state":{"values":{"1 point a round":7.03519e-05,"4 points a round":3.85755e-05}}} +]} diff --git a/examples/bo_hartmann6/trace.py b/examples/bo_hartmann6/trace.py new file mode 100644 index 00000000..9f411d33 --- /dev/null +++ b/examples/bo_hartmann6/trace.py @@ -0,0 +1,70 @@ +"""The trace of the runs for the plot on the example's page, written to the file that +``GENOXIDE_TRACE`` names: the best value's distance above the global minimum after each round, of +the search 4 points a round and of the search one point a round, which goes on for more rounds. +The Rust example writes the same file.""" + +import json +import math +import os + +# the lines of the plot +SERIES = ["4 points a round", "1 point a round"] + + +def write(batched, single): + """Writes the trace, if GENOXIDE_TRACE names a file, from the distances above the minimum + after each round of both searches; a search that has stopped keeps its last value.""" + path = os.environ.get("GENOXIDE_TRACE") + if not path: + return + rounds = max(len(batched), len(single)) + + def at(values, round_): + return values[min(round_, len(values) - 1)] + + frames = [ + { + "generation": round_, + "best": at(batched, round_), + "state": {"values": dict(zip(SERIES, [at(batched, round_), at(single, round_)]))}, + } + for round_ in range(rounds) + ] + settings = { + "format": 1, + "example": "bo_hartmann6", + "objective": "minimize", + "x_label": "rounds", + "y_label": "best value's distance above the global minimum", + "log_y": True, + "optimum": 0.0, + "plot": "multi-curve", + "problem": {"series": SERIES}, + } + write_trace(path, settings, frames) + + +# ---- the same in every example's trace ---------------------------------------------------------- + + +def write_trace(path, settings, frames): + """Writes the settings and the frames to ``path``, a frame per line.""" + lines = ",\n".join(map(to_json, frames)) + with open(path, "w", encoding="utf-8", newline="\n") as file: + file.write(f'{to_json(settings)[:-1]},"frames":[\n{lines}\n]}}\n') + + +def to_json(value): + """Compact JSON with sorted keys, and numbers rounded to 6 significant digits, as the Rust + example writes it.""" + return json.dumps(rounded(value), sort_keys=True, separators=(",", ":"), ensure_ascii=False) + + +def rounded(value): + if isinstance(value, dict): + return {key: rounded(item) for key, item in value.items()} + if isinstance(value, (list, tuple)): + return [rounded(item) for item in value] + if isinstance(value, float): + return float(f"{value:.5e}") if math.isfinite(value) else None + return value diff --git a/examples/bo_hartmann6/trace.rs b/examples/bo_hartmann6/trace.rs new file mode 100644 index 00000000..ccef2f3d --- /dev/null +++ b/examples/bo_hartmann6/trace.rs @@ -0,0 +1,90 @@ +//! The trace of the runs for the plot on the example's page, written to the file that +//! `GENOXIDE_TRACE` names: the best value's distance above the global minimum after each round, of +//! the search 4 points a round and of the search one point a round, which goes on for more rounds. +//! The Python example writes the same file. + +use serde_json::{Value, json}; + +// the lines of the plot +const SERIES: [&str; 2] = ["4 points a round", "1 point a round"]; + +// writes the trace, if GENOXIDE_TRACE names a file, from the distances above the minimum after +// each round of both searches; a search that has stopped keeps its last value +pub fn write(batched: &[f64], single: &[f64]) { + let Ok(path) = std::env::var("GENOXIDE_TRACE") else { + return; + }; + let rounds = batched.len().max(single.len()); + let at = |values: &[f64], round: usize| values[round.min(values.len() - 1)]; + let frames = (0..rounds) + .map(|round| { + let values: serde_json::Map = SERIES + .iter() + .zip([at(batched, round), at(single, round)]) + .map(|(name, value)| (name.to_string(), json!(value))) + .collect(); + json!({ + "generation": round, + "best": at(batched, round), + "state": { "values": values }, + }) + }) + .collect(); + let settings = json!({ + "format": 1, + "example": "bo_hartmann6", + "objective": "minimize", + "x_label": "rounds", + "y_label": "best value's distance above the global minimum", + "log_y": true, + "optimum": 0.0, + "plot": "multi-curve", + "problem": { "series": SERIES }, + }); + write_trace(&path, settings, frames); +} + +// ---- the same in every example's trace --------------------------------------------------------- + +// writes the settings and the frames to `path`, a frame per line +fn write_trace(path: &str, settings: Value, frames: Vec) { + let frames: Vec = frames.iter().map(to_json).collect(); + let settings = to_json(&settings); + let head = &settings[..settings.len() - 1]; + let text = format!("{head},\"frames\":[\n{}\n]}}\n", frames.join(",\n")); + std::fs::write(path, text).expect("the trace is written"); +} + +// compact JSON with sorted keys, and numbers rounded to 6 significant digits and written as +// Python writes them (7542.0, 1e-08): the Python example writes the same file +fn to_json(value: &Value) -> String { + let join = |items: Vec| items.join(","); + match value { + Value::Number(number) if number.is_f64() => python_float(number.as_f64().expect("f64")), + Value::Array(items) => format!("[{}]", join(items.iter().map(to_json).collect())), + Value::Object(map) => { + let entry = + |(key, item): (&String, &Value)| format!("{}:{}", json!(key), to_json(item)); + format!("{{{}}}", join(map.iter().map(entry).collect())) + } + other => other.to_string(), + } +} + +fn python_float(value: f64) -> String { + let rounded: f64 = format!("{value:.5e}").parse().expect("a number"); + let shortest = format!("{rounded:e}"); + let (mantissa, exponent) = shortest.split_once('e').expect("an exponent"); + let exponent: i32 = exponent.parse().expect("an exponent"); + if (-4..16).contains(&exponent) { + let text = rounded.to_string(); + if text.contains('.') { + text + } else { + text + ".0" + } + } else { + let sign = if exponent < 0 { '-' } else { '+' }; + format!("{mantissa}e{sign}{:02}", exponent.abs()) + } +} diff --git a/examples/check_traces.py b/examples/check_traces.py index f11cad50..86467613 100644 --- a/examples/check_traces.py +++ b/examples/check_traces.py @@ -5,8 +5,8 @@ import json import sys -# an example whose numbers depend on the machine -SKIP = {"asynchronous"} +# the examples whose numbers depend on the machine +SKIP = {"asynchronous", "bo_asynchronous"} failed = [] for path in sorted(glob.glob("examples/*/trace.json")): diff --git a/python/README.md b/python/README.md index 5c94c59a..8dc3dbab 100644 --- a/python/README.md +++ b/python/README.md @@ -10,7 +10,7 @@ The algorithms of [genoxide](https://github.com/tachsin/genoxide), a Rust librar - L-BFGS-B for smooth functions with a gradient, yours, finite differences or the test problems' own - first-order gradient methods for up to millions of parameters: gradient descent, momentum, Nesterov, Adam and AdamW - MMA and GCMMA, the method of moving asymptotes, for millions of variables with few constraints, from gradients -- Bayesian optimization for expensive functions, with Gaussian processes (`gx.model.gp`) you can also fit on their own +- Bayesian optimization for expensive functions, in batches evaluated in parallel, with constraints and on integer genes, with Gaussian processes (`gx.model.gp`) you can also fit on their own - continuation: a gradient method through stages of one problem, its state kept between them - differential evolution, evolution strategies, CMA-ES and particle swarm optimization - NEAT, OpenAI's evolution strategy, neural networks and pole-balancing tasks, for neuroevolution @@ -115,7 +115,7 @@ An exception in the fitness function stops the run, and `run` raises it. So does | Yes / no choices (subsets) | `Binary` | `Ga` with `UniformCrossover()` or `PointCrossover(points)`, and `BitFlip` | | An order (tours, sequencing) | `Permutation` | `LocalSearch`, which often beats a GA on permutations; `Ga` with `OrderCrossover()` (sequences) or `EdgeRecombinationCrossover()` (tours) | | Reals in ranges | `Real` | `Cmaes`; `De`; `Es`; `Ga` with `SimulatedBinaryCrossover(eta)` and `PolynomialMutation(eta)`; `Lbfgsb` for a local minimum of a smooth function, any number of genes; `NelderMead` for a local minimum in a few dimensions | -| An expensive function of a few reals (tens to a few hundred evaluations) | `Real` | `Bo`, Bayesian optimization, up to about 10 to 20 genes | +| An expensive function of a few reals or integers (tens to a few hundred evaluations) | `Real`, `Integer` | `Bo`, Bayesian optimization, up to about 10 to 20 genes | | A smooth function of many reals, with its gradient | `Real` | `FirstOrder` (Adam, momentum, Nesterov), up to millions of genes | | Smooth, with gradients: very many reals (up to millions), few inequality constraints | `Real` | `Mma`; `method="gcmma"` to converge from any start | | A neural network's weights | `Real`, from `network.representation(bounds)` | `Cmaes` up to a few hundred weights; `OpenEs` for thousands and more | @@ -133,7 +133,7 @@ An exception in the fitness function stops the run, and `run` raises it. So does - `FirstOrder` steps along the gradient of a smooth function by a step rule, without a line search: `step="adam"` (Kingma and Ba's, the default), `"adamw"` (with decoupled weight decay), `"momentum"`, `"nesterov"` or `"gradient"`. One gradient per generation, from `run(..., gradient=...)` as for `Lbfgsb`, from a problem of `genoxide.problems` in Rust, or by finite differences (`n` more evaluations per generation, up to 10,000 genes). Memory and work per step are linear in the genes, for up to millions of them. It stops with the stop reason `"converged"` when the gradient vanishes; a `control` lowers the learning rate over the run (a schedule), which Adam needs to settle on the minimum. - `Mma`, Svanberg's method of moving asymptotes, uses the gradient of the score and of each inequality constraint `g(x) <= 0`: the gradient as for `Lbfgsb` (no finite differences), and with `run(f, gradient=True, constraints=m)`, `f` returns `(value, gradient, g, jacobian)`, `g` the constraints' values and `jacobian` an array of a row per constraint. An iteration is one evaluation and a few passes over the genes, with no matrix of them: for up to millions of genes and up to a few hundred constraints. It stops on its own once it has converged (the stop reason `"converged"`). `method="gcmma"` converges from any start, at the cost of more evaluations; `constraint_cost` must exceed the constraints' multipliers. - `Continuation` runs `FirstOrder`, `Lbfgsb` or `Mma` through stages of one problem, a smooth version first and sharper ones after (a smoothing that shrinks, a p-norm's p that grows, a penalty raised): `on_stage(index)` sets the stage's parameters for the fitness function, and the method goes on from its point with its state (Adam's averages, MMA's asymptotes; `keep="point"` keeps only the point). A stage ends when the method converges or after `generations`; the run, after the last stage. -- `Bo`, Bayesian optimization, is for expensive functions, where each evaluation counts: a Gaussian process (`gx.model.gp`) models the function from every evaluation, and an acquisition function of its posterior chooses the next point, one per generation after an initial design of `2(n + 1)` points. The log expected improvement by default; `"ei"`, `gx.ProbabilityOfImprovement(xi)` and `gx.UpperConfidenceBound(beta)` (whose `beta` a `control` can change). `output="log"` models the logarithm of the values, for objectives that span orders of magnitude. The model's fit costs O(N^3) for N evaluations: up to a few hundred evaluations, in up to about 10 to 20 genes. A running `Bo` gives its `model` and `acquisition_at(points)` to a `control`, e.g. for a plot. +- `Bo`, Bayesian optimization, is for expensive functions, where each evaluation counts: a Gaussian process (`gx.model.gp`) models the function from every evaluation, and an acquisition function of its posterior chooses the next point, one per generation after an initial design of `2(n + 1)` points. The log expected improvement by default; `"ei"`, `gx.ProbabilityOfImprovement(xi)` and `gx.UpperConfidenceBound(beta)` (whose `beta` a `control` can change). `output="log"` models the logarithm of the values, for objectives that span orders of magnitude. `batch=q` evaluates q points a generation, in parallel with `run(..., parallel=True)`, each chosen after the ones before it are added to the model with a `fantasy` value (`"believer"`, or the constant liar `"liar-min"`, `"liar-mean"`, `"liar-max"`). `run(f, constraints=m)` with `f` returning `(value, g)` models each constraint's values and weighs the acquisition by the probability of feasibility; the test problems with constraints give theirs. On an `Integer` genome, the genes are rounded inside the model and the acquisition is maximized on the lattice. The model's fit costs O(N^3) for N evaluations: up to a few hundred evaluations, in up to about 10 to 20 genes. A running `Bo` gives its `model`, `acquisition_at(points)` and `probability_of_feasibility_at(points)` to a `control`, e.g. for a plot. - `Islands` of `Ga`s or `De`s evolve apart and exchange their best: more diverse than one large population, and often faster on multimodal problems. ## Algorithms @@ -149,7 +149,7 @@ An exception in the fitness function stops the run, and `run` raises it. So does | `Neat` | its networks | `inputs`, `outputs` (needed), `population_size` (150), `compatibility` (`(1.0, 1.0, 0.4, 3.0)`: c1, c2, c3, threshold), `weight_mutation` (`(0.8, 0.1)`: rate, replace), `weight_deviations` (`(1.0, 1.0)`), `structural_mutation` (`(0.03, 0.05)`: add node, add connection), `reproduction` (`(0.25, 0.001, 0.75)`), `selection` (`(5, 0.2)`: elitism size, survival), `stagnation` (15), `activation` (`"steep_sigmoid"`), `feed_forward` (True), `initial` (`"fully_connected"`; `"unconnected"`), `sharing` (`"normalized"`; `"raw"`, the paper's) | | `Pso` | real | `population_size` (needed), `ring` (neighbors on each side) | | `NelderMead` | real | `coefficients` (`"adaptive"`, Gao and Han's; `"standard"`; `(reflection, expansion, contraction, shrink)`), `initial_step` (0.1 of each range) or `initial_step_absolute` (a distance), `tolerance` (1e-9 of the initial step), `restarts` (none; random restarts), `speculative` (False), `initial_genome` (a random point) | -| `Bo` | real | `initial_points` (2(n + 1)), `initial_genomes` (none), `acquisition` (`"log-ei"`; `"ei"`, `ProbabilityOfImprovement(xi)`, `UpperConfidenceBound(beta)`), `kernel` (`"matern52"`; `"squared_exponential"`), `noise` (0: the model interpolates; a fixed fraction of the values' variance, or `gx.model.gp.Learned(min)`), `output` (`"standardize"`; `"log"`), `raw_samples` (1000), `acquisition_starts` (10), `hyperparameter_starts` (5) | +| `Bo` | real, integer | `initial_points` (2(n + 1)), `initial_genomes` (none), `acquisition` (`"log-ei"`; `"ei"`, `ProbabilityOfImprovement(xi)`, `UpperConfidenceBound(beta)`), `kernel` (`"matern52"`; `"squared_exponential"`), `noise` (0: the model interpolates; a fixed fraction of the values' variance, or `gx.model.gp.Learned(min)`), `output` (`"standardize"`; `"log"`), `raw_samples` (1000), `acquisition_starts` (10), `hyperparameter_starts` (5), `batch` (1), `fantasy` (`"believer"`; `"liar-min"`, `"liar-mean"`, `"liar-max"`) | | `Lbfgsb` | real | `memory` (10), `gradients` (`"auto"`; `"supplied"`, `"forward"`, `"central"`), `difference_step` (√ε forward, ε^(1/3) central), `gradient_tolerance` (1e-5), `function_tolerance` (2.2e-9), `max_line_search` (20), `restarts` (none; random restarts), `initial_genome` (a random point) | | `Continuation` | real | `algorithm` (a `FirstOrder`, `Lbfgsb` or `Mma`), `stages`, `on_stage` (needed), `generations` (none: each stage to convergence), `keep` (`"state"`; `"point"`), `on_stage_finished` (none); `Lbfgsb(keep_pairs=True)` keeps its pairs between stages; the result's `stages` | | `Mma` | real | `method` (`"mma"`; `"gcmma"`), `asymptote_initial` (0.5 of each range), `asymptote_decrease` (0.7), `asymptote_increase` (1.2), `move_limit` (0.5 of each range), `constraint_cost` (1000), `kkt_tolerance` (1e-9), `step_tolerance` (1e-10 of each range), `restoration` (True), `parallel_sums` (False), `initial_genome` (a random point); `run(f, gradient=True, constraints=m, ...)` | @@ -640,7 +640,7 @@ result = ga.run(lambda bits: bits.sum(), generations=1_000, on_generation=report | `Pso` | `RunningPso` | `inertia`, `acceleration` (`(cognitive, social)`) | | `LocalSearch` | `RunningLocalSearch` | `neighbor`, `neighbors` | | `NelderMead` | `RunningNelderMead` | none: its steps follow from its simplex. It reads `converged`, `size` (of the simplex, a fraction of each range), `iterations` and `restart_count` | -| `Bo` | `RunningBo` | `acquisition` (e.g. `UpperConfidenceBound(beta)` with `beta` on a schedule). It reads `initial_points`, `model` (the `gx.model.gp.GaussianProcess` that chose the last point, None in generation 0) and `acquisition_at(points)` | +| `Bo` | `RunningBo` | `acquisition` (e.g. `UpperConfidenceBound(beta)` with `beta` on a schedule), `batch`, `fantasy`. It reads `initial_points`, `constraints`, `model` (the `gx.model.gp.GaussianProcess` that chose the last point, None in generation 0), `acquisition_at(points)` and `probability_of_feasibility_at(points)` | | `Lbfgsb` | `RunningLbfgsb` | `memory`. It reads `pairs`, `converged` (the criterion), `projected_gradient`, `iterations`, `gradients` (the source in use), `gradient_evaluations` and `stencil_evaluations` (the cost of finite differences), `skipped_pairs`, `memory_resets` and `restart_count` | | `Mma` | `RunningMma` | none: its steps follow from its approximations. It reads `converged` (`"kkt"`, `"step"` or None), `iterations`, `inner_iterations`, `multipliers` and `kkt_residual` | | `Continuation` | the wrapped method's | as the wrapped method | @@ -795,6 +795,7 @@ Some names differ: | `Lbfgsb.run(f, gradient=g)`, `gradient=True` | `Differentiable(\|x, gradient\| ...)` | | `Mma(method="gcmma")` | `.method(mma::Method::Gcmma)` | | `Bo(acquisition=gx.UpperConfidenceBound(beta))`, `Bo(output="log")`, `Bo(noise=gx.model.gp.Learned(1e-6))` | `.acquisition(bo::Acquisition::UpperConfidenceBound { beta })`, `.output(bo::Output::Log)`, `.noise(model::gp::Noise::Learned { min: 1e-6 })` | +| `Bo(batch=4, fantasy="liar-min")`, `bo.run(f, constraints=2)` | `.batch(4).fantasy(bo::Fantasy::ConstantLiar(bo::Lie::Min))`, `Constrained::new(2, f)` | | `gx.model.gp.GaussianProcess.fit(genome, points, values)`, `model.predict(points)` | `GaussianProcess::builder(real).fit(&points, &values)?`, `model.predict(&x)` | | `Mma.run(f, gradient=True, constraints=m)`, `f` returning `(value, gradient, g, jacobian)` | `Constrained::differentiable(m, \|x, gradient, g, jacobian\| value)` | | `Continuation(method, stages=n, on_stage=f, generations=g, keep="point")`, `result.stages` | `Continuation::builder(method).stages(n).on_stage(\|stage, _\| ...).generations(g).keep(Keep::Point)`, `continuation.stages()` | diff --git a/python/genoxide/__init__.py b/python/genoxide/__init__.py index 62d8e347..4afe8015 100644 --- a/python/genoxide/__init__.py +++ b/python/genoxide/__init__.py @@ -1700,6 +1700,33 @@ def initial_points(self) -> int: """The points of the initial design.""" return int(self._get("initial_points")) + @property + def batch(self) -> int: + """The points of each generation after the initial design, at least 1. A new value + applies from the next generation.""" + return int(self._get("batch")) + + @batch.setter + def batch(self, value: int) -> None: + self._set("batch", _whole("batch", value, minimum=None)) + + @property + def fantasy(self) -> BoFantasy: + """What the points of a batch are taken to be worth while the next ones are chosen: + "believer", "liar-min", "liar-mean" or "liar-max". A new value applies from the next + generation.""" + return cast(BoFantasy, self._get("fantasy")) + + @fantasy.setter + def fantasy(self, value: BoFantasy) -> None: + self._set("fantasy", _fantasy(value)) + + @property + def constraints(self) -> int: + """The number of inequality constraints whose values the fitness function gives: 0 + without constraints.""" + return int(self._get("constraints") or 0) + @property def model(self) -> _surrogate.GaussianProcess | None: """The Gaussian process that chose the last point, fitted to the evaluations before it: @@ -1720,6 +1747,17 @@ def acquisition_at(self, points: Any) -> np.ndarray: rows = rows.reshape(1, -1) return np.asarray(self._native.acquisition(rows)) + def probability_of_feasibility_at(self, points: Any) -> np.ndarray: + """The probability that ``points`` are feasible under the constraints' models, a point + per row (a 1-D array is one point): the product of the probabilities that each + constraint's value is at most 0, the models taken as independent. 1 without constraints. + Raises a ``ValueError`` without a model.""" + rows = np.ascontiguousarray(points, dtype=np.float64) + if rows.ndim == 1: + rows = rows.reshape(1, -1) + return np.asarray(self._native.feasibility(rows)) + + class RunningLbfgsb(Running): """A running :class:`Lbfgsb`, for ``control``: its ``memory``, which can change, and its state, read-only. ``reevaluate()`` scores the current point again and drops the correction @@ -3624,6 +3662,43 @@ def _describe(self) -> dict[str, Any]: BoAcquisition = Union[Literal["log-ei", "ei"], ProbabilityOfImprovement, UpperConfidenceBound] +# what Bayesian optimization takes a point not evaluated yet to be worth +BoFantasy = Literal["believer", "liar-min", "liar-mean", "liar-max"] + + +def _fantasy(fantasy: Any) -> str: + if isinstance(fantasy, str) and fantasy in ("believer", "liar-min", "liar-mean", "liar-max"): + return fantasy + raise ValueError( + f'fantasy is "believer", "liar-min", "liar-mean" or "liar-max", not {fantasy!r}' + ) + + +def _with_values( + function: Callable[[np.ndarray], Any], constraints: int +) -> Callable[[np.ndarray], Any]: + """A function returning ``(value, g)``, ``g`` the ``constraints`` values a float64 array.""" + + def evaluate(genome: np.ndarray) -> tuple[Any, np.ndarray]: + result = function(genome) + if not (isinstance(result, tuple) and len(result) == 2): + raise TypeError( + "with constraints, the fitness function returns a tuple (value, constraint " + "values), not " + + type(result).__name__ + + (f" of length {len(result)}" if isinstance(result, tuple) else "") + ) + value, values = result + values = np.asarray(values, dtype=np.float64).reshape(-1) + if values.shape != (constraints,): + raise ValueError( + f"the fitness function returns {values.size} constraint values, for " + f"{constraints} constraints" + ) + return value, values + + return evaluate + def _acquisition(acquisition: Any) -> dict[str, Any]: if isinstance(acquisition, (ProbabilityOfImprovement, UpperConfidenceBound)): @@ -3648,22 +3723,32 @@ def _acquisition_of(description: dict[str, Any]) -> BoAcquisition: class Bo(_SingleObjective): - """Bayesian optimization. Real genomes. + """Bayesian optimization. Real or Integer genomes. For expensive black-box functions, such as a simulation that runs for minutes or a physical experiment, where tens to a few hundred evaluations must do. A Gaussian process (:mod:`genoxide.model.gp`) models the function from every evaluation so far, and an - acquisition function of its posterior picks the next point, trading the model's best guesses + acquisition function of its posterior picks the next points, trading the model's best guesses against its uncertainty. Generation 0 evaluates an initial design: the ``initial_genomes``, and a Latin hypercube (McKay, Beckman and Conover 1979) for the rest of ``initial_points``. - Each later generation fits the model and evaluates one point, the acquisition's maximum: from - ``raw_samples`` random points, then L-BFGS-B with the acquisition's gradient from the best - ``acquisition_starts`` of them and from the best point so far. A point is never evaluated - twice. The model fits the scores to minimize (negated when maximizing), through ``output``; - an invalid score enters it at the worst value of the others, so the search learns to avoid - where the function fails, and a constraint violation is ignored by the search (use a - penalty). The model's fit costs O(N^3) for N evaluations: up to a few hundred evaluations of - a function far more expensive than that, in up to about 10 to 20 genes. + Each later generation fits the model and evaluates ``batch`` points, by default one, the + acquisition's maximum: from ``raw_samples`` random points, then L-BFGS-B with the + acquisition's gradient from the best ``acquisition_starts`` of them and from the best point so + far. Each further point of a batch is chosen after the ones before it are added to the model + with a ``fantasy`` value (Ginsbourger, Le Riche and Carraro 2010), the hyperparameters kept; + with ``parallel=True`` in :meth:`run`, the points of a batch are evaluated together. A point + is never evaluated twice. The model fits the scores to minimize (negated when maximizing), + through ``output``; an invalid score enters it at the worst value of the others, so the search + learns to avoid where the function fails. With ``constraints=m`` in :meth:`run` (or a test + problem of :mod:`genoxide.problems` with constraints, which gives their values), a Gaussian + process models each constraint, and the acquisition is weighed by the probability that a + point is feasible (Gardner et al. 2014); until a point is feasible, that probability alone is + maximized. Without them, a constraint violation is ignored by the search (use a penalty). On + an :class:`Integer` genome, the genes are rounded to the nearest integer inside the model's + kernel (Garrido-Merchán and Hernández-Lobato 2020), and the acquisition is maximized on the + lattice, by a hill climb of steps of one in one gene. The model's fit costs O(N^3) for N + evaluations: up to a few hundred evaluations of a function far more expensive than that, in + up to about 10 to 20 genes. Branin's function, whose three global minima are 0.397887, in 40 evaluations:: @@ -3674,12 +3759,27 @@ class Bo(_SingleObjective): result = bo.run(problem, evaluations=40) print(result.best_fitness) # 0.3978873... + A constraint's values one by one: the fitness function returns ``(value, g)``, feasible + where every value of ``g`` is at most 0:: + + import numpy as np + + import genoxide as gx + + def disc(x): + # the sum inside the unit disc: the minimum -1.414... at (-0.707..., -0.707...) + return x[0] + x[1], np.array([x[0] ** 2 + x[1] ** 2 - 1.0]) + + bo = gx.Bo(gx.Real((-2.0, 2.0), length=2), objective="minimize", seed=1) + result = bo.run(disc, constraints=1, evaluations=30) + print(result.best_fitness) # -1.41... + The model, its likelihood's maximization and the acquisition's are seeded and portable: a seed repeats the run on every platform, and gives a Rust program's results. Parameters ---------- - genome : Real + genome : Real or Integer The search space. At least one gene needs ``low < high``. initial_points : int, optional The points of the initial design, at least 1. None is 2(n + 1) for the n genes with @@ -3714,6 +3814,16 @@ class Bo(_SingleObjective): hyperparameter_starts : int, default 5 The starts of the likelihood's maximization at each fit, at least 1: the last fit's hyperparameters, then random ones. + batch : int, default 1 + The points of each generation after the initial design, at least 1, chosen one after the + other, each after the ones before it are added to the model with a ``fantasy`` value: + with ``parallel=True``, a generation takes about as long as one evaluation. A batch needs + more evaluations than a point at a time, and fewer generations. + fantasy : {"believer", "liar-min", "liar-mean", "liar-max"}, default "believer" + What a point of a batch is taken to be worth while the next ones are chosen: the model's + mean there (the Kriging believer, which only removes the uncertainty there), or a lie, + the lowest, mean or highest value the model fits (the constant liar: the higher, the + farther the next points go). The constraints' models take their mean either way. objective : {"maximize", "minimize"}, default "maximize" Whether higher or lower scores are better. seed : int, optional @@ -3724,14 +3834,20 @@ class Bo(_SingleObjective): of expensive black-box functions. *Journal of Global Optimization* 13(4): 455-492. Rasmussen, C. E. and Williams, C. K. I. (2006). *Gaussian Processes for Machine Learning.* MIT Press. Ament, S., Daulton, S., Eriksson, D., Balandat, M. and Bakshy, E. (2023). Unexpected - improvements to expected improvement for Bayesian optimization. *NeurIPS 2023*. + improvements to expected improvement for Bayesian optimization. *NeurIPS 2023*. Ginsbourger, + D., Le Riche, R. and Carraro, L. (2010). Kriging is well-suited to parallelize optimization. + In *Computational Intelligence in Expensive Optimization Problems*, Springer: 131-162. + Gardner, J. R., Kusner, M. J., Xu, Z., Weinberger, K. Q. and Cunningham, J. P. (2014). + Bayesian optimization with inequality constraints. *ICML 2014*. Garrido-Merchán, E. C. and + Hernández-Lobato, D. (2020). Dealing with categorical and integer-valued variables in + Bayesian optimization with Gaussian processes. *Neurocomputing* 380: 20-35. """ _running = RunningBo def __init__( self, - genome: Real, + genome: Real | Integer, *, initial_points: int | None = None, initial_genomes: Sequence[Sequence[float]] | np.ndarray | None = None, @@ -3742,6 +3858,8 @@ def __init__( raw_samples: int = 1000, acquisition_starts: int = 10, hyperparameter_starts: int = 5, + batch: int = 1, + fantasy: BoFantasy = "believer", objective: ObjectiveName = "maximize", seed: int | None = None, ) -> None: @@ -3756,6 +3874,8 @@ def __init__( self.raw_samples = raw_samples self.acquisition_starts = acquisition_starts self.hyperparameter_starts = hyperparameter_starts + self.batch = batch + self.fantasy = fantasy self.seed = seed def _describe(self) -> dict[str, Any]: @@ -3783,9 +3903,103 @@ def _describe(self) -> dict[str, Any]: "raw_samples": _whole("raw_samples", self.raw_samples), "acquisition_starts": _whole("acquisition_starts", self.acquisition_starts), "hyperparameter_starts": _whole("hyperparameter_starts", self.hyperparameter_starts), + "batch": _whole("batch", self.batch), + "fantasy": _fantasy(self.fantasy), "seed": _optional_whole("seed", self.seed), } + def run( + self, + fitness: Callable[[np.ndarray], Any], + *, + constraints: int = 0, + generations: int | None = None, + evaluations: int | None = None, + target: float | None = None, + time: float | None = None, + stagnation: int | None = None, + batch: bool = False, + parallel: bool = False, + on_generation: Callable[[Progress], bool | None] | None = None, + control: Callable[[Any, Progress], Any] | None = None, + checkpoint: str | os.PathLike[str] | None = None, + checkpoint_every: int | None = None, + resume: str | os.PathLike[str] | None = None, + ) -> Result: + """Runs the search until the first stop condition, as :meth:`Ga.run`, with the values of + ``constraints`` inequality constraints if there are any. + + Parameters + ---------- + fitness : callable or problems.Problem + As for :meth:`Ga.run`. With ``constraints`` above 0, it returns ``(value, g)``, + ``g`` an array of the constraints' values, feasible at 0 or below: a Gaussian process + models each. A problem of :mod:`genoxide.problems` with constraints gives their + values in Rust: leave ``constraints`` out. + constraints : int, default 0 + The number of inequality constraints whose values ``fitness`` returns. + + The other parameters are those of :meth:`Ga.run`; with constraints, ``batch`` is False: + the function is called a point at a time, in parallel with ``parallel=True``. + + Returns + ------- + Result + The best solution found by Deb's rules, and what the run took. + + Raises + ------ + ValueError + As :meth:`Ga.run`; for constraints with ``batch=True`` or a problem of + :mod:`genoxide.problems`, or a function that returns another number of constraint + values; and for the upper confidence bound with constraints. + TypeError + As :meth:`Ga.run`; and if a function with constraints doesn't return a pair. + """ + count = _whole("constraints", constraints) + if count == 0: + return _SingleObjective.run( + self, + fitness, + generations=generations, + evaluations=evaluations, + target=target, + time=time, + stagnation=stagnation, + batch=batch, + parallel=parallel, + on_generation=on_generation, + control=control, + checkpoint=checkpoint, + checkpoint_every=checkpoint_every, + resume=resume, + ) + if batch: + raise ValueError("constraints need batch=False: their values come a point at a time") + _check_callable(fitness) + if isinstance(fitness, (problems.Problem, problems.MultiProblem, problems.control.Balance)): + raise ValueError( + f"{type(fitness).__name__} is evaluated in Rust, with its own constraints: leave " + "constraints out" + ) + stop = _stop(generations, evaluations, target, time, stagnation) + callback = _on_generation(on_generation, Progress) + controls = _control(control, self) + saving = _checkpoints(checkpoint, checkpoint_every, resume) + return Result( + **self._run( + _with_values(fitness, count), + stop, + False, + parallel, + callback, + None, + controls, + saving, + constraints=count, + ) + ) + class _GradientMethod(_SingleObjective): """A gradient-based method: its ``run`` takes the gradient of the fitness function.""" diff --git a/python/genoxide/_genoxide.pyi b/python/genoxide/_genoxide.pyi index 6be5dc53..c55ab5c8 100644 --- a/python/genoxide/_genoxide.pyi +++ b/python/genoxide/_genoxide.pyi @@ -33,6 +33,7 @@ class Running: def reevaluate(self) -> None: ... def model(self) -> GaussianProcess | None: ... def acquisition(self, points: np.ndarray) -> np.ndarray: ... + def feasibility(self, points: np.ndarray) -> np.ndarray: ... class GaussianProcess: """A Gaussian process of ``genoxide::model::gp``, fitted from its description.""" diff --git a/python/src/config.rs b/python/src/config.rs index f0f204ce..5f979819 100644 --- a/python/src/config.rs +++ b/python/src/config.rs @@ -153,21 +153,7 @@ pub enum Algorithm { seed: Option, }, /// Bayesian optimization. - Bo { - /// The points of the initial design. - initial_points: Option, - /// Genomes evaluated first, in the initial design. - initial_genomes: Option>>, - acquisition: Option, - kernel: Option, - noise: Option, - output: Option, - /// The random points at which the acquisition is evaluated before its maximization. - raw_samples: Option, - acquisition_starts: Option, - hyperparameter_starts: Option, - seed: Option, - }, + Bo(Bo), NelderMead { coefficients: Option, /// The size of the first simplex, as a fraction of each gene's range. @@ -635,6 +621,41 @@ pub enum NoiseConfig { Learned { min: f64 }, } +/// Bayesian optimization's settings, for a Real or an Integer genome. +#[derive(Clone, Debug, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Bo { + /// The points of the initial design. + pub initial_points: Option, + /// Genomes evaluated first, in the initial design. + pub initial_genomes: Option>>, + pub acquisition: Option, + pub kernel: Option, + pub noise: Option, + pub output: Option, + /// The random points at which the acquisition is evaluated before its maximization. + pub raw_samples: Option, + pub acquisition_starts: Option, + pub hyperparameter_starts: Option, + /// The points of each generation after the initial design. + pub batch: Option, + pub fantasy: Option, + pub seed: Option, +} + +/// What Bayesian optimization takes a point not evaluated yet to be worth. +#[derive(Clone, Copy, Debug, Deserialize)] +pub enum BoFantasy { + #[serde(rename = "believer")] + Believer, + #[serde(rename = "liar-min")] + LiarMin, + #[serde(rename = "liar-mean")] + LiarMean, + #[serde(rename = "liar-max")] + LiarMax, +} + /// What the model of Bayesian optimization fits. #[derive(Clone, Copy, Debug, Deserialize)] #[serde(rename_all = "snake_case")] diff --git a/python/src/control.rs b/python/src/control.rs index 1a6284e5..efd9c9f6 100644 --- a/python/src/control.rs +++ b/python/src/control.rs @@ -48,6 +48,13 @@ pub trait Settings: Send + Sync + 'static { let _ = (algorithm, points); Err("the algorithm has no acquisition function".to_string()) } + + /// The probability of feasibility at `points`, a point per row, for the algorithms that + /// model constraints. + fn feasibility(&self, algorithm: &A, points: ArrayView2<'_, f64>) -> Result> { + let _ = (algorithm, points); + Err("the algorithm has no model of the constraints".to_string()) + } } fn unknown(name: &str) -> String { @@ -448,12 +455,12 @@ where } } -/// Bayesian optimization's acquisition function, which can change, its model and its -/// acquisition's values, for a plot. +/// Bayesian optimization's acquisition function, batch and fantasy, which can change, its model +/// and its acquisition's values and probability of feasibility, for a plot. pub struct BoSettings; -impl Settings for BoSettings { - fn get(&self, bo: &Bo, name: &str) -> Result { +impl Settings> for BoSettings { + fn get(&self, bo: &Bo, name: &str) -> Result { match name { "acquisition" => Ok(match bo.acquisition() { bo::Acquisition::ExpectedImprovement => json!({"type": "expected_improvement"}), @@ -473,49 +480,83 @@ impl Settings for BoSettings { } }), "initial_points" => Ok(json!(bo.initial_points())), + "batch" => Ok(json!(bo.batch())), + "fantasy" => Ok(json!(match bo.fantasy() { + bo::Fantasy::KrigingBeliever => "believer", + bo::Fantasy::ConstantLiar(bo::Lie::Min) => "liar-min", + bo::Fantasy::ConstantLiar(bo::Lie::Mean) => "liar-mean", + bo::Fantasy::ConstantLiar(bo::Lie::Max) => "liar-max", + fantasy => { + return Err(format!("a fantasy the package doesn't know: {fantasy:?}")); + } + })), + "constraints" => Ok(json!(bo.constraints())), _ => Err(unknown(name)), } } - fn set(&self, bo: &mut Bo, name: &str, value: &str) -> Result<()> { + fn set(&self, bo: &mut Bo, name: &str, value: &str) -> Result<()> { match name { "acquisition" => { let acquisition: config::Acquisition = parse(name, value)?; setting(bo.set_acquisition(crate::model::acquisition(acquisition))) } + "batch" => setting(bo.set_batch(parse(name, value)?)), + "fantasy" => { + let fantasy: config::BoFantasy = parse(name, value)?; + bo.set_fantasy(crate::run::bo_fantasy(fantasy)); + Ok(()) + } _ => Err(unknown(name)), } } - fn model(&self, bo: &Bo) -> Option { - let genes = bo.real().genome_len(); + fn model(&self, bo: &Bo) -> Option { + let genes = bo.representation().genome_len(); bo.model() .map(|model| PyGaussianProcess::new(model.clone(), genes)) } - fn acquisition(&self, bo: &Bo, points: ArrayView2<'_, f64>) -> Result> { - let genes = bo.real().genome_len(); - if points.ncols() != genes { - return Err(format!( - "points is a point per row, a value per gene of the genome's {genes}, not of shape ({}, {})", - points.nrows(), - points.ncols() - )); + fn acquisition(&self, bo: &Bo, points: ArrayView2<'_, f64>) -> Result> { + at_points(bo, points, |bo, point| bo.acquisition_at(point)) + } + + fn feasibility(&self, bo: &Bo, points: ArrayView2<'_, f64>) -> Result> { + at_points(bo, points, |bo, point| { + bo.probability_of_feasibility_at(point) + }) + } +} + +// `f` of the model at `points`, a point per row: an error without a model +fn at_points( + bo: &Bo, + points: ArrayView2<'_, f64>, + f: impl Fn(&Bo, &[f64]) -> Option, +) -> Result> { + let genes = bo.representation().genome_len(); + if points.ncols() != genes { + return Err(format!( + "points is a point per row, a value per gene of the genome's {genes}, not of shape ({}, {})", + points.nrows(), + points.ncols() + )); + } + let mut point = vec![0.0; genes]; + let mut values = Vec::with_capacity(points.nrows()); + for row in points.rows() { + for (x, &value) in point.iter_mut().zip(row.iter()) { + *x = value; } - let mut point = vec![0.0; genes]; - let mut values = Vec::with_capacity(points.nrows()); - for row in points.rows() { - for (x, &value) in point.iter_mut().zip(row.iter()) { - *x = value; - } - match bo.acquisition_at(&point) { - Some(value) => values.push(value), - None => return Err("no model yet: the first point after the initial design has its model" - .to_string()), + match f(bo, &point) { + Some(value) => values.push(value), + None => { + return Err("no model yet: the first point after the initial design has its model" + .to_string()); } } - Ok(values) } + Ok(values) } /// Where the algorithm is while the control runs, and its settings. @@ -570,6 +611,7 @@ trait AnySlot: Send + Sync { fn reevaluate(&self) -> PyResult<()>; fn model(&self) -> PyResult>; fn acquisition(&self, points: ArrayView2<'_, f64>) -> PyResult>; + fn feasibility(&self, points: ArrayView2<'_, f64>) -> PyResult>; } impl AnySlot for Slot @@ -597,6 +639,10 @@ where fn acquisition(&self, points: ArrayView2<'_, f64>) -> PyResult> { self.with(|algorithm, settings| settings.acquisition(algorithm, points)) } + + fn feasibility(&self, points: ArrayView2<'_, f64>) -> PyResult> { + self.with(|algorithm, settings| settings.feasibility(algorithm, points)) + } } /// The running algorithm, for a control: valid during the control's call only. The Python @@ -647,6 +693,16 @@ impl Running { let values = self.slot.acquisition(points.as_array())?; Ok(PyArray1::from_vec(py, values)) } + + /// The probability of feasibility at `points`, a point per row. + fn feasibility<'py>( + &self, + py: Python<'py>, + points: PyReadonlyArray2<'py, f64>, + ) -> PyResult>> { + let values = self.slot.feasibility(points.as_array())?; + Ok(PyArray1::from_vec(py, values)) + } } /// A continuation: the wrapped method's settings, and the current stage, read-only. diff --git a/python/src/fitness.rs b/python/src/fitness.rs index b854c907..437520ab 100644 --- a/python/src/fitness.rs +++ b/python/src/fitness.rs @@ -68,7 +68,8 @@ pub struct Shared { // what the genomes need to become Python objects context: GenomeContext, // the number of inequality constraints whose values and Jacobian a function with - // `gradient=True` returns after its gradient (MMA) + // `gradient=True` returns after its gradient (MMA), or whose values a function without a + // gradient returns after its value (Bayesian optimization) constraints: usize, } @@ -97,7 +98,8 @@ impl Shared { /// A function with `gradient=True` that returns `(value, gradient, g, jacobian)`: the values /// of `constraints` inequality constraints `gᵢ(x) <= 0` and their Jacobian after the gradient, - /// its score `(value, Σ max(0, gᵢ))`. Nothing for 0 constraints. + /// its score `(value, Σ max(0, gᵢ))`; without a gradient, `(value, g)`. Nothing for 0 + /// constraints. pub fn with_constraints(self, constraints: usize) -> Self { Self { constraints, @@ -323,7 +325,8 @@ impl Shared { match (&self.gradient, self.constraints) { (Gradient::Combined, 0) => value_without_gradient, (Gradient::Combined, _) => value_without_derivatives, - _ => value, + (_, 0) => value, + _ => value_without_values, } } @@ -421,6 +424,49 @@ fn value_without_derivatives(result: &Bound<'_, PyAny>) -> PyResult { with_constraints(result, None, None) } +// the score of `(value, g)`, `(value, Σ max(0, gᵢ))`: a function with constraints, without a +// gradient, called where their values aren't wanted +fn value_without_values(result: &Bound<'_, PyAny>) -> PyResult { + with_values(result, None, None) +} + +// what a function with constraints and without a gradient returns: `(value, g)`, `g` a float64 +// array the Python package makes. With `values`, the number of constraints is checked and they're +// written into it. The score is `(value, Σ max(0, gᵢ))`, as `genoxide::constraint::Constrained` +// gives it. +fn with_values( + result: &Bound<'_, PyAny>, + constraints: Option, + values: Option<&mut [f64]>, +) -> PyResult { + let pair = result + .cast::() + .ok() + .filter(|tuple| tuple.len() == 2) + .ok_or_else(|| { + PyTypeError::new_err(format!( + "with constraints, the fitness function returns a tuple (value, constraint values), not {}", + type_name(result) + )) + })?; + let value: f64 = pair.get_item(0)?.extract()?; + let g: PyReadonlyArray1<'_, f64> = pair.get_item(1)?.extract()?; + let g = g.as_array(); + if let Some(constraints) = constraints + && g.len() != constraints + { + return Err(PyValueError::new_err(format!( + "the fitness function returns {} constraint values, for {constraints} constraints", + g.len() + ))); + } + if let Some(values) = values { + copy("the constraint values", values, g)?; + } + let violation = g.iter().map(|&g| at_most(g, 0.0)).sum(); + Ok(Value::Constrained(value, violation)) +} + // what a function with `gradient=True` and constraints returns: `(value, gradient, g, jacobian)`, // float64 arrays the Python package makes (the Jacobian 2-D, a row per constraint). With // `extras`, the number of constraints is checked, and the derivatives and values it wants are @@ -728,9 +774,14 @@ impl FitnessFunction for Single<'_> { match self.problem { Some(Native::Real(problem)) => problem.provides(), Some(_) => Provided::NOTHING, - None if self.shared.constraints > 0 => Provided::GRADIENT - .with_inequalities(self.shared.constraints) - .with_constraint_jacobian(), + None if self.shared.constraints > 0 && self.shared.provides_gradient() => { + Provided::GRADIENT + .with_inequalities(self.shared.constraints) + .with_constraint_jacobian() + } + None if self.shared.constraints > 0 => { + Provided::NOTHING.with_inequalities(self.shared.constraints) + } None if self.shared.provides_gradient() => Provided::GRADIENT, None => Provided::NOTHING, } @@ -751,7 +802,7 @@ impl FitnessFunction for Single<'_> { None => {} } // the derivatives and constraints wanted, the gradient among them or not - if shared.constraints > 0 { + if shared.constraints > 0 && shared.provides_gradient() { return Python::attach(|py| { shared.call_genome(py, genome, |result| { with_constraints(result, Some(shared.constraints), Some(&mut *extras)) @@ -759,6 +810,14 @@ impl FitnessFunction for Single<'_> { }) .unwrap_or(Value::Invalid); } + if shared.constraints > 0 { + return Python::attach(|py| { + shared.call_genome(py, genome, |result| { + with_values(result, Some(shared.constraints), extras.inequalities()) + }) + }) + .unwrap_or(Value::Invalid); + } let Some(gradient) = extras.gradient() else { return FitnessFunction::::evaluate(self, genome); }; diff --git a/python/src/run.rs b/python/src/run.rs index 23d03e38..ae506b1e 100644 --- a/python/src/run.rs +++ b/python/src/run.rs @@ -152,17 +152,27 @@ pub fn run<'py>( |crossover| ListCrossover::new(crossover, "binary"), bit_flip, ), - config::Genome::Integer { bounds } => with_operators( - py, - genome_setting( + config::Genome::Integer { bounds } => { + let integer = genome_setting( Integer::new(bounds.iter().map(|&(low, high)| low..=high)), "Integer", - ), - run.algorithm, - &context, - |crossover| ListCrossover::new(crossover, "integer"), - integer_mutation, - ), + ); + match run.algorithm { + // Bayesian optimization on the integer lattice + config::Algorithm::Bo(settings) => (|| { + let bo = build_bo(integer?, settings, &context, integer_genome)?; + generational(py, bo, BoSettings, &context) + })(), + algorithm => with_operators( + py, + integer, + algorithm, + &context, + |crossover| ListCrossover::new(crossover, "integer"), + integer_mutation, + ), + } + } config::Genome::Real { bounds } => { let real = genome_setting( Real::new(bounds.iter().map(|&(low, high)| low..=high)), @@ -421,6 +431,104 @@ impl Context { } // differential evolution, as `de` describes it +// Bayesian optimization on `representation`, its initial genomes made by `genome` +fn build_bo( + representation: R, + settings: config::Bo, + context: &Context, + genome: fn(Vec) -> Result, +) -> std::result::Result, Failure> { + let config::Bo { + initial_points, + initial_genomes, + acquisition, + kernel, + noise, + output, + raw_samples, + acquisition_starts, + hyperparameter_starts, + batch, + fantasy, + seed, + } = settings; + let mut builder = Bo::builder(representation).objective(context.single_objective()?); + if let Some(points) = initial_points { + builder = builder.initial_points(points); + } + if let Some(genomes) = initial_genomes { + let genomes: Result> = genomes.into_iter().map(genome).collect(); + builder = builder.initial_genomes(genomes?); + } + if let Some(acquisition) = acquisition { + builder = builder.acquisition(crate::model::acquisition(acquisition)); + } + if let Some(kernel) = kernel { + builder = builder.kernel(crate::model::kernel(kernel)); + } + if let Some(noise) = noise { + builder = builder.noise(crate::model::noise(noise)); + } + if let Some(output) = output { + builder = builder.output(match output { + config::BoOutput::Standardize => bo::Output::Standardize, + config::BoOutput::Log => bo::Output::Log, + }); + } + if let Some(samples) = raw_samples { + builder = builder.raw_samples(samples); + } + if let Some(starts) = acquisition_starts { + builder = builder.acquisition_starts(starts); + } + if let Some(starts) = hyperparameter_starts { + builder = builder.hyperparameter_starts(starts); + } + if let Some(batch) = batch { + builder = builder.batch(batch); + } + if let Some(fantasy) = fantasy { + builder = builder.fantasy(bo_fantasy(fantasy)); + } + if let Some(seed) = seed { + builder = builder.seed(seed); + } + // the only genomes the builder checks are the initial ones + Ok(setting(builder.build().map_err(|error| match error { + genoxide::Error::InvalidGenome { reason } => genoxide::Error::InvalidSetting { + setting: "initial_genomes", + reason, + }, + error => error, + }))?) +} + +/// Bayesian optimization's fantasy of a configuration. +pub fn bo_fantasy(fantasy: config::BoFantasy) -> bo::Fantasy { + match fantasy { + config::BoFantasy::Believer => bo::Fantasy::KrigingBeliever, + config::BoFantasy::LiarMin => bo::Fantasy::ConstantLiar(bo::Lie::Min), + config::BoFantasy::LiarMean => bo::Fantasy::ConstantLiar(bo::Lie::Mean), + config::BoFantasy::LiarMax => bo::Fantasy::ConstantLiar(bo::Lie::Max), + } +} + +// an integer genome of whole numbers, for Bayesian optimization's initial genomes +fn integer_genome(genes: Vec) -> Result { + genes + .into_iter() + .map(|gene| { + if gene.fract() == 0.0 && gene.abs() <= 2f64.powi(53) { + Ok(gene as i64) + } else { + Err(format!( + "the initial genomes of an Integer genome are whole numbers, not {gene}" + )) + } + }) + .collect() +} + fn build_de(real: Real, de: config::De, context: &Context) -> std::result::Result { let config::De { population_size, @@ -850,60 +958,8 @@ fn real_algorithm<'py>( )?; generational(py, open_es, OpenEsSettings, context) } - config::Algorithm::Bo { - initial_points, - initial_genomes, - acquisition, - kernel, - noise, - output, - raw_samples, - acquisition_starts, - hyperparameter_starts, - seed, - } => { - let mut builder = Bo::builder(real).objective(context.single_objective()?); - if let Some(points) = initial_points { - builder = builder.initial_points(points); - } - if let Some(genomes) = initial_genomes { - builder = builder.initial_genomes(genomes.into_iter().map(Reals::from)); - } - if let Some(acquisition) = acquisition { - builder = builder.acquisition(crate::model::acquisition(acquisition)); - } - if let Some(kernel) = kernel { - builder = builder.kernel(crate::model::kernel(kernel)); - } - if let Some(noise) = noise { - builder = builder.noise(crate::model::noise(noise)); - } - if let Some(output) = output { - builder = builder.output(match output { - config::BoOutput::Standardize => bo::Output::Standardize, - config::BoOutput::Log => bo::Output::Log, - }); - } - if let Some(samples) = raw_samples { - builder = builder.raw_samples(samples); - } - if let Some(starts) = acquisition_starts { - builder = builder.acquisition_starts(starts); - } - if let Some(starts) = hyperparameter_starts { - builder = builder.hyperparameter_starts(starts); - } - if let Some(seed) = seed { - builder = builder.seed(seed); - } - // the only genomes the builder checks are the initial ones - let bo = setting(builder.build().map_err(|error| match error { - genoxide::Error::InvalidGenome { reason } => genoxide::Error::InvalidSetting { - setting: "initial_genomes", - reason, - }, - error => error, - }))?; + config::Algorithm::Bo(settings) => { + let bo = build_bo(real, settings, context, |genome| Ok(Reals::from(genome)))?; generational(py, bo, BoSettings, context) } config::Algorithm::NelderMead { @@ -1339,7 +1395,7 @@ where config::Algorithm::NelderMead { .. } => { Err("NelderMead needs a Real genome".to_string().into()) } - config::Algorithm::Bo { .. } => Err("Bo needs a Real genome".to_string().into()), + config::Algorithm::Bo(_) => Err("Bo needs a Real or an Integer genome".to_string().into()), config::Algorithm::Lbfgsb { .. } => Err("Lbfgsb needs a Real genome".to_string().into()), config::Algorithm::Mma { .. } => Err("Mma needs a Real genome".to_string().into()), config::Algorithm::Continuation { .. } => { diff --git a/python/tests/test_bo.py b/python/tests/test_bo.py index e7f2d96e..606be9f2 100644 --- a/python/tests/test_bo.py +++ b/python/tests/test_bo.py @@ -271,3 +271,136 @@ def test_wrong_model_settings_are_value_errors(arguments, message): model = gx.model.gp.GaussianProcess.fit(gx.Real((0, 1), length=2), [[0.1, 0.2]], [1.0]) with pytest.raises(ValueError, match="genes"): model.predict([[0.5]]) + + +# ---- batches, constraints and integer genomes ---------------------------------------------------- + + +def test_a_batch_runs_as_in_rust_and_the_same_in_parallel(): + # the bo_hartmann6 example: 4 points a round, 15 rounds to within 1e-4 + problem = gx.problems.Hartmann6() + bo = gx.Bo(problem.genome, batch=4, objective="minimize", seed=3) + target = problem.optimum.value + 1e-4 + result = bo.run(problem, target=target, evaluations=200, parallel=True) + assert (result.evaluations, result.generations) == (74, 15) + assert result.best_fitness == -3.322329435896455 + again = bo.run(problem, target=target, evaluations=200) + assert again.best_fitness == result.best_fitness + assert np.array_equal(again.best_genome, result.best_genome) + + +def test_the_fantasies_and_the_batch_change_during_a_run(): + problem = gx.problems.Branin() + seen = [] + + def control(running, progress): + seen.append((running.batch, running.fantasy, running.constraints)) + running.batch = 2 + running.fantasy = "liar-max" + + bo = gx.Bo(problem.genome, batch=3, fantasy="liar-mean", objective="minimize", seed=1) + result = bo.run(problem, generations=3, control=control) + assert seen[0] == (3, "liar-mean", 0) + assert seen[1] == (2, "liar-max", 0) + # the design of 6, then 2 a round: the control runs after generation 0 already + assert result.evaluations == 6 + 2 + 2 + 2 + for fantasy in ("believer", "liar-min", "liar-mean", "liar-max"): + bo = gx.Bo(problem.genome, batch=4, fantasy=fantasy, objective="minimize", seed=2) + assert bo.run(problem, generations=10).best_fitness < 0.397887 + 0.05 + + +def toy(x): + """Gramacy et al.'s (2016) toy problem: x1 + x2 and its two constraints' values.""" + wave = gx.math.sin(2.0 * math.pi * (x[0] * x[0] - 2.0 * x[1])) + return x[0] + x[1], np.array( + [1.5 - x[0] - 2.0 * x[1] - 0.5 * wave, x[0] * x[0] + x[1] * x[1] - 1.5] + ) + + +def test_constrained_bayesian_optimization_reaches_the_feasible_minimum(): + # the bo_constrained example: 21 evaluations to within 1e-5 of 0.5997880520100676 + minimum = 0.5997880520100676 + probabilities = [] + + def control(running, progress): + if running.model is not None: + points = progress.population + probabilities.append(running.probability_of_feasibility_at(points)) + assert running.constraints == 2 + + bo = gx.Bo(gx.Real((0.0, 1.0), length=2), objective="minimize", seed=1) + result = bo.run(toy, constraints=2, target=minimum + 1e-5, evaluations=60, control=control) + assert result.evaluations == 21 + assert result.best_fitness - minimum <= 1e-5 + assert np.all(toy(result.best_genome)[1] <= 0.0) + # at the points evaluated, about 0 or 1: the models interpolate the values + assert np.all((probabilities[-1] < 0.5) | (probabilities[-1] > 0.5)) + # in parallel, the same run + parallel = bo.run(toy, constraints=2, target=minimum + 1e-5, evaluations=60, parallel=True) + assert np.array_equal(parallel.best_genome, result.best_genome) + # a test problem gives its constraints' values in Rust + g24 = gx.problems.cec2006.G24() + bo = gx.Bo(g24.genome, objective="minimize", seed=1) + result = bo.run(g24, target=g24.optimum.value + 1e-4, evaluations=60) + assert result.stop_reason == "target" + + +def test_wrong_constraints_are_errors(): + bo = gx.Bo(gx.Real((0.0, 1.0), length=2), objective="minimize", seed=1) + with pytest.raises(ValueError, match="2 constraint values, for 3"): + bo.run(toy, constraints=3, evaluations=10) + with pytest.raises(TypeError, match="value, constraint values"): + bo.run(lambda x: float(x[0]), constraints=1, evaluations=10) + with pytest.raises(ValueError, match="batch=False"): + bo.run(toy, constraints=2, batch=True, evaluations=10) + with pytest.raises(ValueError, match="own constraints"): + bo.run(gx.problems.cec2006.G24(), constraints=2, evaluations=10) + ucb = gx.Bo(gx.Real((0.0, 1.0), length=2), acquisition=gx.UpperConfidenceBound(2.0)) + with pytest.raises(ValueError, match="upper confidence bound"): + ucb.run(toy, constraints=2, evaluations=10) + + +def test_integer_genes_are_searched_on_their_lattice(): + def quadratic(x): + return float((x[0] - 2.6) ** 2 + 2 * (x[1] + 1.3) ** 2 + (x[2] - 0.4) ** 2) + + integer = gx.Integer((-10, 10), length=3) + bo = gx.Bo(integer, initial_genomes=[[0, 0, 0]], objective="minimize", seed=1) + calls = [] + + def recorded(x): + calls.append(tuple(int(gene) for gene in x)) + return quadratic(x) + + # the integer minimum, (3, -1, 0): 0.16 + 0.18 + 0.16 + result = bo.run(recorded, target=0.5, evaluations=60) + assert result.stop_reason == "target" + assert list(result.best_genome) == [3, -1, 0] + assert calls[0] == (0, 0, 0) + assert len(set(calls)) == len(calls) + with pytest.raises(ValueError, match="whole numbers"): + gx.Bo(integer, initial_genomes=[[0.5, 0, 0]]).run(quadratic, evaluations=10) + + +def test_a_checkpoint_resumes_a_constrained_batch(tmp_path): + space = gx.Real((0.0, 1.0), length=2) + bo = gx.Bo(space, batch=2, objective="minimize", seed=4) + whole = bo.run(toy, constraints=2, evaluations=20) + path = tmp_path / "bo.ckpt" + bo.run(toy, constraints=2, evaluations=12, checkpoint=path, checkpoint_every=1) + resumed = bo.run(toy, constraints=2, evaluations=20, resume=path) + assert resumed.best_fitness == whole.best_fitness + assert np.array_equal(resumed.best_genome, whole.best_genome) + + +@pytest.mark.parametrize( + "settings, message", + [ + ({"batch": 0}, "batch"), + ({"fantasy": "liar"}, "fantasy"), + ], +) +def test_wrong_batch_settings_are_value_errors(settings, message): + bo = gx.Bo(gx.Real((0, 1), length=2), **settings) + with pytest.raises(ValueError, match=message): + bo.run(lambda x: float(np.sum(x)), evaluations=10) diff --git a/site/components/projects/genoxide/player/plots/SurrogatePlot.jsx b/site/components/projects/genoxide/player/plots/SurrogatePlot.jsx index 5b02070a..951450ff 100644 --- a/site/components/projects/genoxide/player/plots/SurrogatePlot.jsx +++ b/site/components/projects/genoxide/player/plots/SurrogatePlot.jsx @@ -17,7 +17,10 @@ const LEVELS = [0.12, 0.24, 0.36, 0.48, 0.6, 0.72, 0.84]; * (`state.polished`). Right, the acquisition function that chose the newest * point (`state.acquisition`, thousandths of its shaded range), darker where * the point is more worth evaluating. Generation 0, the initial design, has no - * model: the left panel shows the points alone. + * model: the left panel shows the points alone. `problem.panels`, if given, + * names what the two grids are (each `{ title, shading }`), e.g. a model's + * probability of feasibility on the left; `problem.minima_label` names the + * known minima. */ export default function SurrogatePlot({ trace, frame, dark }) { const problem = trace.problem ?? {}; @@ -26,6 +29,10 @@ export default function SurrogatePlot({ trace, frame, dark }) { [0, 1], ]; const minima = problem.minima ?? []; + const [left, right] = [ + { title: "The model's mean", shading: "shading: higher mean (log)", ...problem.panels?.[0] }, + { title: "The log expected improvement", shading: "shading: more worth evaluating", ...problem.panels?.[1] }, + ]; const state = frame.state ?? {}; const palette = categorical(dark); const legend = [ @@ -33,15 +40,15 @@ export default function SurrogatePlot({ trace, frame, dark }) { { label: "newest", color: palette[2], shape: "square" }, { label: "best", color: palette[0], shape: "diamond" }, ...(state.polished ? [{ label: "the mean's minimum", color: palette[3], shape: "triangle" }] : []), - ...(minima.length ? [{ label: "global minima", shape: "ring", className: "text-base-content" }] : []), + ...(minima.length ? [{ label: problem.minima_label ?? "global minima", shape: "ring", className: "text-base-content" }] : []), ]; return (
Optimization for Rust and Python, in one library: genetic algorithms, evolution strategies, CMA-ES, differential evolution, particle swarms, local search, Nelder-Mead, genetic programming, neuroevolution, multi-objective optimization (NSGA-II, NSGA-III, MOEA/D, SPEA2, SMS-EMOA) gradient-based methods (L-BFGS-B, Adam and momentum, MMA and GCMMA, and continuation in stages) and Bayesian optimization with Gaussian processes, for expensive functions. A seed gives the same results, to the bit, on every platform and thread count. Invalid settings are errors, not panics. Each method is checked against its source paper. +> Optimization for Rust and Python, in one library: genetic algorithms, evolution strategies, CMA-ES, differential evolution, particle swarms, local search, Nelder-Mead, genetic programming, neuroevolution, multi-objective optimization (NSGA-II, NSGA-III, MOEA/D, SPEA2, SMS-EMOA) gradient-based methods (L-BFGS-B, Adam and momentum, MMA and GCMMA, and continuation in stages) and Bayesian optimization with Gaussian processes, for expensive functions (in batches or asynchronously, with constraints, on real or integer genes). A seed gives the same results, to the bit, on every platform and thread count. Invalid settings are errors, not panics. Each method is checked against its source paper. Install: \`cargo add genoxide\` (Rust), \`pip install genoxide\` (Python), or \`cargo install genoxide --features cli\` for a program that runs an optimization described in TOML with any program as the fitness function. genoxide is pre-1.0: check the version and use the guide below, which matches the latest release. diff --git a/src/algorithm/bo.rs b/src/algorithm/bo.rs index 47e80746..89ebb974 100644 --- a/src/algorithm/bo.rs +++ b/src/algorithm/bo.rs @@ -5,9 +5,13 @@ //! standard deviation at a point: they work with any surrogate model that gives those. pub mod acquisition; +mod space; -use super::{Algorithm, Candidates, Lbfgsb, Reevaluate}; -use crate::genome::{Real, Reals, Representation}; +pub use space::Space; + +use super::{Algorithm, Candidates, Incremental, Lbfgsb, Reevaluate}; +use crate::engine::{Evaluations, Provided, Wanted}; +use crate::genome::{Real, Reals}; use crate::model::gp::{self, GaussianProcess, Kernel, Noise, Scaling, map_in_order}; use crate::{Error, Fitness, Individual, Objective, Population, Result, StreamRng}; use rand::Rng; @@ -38,7 +42,8 @@ pub enum Acquisition { }, /// The confidence bound `μ − √β σ`, minimized (`μ + √β σ` maximized when maximizing): /// Srinivas, Krause, Kakade and Seeger (2010). Changeable during a run for a schedule - /// ([`Bo::set_acquisition`]). + /// ([`Bo::set_acquisition`]). Not for a problem with constraints, whose acquisition is + /// weighed by the probability of feasibility: a bound can be negative. UpperConfidenceBound { /// β ≥ 0: 0 exploits the model's mean alone, larger explores more. beta: f64, @@ -62,6 +67,22 @@ impl Acquisition { _ => Ok(()), } } + + // the error for an acquisition that can't be weighed by the probability of feasibility + fn check_constrained(self, constraints: usize) -> Result<()> { + if constraints > 0 && matches!(self, Acquisition::UpperConfidenceBound { .. }) { + return Err(Error::InvalidSetting { + setting: "acquisition", + reason: format!( + "the fitness function gives {constraints} constraints, and Bayesian \ + optimization with constraints weighs the expected improvement (or its \ + logarithm, or the probability of improvement) by the probability of \ + feasibility: the upper confidence bound can't be weighed so" + ), + }); + } + Ok(()) + } } /// What the model of a [`Bo`] fits: a transform of the values to minimize (the scores, negated @@ -90,6 +111,58 @@ pub enum Output { Log, } +/// What a [`Bo`] takes a point to be worth while it isn't evaluated yet: the points of a batch +/// already chosen, as it chooses the next one ([`BoBuilder::batch`]), and under an +/// [`AsyncEngine`](crate::engine::AsyncEngine) the points still being evaluated. The heuristics of +/// Ginsbourger, Le Riche and Carraro (2010, section 4.2): each such point is added to the model +/// with a fantasized value, as if it had been observed, without fitting the hyperparameters again +/// (the constant mean is estimated again, as at every fit). The model is then sure of itself at +/// the point, so the acquisition function looks elsewhere, and how far elsewhere depends on the +/// value. +/// +/// With constraints, each constraint's model takes its posterior mean at the point, whatever the +/// fantasy of the objective. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +#[non_exhaustive] +#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] +pub enum Fantasy { + /// The Kriging believer, the default: the model's posterior mean at the point, which leaves + /// the mean where it was and only removes the uncertainty there (Ginsbourger et al.'s + /// Algorithm 1). A point predicted below the best value lowers the best, and the next points + /// then tend to cluster around it. + /// + /// Measured with batches of 4 against the constant liars over 20 seeds, to f* + 1e-3: Branin + /// within 80 evaluations in 20 runs, a median of 34 evaluations (the lowest lie 34, the mean + /// 42, the highest 54); Hartmann 3 in 20, a median of 30 (30, 40, 40); Hartmann 6 within 200 in + /// 13 (13, 12, 11), the others in the local minimum −3.2032. Under an + /// [`AsyncEngine`](crate::engine::AsyncEngine) with 4 workers, whose every proposal has 3 + /// points fantasized, Hartmann 3 to f* + 1e-4 within 120 evaluations in all 5 runs tried, from + /// 39 to 76 evaluations, against 2 of 5 with the lowest lie, which keeps the region of the + /// points being evaluated as good as the best and so pushes the search away from it. + #[default] + KrigingBeliever, + /// The constant liar: the same value, a [`Lie`], at every such point (Ginsbourger et al.'s + /// Algorithm 2). The higher the lie, the farther the next points go from the earlier ones. + ConstantLiar(Lie), +} + +/// The value of a [`Fantasy::ConstantLiar`]: a statistic of the values the model fits (after the +/// [`Output`] transform), of every point evaluated so far. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +#[non_exhaustive] +#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] +pub enum Lie { + /// The lowest, the best: a soft repulsion, the next points near but not at the earlier ones; + /// on Branin, Ginsbourger et al.'s best strategy of the four, which visited the three minima's + /// regions in 6 points. + #[default] + Min, + /// The mean: the next points spread over the box. + Mean, + /// The highest, the worst: the strongest repulsion, the most exploration. + Max, +} + // the ids of the random streams derived from the seed: they decide the results of seeded runs and // must never change for the same major version mod streams { @@ -97,23 +170,38 @@ mod streams { pub(super) const DESIGN: u64 = 0; // the starts of the hyperparameters' maximization, per generation pub(super) const HYPERPARAMETERS: u64 = 1; - // the raw samples of the acquisition's maximization, per generation + // the raw samples of the acquisition's maximization, per generation (then per point of a + // batch after the first) pub(super) const RAW_SAMPLES: u64 = 2; - // a random point when there's no model to ask, per generation + // a random point when there's no model to ask, per generation (then per point of a batch + // after the first) pub(super) const RANDOM: u64 = 3; + // the starts of the constraints' models' hyperparameters, per generation, then per constraint + pub(super) const CONSTRAINTS: u64 = 4; + // the streams of a proposal to an asynchronous engine, per proposal, then the kinds above + pub(super) const PROPOSALS: u64 = 5; +} + +// when the random numbers of a choice are drawn: a generation of an ask, or a proposal +#[derive(Clone, Copy, Debug)] +enum Step { + Generation(u64), + Proposal(u64), } // the evaluations of the acquisition function per start of its maximization const ACQUISITION_EVALUATIONS: u64 = 200; +// the steps of a hill climb on an integer lattice, at most +const LATTICE_STEPS: usize = 10_000; // the least variance of the model's prediction, in its standardized units, for the acquisition const VARIANCE_FLOOR: f64 = 1e-12; -/// Bayesian optimization on [`Real`] genomes, as an ask / tell [`Algorithm`]: for expensive -/// black-box functions, such as a simulation that runs for minutes or a physical experiment, where -/// tens to a few hundred evaluations must do. A Gaussian process ([`model::gp`](crate::model::gp)) -/// models the function from every evaluation so far, and an [`Acquisition`] function of its -/// posterior picks the next point to evaluate, trading the model's best guesses against its -/// uncertainty. +/// Bayesian optimization on [`Real`] genomes (and [`Integer`](crate::genome::Integer) ones, see +/// [`Space`]), as an ask / tell [`Algorithm`] and an [`Incremental`] one: for expensive black-box +/// functions, such as a simulation that runs for minutes or a physical experiment, where tens to +/// a few hundred evaluations must do. A Gaussian process ([`model::gp`](crate::model::gp)) models +/// the function from every evaluation so far, and an [`Acquisition`] function of its posterior +/// picks the next points to evaluate, trading the model's best guesses against its uncertainty. /// /// - **The initial design** (generation 0): [`initial_points`](BoBuilder::initial_points) points, /// 2(n + 1) for n searched genes by default, the @@ -121,22 +209,41 @@ const VARIANCE_FLOOR: f64 = 1e-12; /// Beckman and Conover, 1979; [`Real::latin_hypercube`]) for the rest. A small design leaves /// most of the budget to the model's choices. For an accurate model of the whole box rather /// than its minimum, Loeppky, Sacks and Welch (2009) recommend 10n points. -/// - **Each later generation** asks one point: the model is fitted to every evaluation (its -/// hyperparameters by maximum likelihood, from the last fit's and random starts), then the -/// acquisition function is maximized in the box: evaluated at -/// [`raw_samples`](BoBuilder::raw_samples) random points, then improved by [`Lbfgsb`] with its -/// analytic gradient from the best [`acquisition_starts`](BoBuilder::acquisition_starts) of them -/// and from the best point evaluated so far. The best result that isn't an evaluated point is -/// asked: a point is never asked twice. +/// - **Each later generation** asks a [batch](BoBuilder::batch) of q points, one by default: the +/// model is fitted to every evaluation (its hyperparameters by maximum likelihood, from the +/// last fit's and random starts), then the acquisition function is maximized in the box: +/// evaluated at [`raw_samples`](BoBuilder::raw_samples) random points, then improved by +/// [`Lbfgsb`] with its analytic gradient from the best +/// [`acquisition_starts`](BoBuilder::acquisition_starts) of them and from the best point +/// evaluated so far. The best result that isn't an evaluated point is asked: a point is never +/// asked twice. Each further point of a batch is chosen the same way after the points before it +/// are added to the model with a [`Fantasy`] value, the hyperparameters kept: the q +/// evaluations of a generation can then run at once +/// ([`Engine::parallel`](crate::Engine::parallel)). +/// - **Asynchronous evaluation**: under an [`AsyncEngine`](crate::engine::AsyncEngine), the +/// design is proposed first, then each proposal is a point chosen by the model of the results +/// so far, with the points still being evaluated added with a [`Fantasy`] value as in a batch; +/// until a result arrives, random points. A generation of the engine is +/// [`initial_points`](Bo::initial_points) evaluations. With one worker, a seed gives the same +/// run every time. /// - **The model** fits the values to minimize, the scores negated when maximizing, through the /// [`Output`] transform. A point with an invalid fitness, or a score that isn't finite, enters /// the model at the worst value of the others, so the search learns to avoid where the function -/// fails; until a point has a valid score, the next point is random. The search uses the score -/// only: a constraint violation is ignored by it (use a penalty), though -/// [`best`](Algorithm::best) compares by Deb's rules, as everywhere in genoxide. -/// - **Cost.** The model's fit is O(N³) for N evaluations, its predictions O(N²): Bayesian -/// optimization suits up to a few hundred evaluations of a function that costs far more than -/// that, in up to about 10 to 20 genes. +/// fails; until a point has a valid score, the next point is random. +/// - **Constraints** `gᵢ(x) ≤ 0` whose values the fitness function gives one by one, such as a +/// [`Constrained`](crate::constraint::Constrained) one or a test problem with +/// [`constraints`](crate::problems::Problem::constraints): a Gaussian process models each, and +/// the acquisition function is weighed by the probability that a point is feasible, `Π P(gᵢ ≤ 0)` +/// under the constraints' models, taken as independent: the expected constrained improvement of +/// Gardner, Kusner, Xu, Weinberger and Cunningham (2014), the improvement being over the best +/// feasible point (with the log expected improvement, its logarithm plus the probability's). +/// Until a feasible point is evaluated, the search maximizes the probability of feasibility +/// alone. Without constraint values, the search uses the score only, a violation being ignored +/// by it (use a penalty), though [`best`](Algorithm::best) compares by Deb's rules, as everywhere +/// in genoxide. +/// - **Cost.** The model's fit is O(N³) for N evaluations, its predictions O(N²), for the +/// objective and each constraint: Bayesian optimization suits up to a few hundred evaluations of +/// a function that costs far more than that, in up to about 10 to 20 genes. /// /// [`model`](Bo::model) gives the model that chose the last point. Every random number comes from /// streams derived from the [seed](BoBuilder::seed), every operation is a sum, product, quotient @@ -144,7 +251,8 @@ const VARIANCE_FLOOR: f64 = 1e-12; /// run on rayon (with the `parallel` feature) with the winner chosen by value, then start: a seed /// gives the same run on every platform and thread count. /// -/// Built with [`Bo::builder`], run with an [`Engine`](crate::Engine). +/// Built with [`Bo::builder`], run with an [`Engine`](crate::Engine) or an +/// [`AsyncEngine`](crate::engine::AsyncEngine). /// /// ``` /// use genoxide::prelude::*; @@ -156,6 +264,14 @@ const VARIANCE_FLOOR: f64 = 1e-12; /// .stop_when(Stop::evaluations(40)) /// .run()?; /// assert!(outcome.best_fitness().score().unwrap() < 0.397887 + 1e-3); +/// +/// // 4 points at a time (evaluated in parallel with `Engine::parallel`): 10 rounds +/// let bo = Bo::builder(Branin.representation()).batch(4).minimize().seed(1).build()?; +/// let outcome = Engine::new(bo, Branin) +/// .stop_when(Stop::evaluations(46)) +/// .run()?; +/// assert_eq!(outcome.generations(), 10); +/// assert!(outcome.best_fitness().score().unwrap() < 0.397887 + 1e-2); /// # Ok::<(), genoxide::Error>(()) /// ``` /// @@ -169,11 +285,22 @@ const VARIANCE_FLOOR: f64 = 1e-12; /// Conover, W. J. (1979). A comparison of three methods for selecting values of input variables /// in the analysis of output from a computer code. *Technometrics* 21(2): 239-245. Loeppky, J. L., /// Sacks, J. and Welch, W. J. (2009). Choosing the sample size of a computer experiment: a -/// practical guide. *Technometrics* 51(4): 366-376. +/// practical guide. *Technometrics* 51(4): 366-376. Ginsbourger, D., Le Riche, R. and Carraro, L. +/// (2010). Kriging is well-suited to parallelize optimization. In *Computational Intelligence in +/// Expensive Optimization Problems*, Springer: 131-162. Gardner, J. R., Kusner, M. J., Xu, Z., +/// Weinberger, K. Q. and Cunningham, J. P. (2014). Bayesian optimization with inequality +/// constraints. *ICML 2014*, PMLR 32(2): 937-945. #[derive(Clone, Debug)] #[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] -pub struct Bo { - real: Real, +#[cfg_attr( + feature = "serde", + serde(bound( + serialize = "R: serde::Serialize, R::Genome: serde::Serialize", + deserialize = "R: serde::Deserialize<'de>, R::Genome: serde::Deserialize<'de>" + )) +)] +pub struct Bo { + representation: R, initial_points: usize, acquisition: Acquisition, kernel: Kernel, @@ -182,36 +309,60 @@ pub struct Bo { raw_samples: usize, acquisition_starts: usize, hyperparameter_starts: usize, + batch: usize, + fantasy: Fantasy, objective: Objective, seed: u64, // the initial design, asked in generation 0 - design: Vec, + design: Vec, // every evaluated point, in the order evaluated - observations: Population, - // the logarithms of the last fit's hyperparameters, the next fit's first start + observations: Population, + // the number of inequality constraints whose values the fitness function gives, once known, + // and their values at each evaluated point, a row per point (NaN for an invalid fitness) + constraints: Option, + constraint_values: Vec, + // the logarithms of the last fit's hyperparameters, the next fit's first start, for the + // objective's model and each constraint's warm: Option>, + constraint_warm: Vec>>, // the points of the current ask - pending: Vec>, + pending: Vec>, indices: Vec, asked: bool, reevaluating: bool, generation: u64, evaluations: u64, - best: Option>, + best: Option>, best_generation: u64, - // the model that chose the last point, and the best value it was told, in its standardized - // units - #[cfg_attr(feature = "serde", serde(skip))] - model: Option, + best_evaluation: u64, + // under an asynchronous engine: the points proposed and not received, in order; those to + // propose again first, after a checkpoint; the design points proposed; and the proposals + proposed: Vec, + requeued: Vec, + design_proposed: usize, + proposals: u64, + // the models that chose the last point, fitted to the evaluations before it #[cfg_attr(feature = "serde", serde(skip))] - model_best: f64, + surrogate: Option, +} + +// the models of a step, in their standardized units +#[derive(Clone, Debug)] +struct Surrogate { + objective: GaussianProcess, + constraints: Vec, + // each constraint's bound 0, standardized + thresholds: Vec, + // the best value of the feasible points, standardized; None without one + best: Option, } impl Bo { - /// A builder for Bayesian optimization on `real`. - pub fn builder(real: Real) -> BoBuilder { + /// A builder for Bayesian optimization on `representation`, a [`Real`] or an + /// [`Integer`](crate::genome::Integer) (see [`Space`]). + pub fn builder(representation: R) -> BoBuilder { BoBuilder { - real, + representation, initial_points: None, initial_genomes: Vec::new(), acquisition: Acquisition::default(), @@ -221,14 +372,42 @@ impl Bo { raw_samples: 1000, acquisition_starts: 10, hyperparameter_starts: 5, + batch: 1, + fantasy: Fantasy::default(), objective: Objective::default(), seed: None, } } +} +impl Bo { /// The representation. - pub fn real(&self) -> &Real { - &self.real + pub fn representation(&self) -> &R { + &self.representation + } + + /// Whether higher or lower fitness is better (as [`Algorithm::objective`] and + /// [`Incremental::objective`]). + pub fn objective(&self) -> Objective { + self.objective + } + + /// Every evaluated point, in the order evaluated (as [`Algorithm::population`] and + /// [`Incremental::population`]). + pub fn population(&self) -> &Population { + &self.observations + } + + /// The best point evaluated so far, by Deb's rules (as [`Algorithm::best`] and + /// [`Incremental::best`]). + pub fn best(&self) -> Option<&Individual> { + self.best.as_ref() + } + + /// The number of fitness values told or received so far (as [`Algorithm::evaluations`] and + /// [`Incremental::evaluations`]). + pub fn evaluations(&self) -> u64 { + self.evaluations } /// The number of points of the initial design. @@ -246,13 +425,41 @@ impl Bo { /// /// # Errors /// - /// [`Error::InvalidSetting`] for a negative or non-finite ξ or β. Nothing changes on errors. + /// [`Error::InvalidSetting`] for a negative or non-finite ξ or β, or the upper confidence + /// bound for a problem with constraints. Nothing changes on errors. pub fn set_acquisition(&mut self, acquisition: Acquisition) -> Result<()> { acquisition.validate()?; + acquisition.check_constrained(self.constraints.unwrap_or(0))?; self.acquisition = acquisition; Ok(()) } + /// The number of points of each generation after the initial design. + pub fn batch(&self) -> usize { + self.batch + } + + /// Changes the number of points of each generation from the next ask on. + /// + /// # Errors + /// + /// [`Error::InvalidSetting`] for 0 or more than 2^24. Nothing changes on errors. + pub fn set_batch(&mut self, batch: usize) -> Result<()> { + check_batch(batch)?; + self.batch = batch; + Ok(()) + } + + /// What the points of a batch, and those still being evaluated, are taken to be worth. + pub fn fantasy(&self) -> Fantasy { + self.fantasy + } + + /// Changes the [`Fantasy`] from the next choice on. + pub fn set_fantasy(&mut self, fantasy: Fantasy) { + self.fantasy = fantasy; + } + /// The model's kernel. pub fn kernel(&self) -> Kernel { self.kernel @@ -288,43 +495,116 @@ impl Bo { self.seed } - /// The Gaussian process that chose the last point asked, fitted to the evaluations before it: - /// a model of the values the search minimizes (the scores, negated when maximizing, through - /// the [`Output`] transform). `None` before the first point after the initial design, after a - /// random point (when no evaluation had a valid score), and after loading a checkpoint until - /// the next point is asked. - pub fn model(&self) -> Option<&GaussianProcess> { - self.model.as_ref() + /// The number of inequality constraints whose values the fitness function gives: `None` + /// before a run has started, and 0 without constraints. + pub fn constraints(&self) -> Option { + self.constraints } - /// The acquisition function at `genome` under [`model`](Bo::model), as the search maximizes - /// it: the log expected improvement, the expected improvement, the logarithm of the - /// probability of improvement, or the negated lower confidence bound, in the model's - /// standardized units. `None` without a model. + /// The values of the inequality constraints `gᵢ(x) ≤ 0` at the evaluated point `index` of the + /// [population](Algorithm::population), in order: empty without constraints, NaN for a point + /// with an invalid fitness. /// /// # Panics /// - /// If `genome` doesn't have a value per gene. - pub fn acquisition_at(&self, genome: &[f64]) -> Option { - let model = self.model.as_ref()?; + /// If `index` is out of bounds. + pub fn constraint_values(&self, index: usize) -> &[f64] { + assert!( + index < self.observations.len(), + "point {index} isn't evaluated" + ); + let m = self.constraints.unwrap_or(0); + &self.constraint_values[index * m..(index + 1) * m] + } + + /// The points proposed to an [`AsyncEngine`](crate::engine::AsyncEngine) whose results + /// haven't arrived, in the order proposed. + pub fn proposed(&self) -> &[R::Genome] { + &self.proposed + } + + /// The Gaussian process that chose the last point asked, fitted to the evaluations before it + /// (without the [fantasies](Fantasy) of a batch or of pending points): a model of the values + /// the search minimizes (the scores, negated when maximizing, through the [`Output`] + /// transform). `None` before the first point after the initial design, after a random point + /// (when no evaluation had a valid score), and after loading a checkpoint until the next point + /// is asked. + pub fn model(&self) -> Option<&GaussianProcess> { + self.surrogate + .as_ref() + .map(|surrogate| &surrogate.objective) + } + + /// The Gaussian processes of the constraints that chose the last point, one per constraint, + /// fitted as [`model`](Bo::model): empty without constraints or without a model. + pub fn constraint_models(&self) -> &[GaussianProcess] { + self.surrogate + .as_ref() + .map_or(&[], |surrogate| &surrogate.constraints[..]) + } + + // the unit-cube point of gene values, on the lattice for integers + fn unit_of(&self, genome: &[f64]) -> Vec { assert_eq!( genome.len(), - self.real.genome_len(), + self.representation.genome_len(), "a genome of {} genes", - self.real.genome_len() + self.representation.genome_len() ); - let scaling = Scaling::new(&self.real); + let scaling = self.representation.scaling(); let mut unit = vec![0.0; scaling.dims()]; scaling.to_unit(genome, &mut unit); - Some(self.search(model).value(&unit, None)) + if self.representation.lattice().is_some() { + let nearest = self.representation.genome_at(&scaling, &unit); + R::to_unit(&scaling, &nearest, &mut unit); + } + unit } - fn search<'a>(&self, model: &'a GaussianProcess) -> Search<'a> { + /// The acquisition function at the gene values `genome` (rounded to the nearest integers for + /// an [`Integer`](crate::genome::Integer) genome) under [`model`](Bo::model), as the search + /// maximizes it: the log expected improvement, the expected improvement, the logarithm of the + /// probability of improvement, or the negated lower confidence bound, in the model's + /// standardized units; with constraints, weighed by the probability of feasibility (its + /// logarithm added to a logarithm), or the logarithm of that probability alone before a + /// feasible point is evaluated. `None` without a model. + /// + /// # Panics + /// + /// If `genome` doesn't have a value per gene. + pub fn acquisition_at(&self, genome: &[f64]) -> Option { + let surrogate = self.surrogate.as_ref()?; + let unit = self.unit_of(genome); + Some(self.search(surrogate).value(&unit, None)) + } + + /// The probability that the gene values `genome` (rounded for an + /// [`Integer`](crate::genome::Integer) genome) are feasible, under the + /// [constraints' models](Bo::constraint_models): `Π P(gᵢ ≤ 0)`. 1 without constraints, `None` + /// without a model. + /// + /// # Panics + /// + /// If `genome` doesn't have a value per gene. + pub fn probability_of_feasibility_at(&self, genome: &[f64]) -> Option { + let surrogate = self.surrogate.as_ref()?; + let unit = self.unit_of(genome); + let mut log = 0.0; + for (model, &threshold) in surrogate.constraints.iter().zip(&surrogate.thresholds) { + let (mean, variance) = model.predict_unit(&unit, None); + let sd = variance.max(VARIANCE_FLOOR).sqrt(); + log += + acquisition::log_probability_of_improvement_derivatives(mean, sd, threshold, 0.0) + [0]; + } + Some(crate::math::exp(log)) + } + + fn search<'a>(&self, surrogate: &'a Surrogate) -> Search<'a> { Search { - model, - best: self.model_best, + surrogate, acquisition: self.acquisition, - scale: model.standardization().1, + scale: surrogate.objective.standardization().1, } } @@ -340,71 +620,252 @@ impl Bo { } // whether `genome` was evaluated already - fn observed(&self, genome: &[f64]) -> bool { + fn observed(&self, genome: &R::Genome) -> bool { self.observations .iter() - .any(|individual| individual.genome()[..] == genome[..]) + .any(|individual| individual.genome() == genome) } - // a random point of the box that wasn't evaluated, from the stream of this generation - fn random_point(&self, generation: u64) -> Reals { - let mut rng = StreamRng::seed_from_u64(self.seed) - .derive(streams::RANDOM) - .derive(generation); - loop { - let genome = self.real.random_genome(&mut rng); - if !self.observed(&genome) { - return genome; + // whether `genome` was evaluated, or is among `others` + fn taken(&self, genome: &R::Genome, others: &[&[R::Genome]]) -> bool { + self.observed(genome) || others.iter().any(|list| list.contains(genome)) + } + + // the random stream of `kind` for `step`, and its point `pick` of a batch + fn stream(&self, kind: u64, step: Step, pick: usize) -> StreamRng { + let root = StreamRng::seed_from_u64(self.seed); + let rng = match step { + Step::Generation(generation) => root.derive(kind).derive(generation), + Step::Proposal(proposal) => root + .derive(streams::PROPOSALS) + .derive(proposal) + .derive(kind), + }; + if pick == 0 { + rng + } else { + rng.derive(pick as u64) + } + } + + // a random point of the box that wasn't evaluated and isn't among `others`, from the stream of + // this step and pick; None once every point of a lattice is taken + fn random_point(&self, step: Step, pick: usize, others: &[&[R::Genome]]) -> Option { + let mut rng = self.stream(streams::RANDOM, step, pick); + // a few draws, then, on a lattice, its first point not taken + for _ in 0..1000 { + let genome = self.representation.random_genome(&mut rng); + if !self.taken(&genome, others) { + return Some(genome); } } + let taken = self.observations.len() + others.iter().map(|list| list.len()).sum::(); + match self.representation.lattice() { + None => loop { + let genome = self.representation.random_genome(&mut rng); + if !self.taken(&genome, others) { + return Some(genome); + } + }, + Some(points) if points > taken as u128 && points <= 1 << 24 => self + .representation + .enumerate(1 << 24)? + .into_iter() + .find(|genome| !self.taken(genome, others)), + Some(_) => None, + } } - // the point to evaluate next: the model fitted, the acquisition maximized - fn next_point(&mut self) -> Reals { - let generation = self.generation + 1; - let root = StreamRng::seed_from_u64(self.seed); - let scaling = Scaling::new(&self.real); + // the models of the evaluations, and the unit-cube point to start the acquisition's + // maximization from (the best point evaluated) and the values of the lies, standardized: min, + // mean, max; None if there's no valid value, or a model doesn't factor + fn fit(&mut self, step: Step) -> Option<(Surrogate, Vec, [f64; 3])> { + let scaling = self.representation.scaling(); let dims = scaling.dims(); let count = self.observations.len(); let mut x = vec![0.0; count * dims]; let mut values = Vec::with_capacity(count); for (individual, unit) in self.observations.iter().zip(x.chunks_exact_mut(dims)) { - scaling.to_unit(individual.genome(), unit); + R::to_unit(&scaling, individual.genome(), unit); values.push(self.model_value(individual.fitness())); } - self.model = None; - let Some(targets) = transform(&values, self.output) else { - return self.random_point(generation); - }; - let settings = gp::Settings { + let targets = transform(&values, self.output)?; + let m = self.constraints.unwrap_or(0); + let settings = |seed| gp::Settings { kernel: self.kernel, noise: self.noise, starts: self.hyperparameter_starts, - seed: root - .derive(streams::HYPERPARAMETERS) - .derive(generation) - .next_u64(), + seed, }; - let fitted = - GaussianProcess::fit_unit(settings, scaling.clone(), x, &targets, self.warm.as_deref()); + let seed = self.stream(streams::HYPERPARAMETERS, step, 0).next_u64(); + let points = if m > 0 { x.clone() } else { Vec::new() }; + let fitted = GaussianProcess::fit_unit( + settings(seed), + scaling.clone(), + x, + &targets, + self.warm.as_deref(), + ); // a kernel matrix that doesn't factor, even with jitter: a random point instead - let Ok(model) = fitted else { - return self.random_point(generation); - }; + let model = fitted.ok()?; self.warm = Some(model.log_parameters().to_vec()); - // the best point evaluated, by the model's values: the incumbent, and a start + // the constraints' models, of their values standardized, the invalid ones at the worst + let mut constraints = Vec::with_capacity(m); + let mut thresholds = Vec::with_capacity(m); + let constraint_seeds = self.stream(streams::CONSTRAINTS, step, 0); + for i in 0..m { + let column: Vec> = (0..count) + .map(|o| { + let g = self.constraint_values[o * m + i]; + g.is_finite().then_some(g) + }) + .collect(); + let targets = transform(&column, Output::Standardize)?; + let seed = constraint_seeds.derive(i as u64).next_u64(); + let warm = self.constraint_warm[i].as_deref(); + let fitted = GaussianProcess::fit_unit( + settings(seed), + scaling.clone(), + points.clone(), + &targets, + warm, + ); + let constraint = fitted.ok()?; + self.constraint_warm[i] = Some(constraint.log_parameters().to_vec()); + let (mean, scale) = constraint.standardization(); + thresholds.push((0.0 - mean) / scale); + constraints.push(constraint); + } + // the best point evaluated, by the model's values (of the feasible points, with + // constraints): the incumbent, and a start let mut incumbent: Option = None; for (index, value) in values.iter().enumerate() { - if value.is_some() && incumbent.is_none_or(|best| targets[index] < targets[best]) { + let feasible = m == 0 + || self.observations.as_slice()[index] + .fitness() + .is_some_and(Fitness::is_feasible); + if value.is_some() + && feasible + && incumbent.is_none_or(|best| targets[index] < targets[best]) + { incumbent = Some(index); } } - let incumbent = incumbent.unwrap_or(0); let (y_mean, y_scale) = model.standardization(); - self.model_best = (targets[incumbent] - y_mean) / y_scale; - let search = self.search(&model); + let best = incumbent.map(|index| (targets[index] - y_mean) / y_scale); + // without a feasible point, the start is the least infeasible one + let start = incumbent.unwrap_or_else(|| { + let mut least = 0; + let mut violation = f64::INFINITY; + for (index, individual) in self.observations.iter().enumerate() { + let v = individual + .fitness() + .filter(|fitness| fitness.is_valid()) + .map_or(f64::INFINITY, Fitness::violation); + if v < violation { + (least, violation) = (index, v); + } + } + least + }); + let start = model.unit_point(start).to_vec(); + let standardized = model.targets(); + let (mut low, mut sum, mut high) = (f64::INFINITY, 0.0, f64::NEG_INFINITY); + for &y in standardized { + low = low.min(y); + sum += y; + high = high.max(y); + } + let lies = [low, sum / standardized.len() as f64, high]; + let surrogate = Surrogate { + objective: model, + constraints, + thresholds, + best, + }; + Some((surrogate, start, lies)) + } + + // the fantasized values of the point `unit` under `current`: the objective's, and each + // constraint's + fn fantasize(&self, current: &Surrogate, unit: &[f64], lies: [f64; 3]) -> (f64, Vec) { + let objective = match self.fantasy { + Fantasy::KrigingBeliever => current.objective.predict_unit(unit, None).0, + Fantasy::ConstantLiar(Lie::Min) => lies[0], + Fantasy::ConstantLiar(Lie::Mean) => lies[1], + Fantasy::ConstantLiar(Lie::Max) => lies[2], + }; + let constraints = current + .constraints + .iter() + .map(|model| model.predict_unit(unit, None).0) + .collect(); + (objective, constraints) + } + + // up to `count` new points: the models fitted to the evaluations, the `pending` points added + // with fantasized values, then the acquisition maximized, each point after the first with + // the points before it added alike + fn choose(&mut self, count: usize, pending: &[R::Genome], step: Step) -> Vec { + let mut chosen: Vec = Vec::with_capacity(count); + self.surrogate = None; + let Some((base, start, lies)) = self.fit(step) else { + for pick in 0..count { + match self.random_point(step, pick, &[pending, &chosen]) { + Some(genome) => chosen.push(genome), + None => break, + } + } + return chosen; + }; + let scaling = self.representation.scaling(); + let dims = scaling.dims(); + let mut fantasies = Fantasies::default(); + let mut current: Option = None; + let mut unit = vec![0.0; dims]; + for genome in pending { + R::to_unit(&scaling, genome, &mut unit); + let (objective, constraints) = + self.fantasize(current.as_ref().unwrap_or(&base), &unit, lies); + fantasies.push(&unit, objective, &constraints); + current = condition(&base, &fantasies).or(current); + } + for pick in 0..count { + let surrogate = current.as_ref().unwrap_or(&base); + let point = self + .maximize(surrogate, &scaling, &start, step, pick, &[pending, &chosen]) + .or_else(|| self.random_point(step, pick, &[pending, &chosen])); + let Some(point) = point else { break }; + if pick + 1 < count { + R::to_unit(&scaling, &point, &mut unit); + let (objective, constraints) = self.fantasize(surrogate, &unit, lies); + fantasies.push(&unit, objective, &constraints); + current = condition(&base, &fantasies).or(current); + } + chosen.push(point); + } + self.surrogate = Some(base); + chosen + } + + // the acquisition's best point under `surrogate` that isn't taken: L-BFGS-B from the raw + // samples' best and from `start` for reals, a hill climb on the lattice for integers + fn maximize( + &self, + surrogate: &Surrogate, + scaling: &Scaling, + start: &[f64], + step: Step, + pick: usize, + others: &[&[R::Genome]], + ) -> Option { + if self.representation.lattice().is_some() { + return self.maximize_lattice(surrogate, scaling, start, step, pick, others); + } + let search = self.search(surrogate); + let dims = scaling.dims(); // the raw samples, and the best of them as starts after the incumbent - let mut rng = root.derive(streams::RAW_SAMPLES).derive(generation); + let mut rng = self.stream(streams::RAW_SAMPLES, step, pick); let raw: Vec = (0..self.raw_samples * dims) .map(|_| rng.unit_f64()) .collect(); @@ -416,7 +877,7 @@ impl Bo { let key = |v: f64| if v.is_nan() { f64::NEG_INFINITY } else { v }; order.sort_by(|&a, &b| key(raw_values[b]).total_cmp(&key(raw_values[a]))); let mut starts = Vec::with_capacity(self.acquisition_starts + 1); - starts.push(model.unit_point(incumbent).to_vec()); + starts.push(start.to_vec()); for &i in order.iter().take(self.acquisition_starts) { starts.push(raw[i * dims..(i + 1) * dims].to_vec()); } @@ -442,14 +903,85 @@ impl Bo { .collect(); // highest first; stable, so ties keep the earlier start candidates.sort_by(|a, b| b.0.total_cmp(&a.0)); - self.model = Some(model); - for (_, unit) in &candidates { - let genome = scaling.to_genome(unit); - if !self.observed(&genome) { - return genome; + candidates + .iter() + .map(|(_, unit)| self.representation.genome_at(scaling, unit)) + .find(|genome| !self.taken(genome, others)) + } + + // the acquisition's best lattice point that isn't taken: the raw samples (every point of a + // small lattice), then a hill climb from the best of them and from `start` + fn maximize_lattice( + &self, + surrogate: &Surrogate, + scaling: &Scaling, + start: &[f64], + step: Step, + pick: usize, + others: &[&[R::Genome]], + ) -> Option { + let search = self.search(surrogate); + let dims = scaling.dims(); + // the acquisition at a lattice point, −∞ at a point taken + let value = |genome: &R::Genome| { + if self.taken(genome, others) { + return f64::NEG_INFINITY; + } + let mut unit = vec![0.0; dims]; + R::to_unit(scaling, genome, &mut unit); + let value = search.value(&unit, None); + if value.is_nan() { + f64::NEG_INFINITY + } else { + value + } + }; + let raw = match self.representation.enumerate(self.raw_samples) { + Some(all) => all, + None => { + let mut rng = self.stream(streams::RAW_SAMPLES, step, pick); + (0..self.raw_samples) + .map(|_| self.representation.random_genome(&mut rng)) + .collect() } + }; + let raw_values = map_in_order(raw.len(), |i| value(&raw[i])); + let mut order: Vec = (0..raw.len()).collect(); + // highest first; stable, so ties keep the earlier sample + order.sort_by(|&a, &b| raw_values[b].total_cmp(&raw_values[a])); + let mut starts = Vec::with_capacity(self.acquisition_starts + 1); + starts.push(self.representation.genome_at(scaling, start)); + for &i in order.iter().take(self.acquisition_starts) { + starts.push(raw[i].clone()); } - self.random_point(generation) + let climbs = map_in_order(starts.len(), |s| { + let mut point = starts[s].clone(); + let mut best = value(&point); + let mut neighbors = Vec::new(); + for _ in 0..LATTICE_STEPS { + self.representation.neighbors(&point, &mut neighbors); + let mut next: Option<(f64, usize)> = None; + for (n, neighbor) in neighbors.iter().enumerate() { + let v = value(neighbor); + if v > best && next.is_none_or(|(top, _)| v > top) { + next = Some((v, n)); + } + } + let Some((v, n)) = next else { break }; + best = v; + point = neighbors.swap_remove(n); + } + (best, point) + }); + let mut candidates: Vec<(f64, R::Genome)> = climbs; + candidates.extend(order.iter().map(|&i| (raw_values[i], raw[i].clone()))); + // highest first; stable, so ties keep the earlier start + candidates.sort_by(|a, b| b.0.total_cmp(&a.0)); + candidates + .into_iter() + .filter(|(value, _)| *value > f64::NEG_INFINITY) + .map(|(_, genome)| genome) + .find(|genome| !self.taken(genome, others)) } // the points of the next ask @@ -462,8 +994,10 @@ impl Bo { let design = self.design.iter().cloned(); self.pending.extend(design.map(Individual::new)); } else { - let point = self.next_point(); - self.pending.push(Individual::new(point)); + let step = Step::Generation(self.generation + 1); + let pending = self.proposed.clone(); + let points = self.choose(self.batch, &pending, step); + self.pending.extend(points.into_iter().map(Individual::new)); } self.indices.clear(); self.indices.extend(0..self.pending.len()); @@ -487,6 +1021,172 @@ impl Bo { } Ok(()) } + + // the number of constraints from what the fitness function provides + fn prepare_constraints(&mut self, provided: Provided) -> Result<()> { + let m = provided.inequalities; + self.acquisition.check_constrained(m)?; + // evaluations told without a run have no constraint values + let known = match self.constraints { + None if !self.observations.is_empty() => Some(0), + known => known, + }; + match known { + Some(known) if known != m && !self.observations.is_empty() => { + Err(Error::InvalidSetting { + setting: "fitness", + reason: format!( + "the fitness function gives {m} constraints, and the evaluations so far \ + have {known}" + ), + }) + } + _ => { + self.constraints = Some(m); + self.constraint_warm.resize(m, None); + self.constraint_warm.truncate(m); + Ok(()) + } + } + } + + // the constraints' values of an evaluation, checked: an error if they're missing + fn checked_values<'e>( + &self, + evaluations: &Evaluations<'e>, + position: usize, + ) -> Result> { + let m = self.constraints.unwrap_or(0); + if m == 0 { + return Ok(None); + } + match evaluations.inequalities(position) { + Some(values) if values.len() == m => Ok(Some(values)), + _ => Err(Error::InvalidSetting { + setting: "fitness", + reason: format!( + "Bayesian optimization with {m} constraints needs their values with each \ + fitness: tell them with `tell_evaluations` (as the engines do), from a \ + fitness function that provides them" + ), + }), + } + } + + // the constraints' values of a fitness, NaN for an invalid one + fn push_values(&mut self, fitness: Fitness, values: Option<&[f64]>) { + let m = self.constraints.unwrap_or(0); + match values { + Some(values) if fitness.is_valid() => self.constraint_values.extend_from_slice(values), + _ => self + .constraint_values + .extend(std::iter::repeat_n(f64::NAN, m)), + } + } + + // the best so far, from the newly evaluated observations from `first` on: the first on ties + fn update_best(&mut self, first: usize) { + let evaluations = self.evaluations - (self.observations.len() - first) as u64; + for (k, individual) in self.observations.as_slice()[first..].iter().enumerate() { + let fitness = individual.fitness().unwrap_or(Fitness::invalid()); + let better = match &self.best { + Some(best) => self + .objective + .is_better(fitness, best.fitness().unwrap_or(Fitness::invalid())), + None => true, + }; + if better { + self.best = Some(individual.clone()); + self.best_generation = self.generation; + self.best_evaluation = evaluations + k as u64 + 1; + } + } + } + + // takes a result under an asynchronous engine + fn receive_one( + &mut self, + genome: R::Genome, + fitness: Fitness, + values: Option<&[f64]>, + ) -> Result>> { + self.representation.validate(&genome)?; + if let Some(position) = self.proposed.iter().position(|g| *g == genome) { + self.proposed.remove(position); + } else if let Some(position) = self.requeued.iter().position(|g| *g == genome) { + self.requeued.remove(position); + } + let mut individual = Individual::new(genome); + individual.set_fitness(fitness); + if self.observed(individual.genome()) { + return Ok(Some(individual)); + } + self.evaluations += 1; + self.observations.push(individual); + self.push_values(fitness, values); + self.update_best(self.observations.len() - 1); + Ok(None) + } +} + +// the points added to the models with fantasized values, the unit-cube points a row each +#[derive(Default)] +struct Fantasies { + units: Vec, + objective: Vec, + // a row of the constraints' values per point + constraints: Vec, +} + +impl Fantasies { + fn push(&mut self, unit: &[f64], objective: f64, constraints: &[f64]) { + self.units.extend_from_slice(unit); + self.objective.push(objective); + self.constraints.extend_from_slice(constraints); + } +} + +// the models of `base` with the fantasized points added, as if observed, and the best value with +// those of them that are feasible by their fantasized constraints; None if a model doesn't factor +fn condition(base: &Surrogate, fantasies: &Fantasies) -> Option { + let objective = base + .objective + .with_points(&fantasies.units, &fantasies.objective)?; + let m = base.constraints.len(); + let mut constraints = Vec::with_capacity(m); + for (i, model) in base.constraints.iter().enumerate() { + let values: Vec = fantasies + .objective + .iter() + .enumerate() + .map(|(point, _)| fantasies.constraints[point * m + i]) + .collect(); + constraints.push(model.with_points(&fantasies.units, &values)?); + } + let mut best = base.best; + for (point, &value) in fantasies.objective.iter().enumerate() { + let feasible = (0..m).all(|i| fantasies.constraints[point * m + i] <= base.thresholds[i]); + if feasible && best.is_none_or(|best| value < best) { + best = Some(value); + } + } + Some(Surrogate { + objective, + constraints, + thresholds: base.thresholds.clone(), + best, + }) +} + +// the error for a batch of 0, or too large +fn check_batch(batch: usize) -> Result<()> { + if batch == 0 { + return Err(Error::InvalidSetting { + setting: "batch", + reason: "must be at least 1, got 0".to_string(), + }); + } + crate::operator::check_size("batch", batch).map(|_| ()) } // the model's targets from the values to minimize (None for invalid ones): transformed, and the @@ -543,11 +1243,9 @@ fn log_offset(values: &[Option], best: f64) -> f64 { .unwrap_or(1.0) } -// the acquisition function of a model, in its standardized units, as maximized +// the acquisition function of a surrogate, in its standardized units, as maximized struct Search<'a> { - model: &'a GaussianProcess, - // the best value, standardized - best: f64, + surrogate: &'a Surrogate, acquisition: Acquisition, // the model's scale of the values, for ξ scale: f64, @@ -555,7 +1253,75 @@ struct Search<'a> { impl Search<'_> { // the acquisition's value at the unit-cube point `u`, and its gradient - fn value(&self, u: &[f64], gradient: Option<&mut [f64]>) -> f64 { + fn value(&self, u: &[f64], mut gradient: Option<&mut [f64]>) -> f64 { + let surrogate = self.surrogate; + let dims = u.len(); + let mut value = 0.0; + if let Some(best) = surrogate.best { + value = self.objective(u, best, gradient.as_deref_mut()); + } else if let Some(gradient) = gradient.as_deref_mut() { + gradient.fill(0.0); + } + if surrogate.constraints.is_empty() { + return value; + } + // the logarithm of the probability of feasibility, Σ ln Φ((tᵢ − μᵢ)/σᵢ), and its gradient + let mut log_feasible = 0.0; + let mut d_log = vec![0.0; if gradient.is_some() { dims } else { 0 }]; + let (mut dmean, mut dvariance) = (vec![0.0; dims], vec![0.0; dims]); + for (model, &threshold) in surrogate.constraints.iter().zip(&surrogate.thresholds) { + let gradients = gradient + .is_some() + .then_some((&mut dmean[..], &mut dvariance[..])); + let (mean, variance) = model.predict_unit(u, gradients); + let floored = variance < VARIANCE_FLOOR; + let sd = variance.max(VARIANCE_FLOOR).sqrt(); + let [log, by_mean, by_sd] = + acquisition::log_probability_of_improvement_derivatives(mean, sd, threshold, 0.0); + log_feasible += log; + if gradient.is_some() { + for i in 0..dims { + let dsd = if floored { + 0.0 + } else { + dvariance[i] / (2.0 * sd) + }; + d_log[i] += by_mean * dmean[i] + by_sd * dsd; + } + } + } + match (surrogate.best, self.acquisition) { + // before a feasible point: the probability of feasibility alone, by its logarithm + (None, _) => { + if let Some(gradient) = gradient { + gradient.copy_from_slice(&d_log); + } + log_feasible + } + // EI × P: (EI P)′ = P EI′ + EI P (ln P)′ + (Some(_), Acquisition::ExpectedImprovement) => { + let feasible = crate::math::exp(log_feasible); + if let Some(gradient) = gradient { + for i in 0..dims { + gradient[i] = feasible * gradient[i] + value * feasible * d_log[i]; + } + } + value * feasible + } + // a logarithm: ln a + ln P + (Some(_), _) => { + if let Some(gradient) = gradient { + for i in 0..dims { + gradient[i] += d_log[i]; + } + } + value + log_feasible + } + } + } + + // the objective's acquisition at `u`, and its gradient + fn objective(&self, u: &[f64], best: f64, gradient: Option<&mut [f64]>) -> f64 { let dims = u.len(); let (mut dmean, mut dvariance) = (Vec::new(), Vec::new()); let gradients = if gradient.is_some() { @@ -565,21 +1331,21 @@ impl Search<'_> { } else { None }; - let (mean, variance) = self.model.predict_unit(u, gradients); + let (mean, variance) = self.surrogate.objective.predict_unit(u, gradients); let floored = variance < VARIANCE_FLOOR; let sd = variance.max(VARIANCE_FLOOR).sqrt(); let [value, by_mean, by_sd] = match self.acquisition { Acquisition::ExpectedImprovement => { - acquisition::expected_improvement_derivatives(mean, sd, self.best) + acquisition::expected_improvement_derivatives(mean, sd, best) } Acquisition::LogExpectedImprovement => { - acquisition::log_expected_improvement_derivatives(mean, sd, self.best) + acquisition::log_expected_improvement_derivatives(mean, sd, best) } Acquisition::ProbabilityOfImprovement { xi } => { acquisition::log_probability_of_improvement_derivatives( mean, sd, - self.best, + best, xi / self.scale, ) } @@ -601,21 +1367,21 @@ impl Search<'_> { } } -impl Reevaluate for Bo { +impl Reevaluate for Bo { /// As [`Bo::reevaluate`]: the next ask gives every evaluated point again. fn reevaluate(&mut self) -> Result<()> { Bo::reevaluate(self) } } -impl Algorithm for Bo { - type Genome = Reals; +impl Algorithm for Bo { + type Genome = R::Genome; fn objective(&self) -> Objective { self.objective } - fn ask(&mut self) -> Candidates<'_, Reals> { + fn ask(&mut self) -> Candidates<'_, R::Genome> { if !self.asked { self.build_ask(); self.asked = true; @@ -623,7 +1389,16 @@ impl Algorithm for Bo { Candidates::new(&self.pending, &self.indices) } + /// # Errors + /// + /// As [`Algorithm::tell`], and [`Error::InvalidSetting`] after a run with constraints, whose + /// values a tell doesn't have: [`tell_evaluations`](Algorithm::tell_evaluations) gives them. fn tell(&mut self, fitness: &[Fitness]) -> Result<()> { + self.tell_evaluations(&Evaluations::new(fitness)) + } + + fn tell_evaluations(&mut self, evaluations: &Evaluations<'_>) -> Result<()> { + let fitness = evaluations.fitness(); if !self.asked { return Err(Error::TellWithoutAsk); } @@ -633,48 +1408,50 @@ impl Algorithm for Bo { got: fitness.len(), }); } + let mut values = Vec::with_capacity(fitness.len()); + for position in 0..fitness.len() { + values.push(self.checked_values(evaluations, position)?); + } self.asked = false; self.evaluations += fitness.len() as u64; self.indices.clear(); if self.reevaluating { self.reevaluating = false; self.pending.clear(); - for (observation, &fitness) in self.observations.iter_mut().zip(fitness) { + let m = self.constraints.unwrap_or(0); + for (index, (observation, &fitness)) in + self.observations.iter_mut().zip(fitness).enumerate() + { observation.set_fitness(fitness); + let row = &mut self.constraint_values[index * m..(index + 1) * m]; + match values[index] { + Some(values) if fitness.is_valid() => row.copy_from_slice(values), + _ => row.fill(f64::NAN), + } } self.best = None; - update_best( - &mut self.best, - &mut self.best_generation, - self.generation, - self.objective, - self.observations.as_slice(), - ); + self.update_best(0); return Ok(()); } if !self.observations.is_empty() { self.generation += 1; } let first = self.observations.len(); - for (mut individual, &fitness) in self.pending.drain(..).zip(fitness) { + let pending = std::mem::take(&mut self.pending); + for ((mut individual, &fitness), values) in pending.into_iter().zip(fitness).zip(values) { individual.set_fitness(fitness); self.observations.push(individual); + self.push_values(fitness, values); } - update_best( - &mut self.best, - &mut self.best_generation, - self.generation, - self.objective, - &self.observations.as_slice()[first..], - ); + self.update_best(first); Ok(()) } - fn population(&self) -> &Population { + fn population(&self) -> &Population { &self.observations } - fn best(&self) -> Option<&Individual> { + fn best(&self) -> Option<&Individual> { self.best.as_ref() } @@ -689,27 +1466,150 @@ impl Algorithm for Bo { fn best_generation(&self) -> u64 { self.best_generation } + + /// Whether every point of an [`Integer`](crate::genome::Integer) genome's lattice is + /// evaluated: never for [`Real`] genomes. + fn is_finished(&self) -> bool { + !self.reevaluating + && self + .representation + .lattice() + .is_some_and(|points| self.observations.len() as u128 >= points) + } + + /// Takes the number of inequality constraints whose values the fitness function gives, such + /// as a [`Constrained`](crate::constraint::Constrained) one: with constraints, each is + /// modeled. + /// + /// # Errors + /// + /// [`Error::InvalidSetting`] for the upper confidence bound with constraints, or another + /// number of constraints than the evaluations so far have. + fn prepare(&mut self, provided: Provided) -> Result<()> { + self.prepare_constraints(provided) + } + + fn wants(&self) -> Wanted { + if self.constraints.unwrap_or(0) > 0 { + Wanted::NOTHING.with_inequalities() + } else { + Wanted::NOTHING + } + } } -// the best so far, from the evaluated `individuals` in order: the first on ties -fn update_best( - best: &mut Option>, - best_generation: &mut u64, - generation: u64, - objective: Objective, - individuals: &[Individual], -) { - for individual in individuals { - let fitness = individual.fitness().unwrap_or(Fitness::invalid()); - let better = match best { - Some(best) => { - objective.is_better(fitness, best.fitness().unwrap_or(Fitness::invalid())) - } - None => true, +impl Incremental for Bo { + type Genome = R::Genome; + + fn objective(&self) -> Objective { + self.objective + } + + /// The next point: a point proposed before a checkpoint whose result never came, then the + /// initial design, then the points the model chooses with the points still being evaluated + /// added with a [`Fantasy`] value (random points until a result has a valid score). Once + /// every point of an integer lattice is evaluated or being evaluated, the best point again. + fn propose(&mut self) -> R::Genome { + // a point proposed again isn't a new proposal: a resumed run's later proposals draw the + // random numbers that an uninterrupted run's do + let genome = if self.requeued.is_empty() { + let genome = self.next_proposal(); + self.proposals += 1; + genome + } else { + self.requeued.remove(0) + }; + self.proposed.push(genome.clone()); + genome + } + + /// Takes a result: the genome joins the evaluations, unless it's evaluated already (then it's + /// returned). + /// + /// # Errors + /// + /// [`Error::InvalidGenome`] for a genome that doesn't fit the representation, and + /// [`Error::InvalidSetting`] in a run with constraints, whose values + /// [`receive_evaluation`](Incremental::receive_evaluation) gives. Nothing changes on errors. + fn receive( + &mut self, + genome: R::Genome, + fitness: Fitness, + ) -> Result>> { + let fitness = [fitness]; + self.receive_evaluation(genome, &Evaluations::new(&fitness)) + } + + fn receive_evaluation( + &mut self, + genome: R::Genome, + evaluation: &Evaluations<'_>, + ) -> Result>> { + let &[fitness] = evaluation.fitness() else { + return Err(Error::FitnessCount { + expected: 1, + got: evaluation.len(), + }); }; - if better { - *best = Some(individual.clone()); - *best_generation = generation; + let values = self.checked_values(evaluation, 0)?; + self.receive_one(genome, fitness, values) + } + + fn population(&self) -> &Population { + &self.observations + } + + /// The initial design's size: the engine counts a generation every this many evaluations. + fn population_size(&self) -> usize { + self.initial_points + } + + fn best(&self) -> Option<&Individual> { + self.best.as_ref() + } + + fn evaluations(&self) -> u64 { + self.evaluations + } + + fn best_evaluation(&self) -> u64 { + self.best_evaluation + } + + /// As [`Algorithm::prepare`]; besides, the points proposed and not received, whose + /// evaluations a new run no longer has (a run that continues a checkpoint, or another run of + /// the same algorithm), are proposed again first. + fn prepare(&mut self, provided: Provided) -> Result<()> { + self.prepare_constraints(provided)?; + let lost = std::mem::take(&mut self.proposed); + self.requeued.splice(0..0, lost); + Ok(()) + } + + fn wants(&self) -> Wanted { + Algorithm::wants(self) + } +} + +impl Bo { + // a proposal that isn't a requeued point + fn next_proposal(&mut self) -> R::Genome { + while self.design_proposed < self.design.len() { + let genome = self.design[self.design_proposed].clone(); + self.design_proposed += 1; + if !self.taken(&genome, &[&self.proposed]) { + return genome; + } + } + let step = Step::Proposal(self.proposals); + let pending = self.proposed.clone(); + if let Some(genome) = self.choose(1, &pending, step).pop() { + return genome; + } + // every point of the lattice is taken + match &self.best { + Some(best) => best.genome().clone(), + None => self.design[0].clone(), } } } @@ -718,13 +1618,14 @@ fn update_best( /// /// Defaults: maximize; an initial design of 2(n + 1) points for n searched genes; /// [`Acquisition::LogExpectedImprovement`]; the [Matérn 5/2 kernel](Kernel::Matern52) without -/// noise, the model interpolating the values; [`Output::Standardize`]; 1000 raw samples and 10 starts for the acquisition's maximization; -/// 5 starts for the hyperparameters'; a random seed. +/// noise, the model interpolating the values; [`Output::Standardize`]; 1000 raw samples and 10 +/// starts for the acquisition's maximization; 5 starts for the hyperparameters'; batches of 1 +/// point, the [Kriging believer](Fantasy::KrigingBeliever); a random seed. #[derive(Clone, Debug)] -pub struct BoBuilder { - real: Real, +pub struct BoBuilder { + representation: R, initial_points: Option, - initial_genomes: Vec, + initial_genomes: Vec, acquisition: Acquisition, kernel: Kernel, noise: Noise, @@ -732,11 +1633,13 @@ pub struct BoBuilder { raw_samples: usize, acquisition_starts: usize, hyperparameter_starts: usize, + batch: usize, + fantasy: Fantasy, objective: Objective, seed: Option, } -impl BoBuilder { +impl BoBuilder { /// The number of points of the initial design, at least 1: 2(n + 1) for n searched genes /// (genes whose bounds differ) by default, a small design that leaves most evaluations to the /// model. 10n is the usual size for an accurate model of the whole box (Loeppky, Sacks and @@ -749,7 +1652,7 @@ impl BoBuilder { /// Genomes to evaluate first, in the initial design: at most /// [`initial_points`](BoBuilder::initial_points), and distinct. A Latin hypercube sample /// fills the rest of the design. - pub fn initial_genomes>(mut self, genomes: I) -> Self { + pub fn initial_genomes>(mut self, genomes: I) -> Self { self.initial_genomes = genomes.into_iter().collect(); self } @@ -803,6 +1706,26 @@ impl BoBuilder { self } + /// The number of points q of each generation after the initial design, at least 1: 1 by + /// default. A batch of q points is chosen one after the other, each after the points before + /// it are added to the model with a [`Fantasy`] value ([`fantasy`](BoBuilder::fantasy)), and + /// then evaluated together: in parallel with + /// [`Engine::parallel`](crate::Engine::parallel), so that a generation takes about as long + /// as one evaluation. A batch needs more evaluations than one point at a time to come as + /// close, as each point is chosen knowing less, and fewer generations. + pub fn batch(mut self, points: usize) -> Self { + self.batch = points; + self + } + + /// What the points of a batch, and those still being evaluated under an + /// [`AsyncEngine`](crate::engine::AsyncEngine), are taken to be worth: the + /// [Kriging believer](Fantasy::KrigingBeliever) by default. + pub fn fantasy(mut self, fantasy: Fantasy) -> Self { + self.fantasy = fantasy; + self + } + /// Whether higher or lower fitness is better. Maximize by default. pub fn objective(mut self, objective: Objective) -> Self { self.objective = objective; @@ -831,18 +1754,18 @@ impl BoBuilder { /// # Errors /// /// - [`Error::InvalidSetting`] for a representation without a gene that has more than one - /// value, an initial design of 0 points or above 2^24, more initial genomes than initial - /// points or the same genome twice, an invalid [`Acquisition`] parameter or [`Noise`], 0 - /// raw samples, acquisition starts of 0 or more than the raw samples, or hyperparameter - /// starts of 0. + /// value, an initial design of 0 points, above 2^24 or above the points of an integer + /// lattice, more initial genomes than initial points or the same genome twice, an invalid + /// [`Acquisition`] parameter or [`Noise`], 0 raw samples, acquisition starts of 0 or more + /// than the raw samples, hyperparameter starts of 0, or a batch of 0 or above 2^24. /// - [`Error::InvalidGenome`] for an initial genome that doesn't fit the representation. - pub fn build(self) -> Result { + pub fn build(self) -> Result> { let invalid = |setting: &'static str, reason: String| Err(Error::InvalidSetting { setting, reason }); - let dims = self.real.variable_genes().len(); + let dims = self.representation.scaling().dims(); if dims == 0 { return invalid( - "real", + "representation", "Bayesian optimization needs a gene with more than one value".to_string(), ); } @@ -851,6 +1774,22 @@ impl BoBuilder { return invalid("initial_points", "must be at least 1, got 0".to_string()); } crate::operator::check_size("initial_points", initial_points)?; + if let Some(points) = self.representation.lattice() + && initial_points as u128 > points + && self.initial_points.is_some() + { + return invalid( + "initial_points", + format!("at most the {points} points of the lattice, got {initial_points}"), + ); + } + // the default design, on a lattice of fewer points: the whole lattice + let initial_points = self + .representation + .lattice() + .map_or(initial_points, |points| { + initial_points.min(points.min(usize::MAX as u128) as usize) + }); if self.initial_genomes.len() > initial_points { return invalid( "initial_genomes", @@ -861,11 +1800,8 @@ impl BoBuilder { ); } for (index, genome) in self.initial_genomes.iter().enumerate() { - self.real.validate(genome)?; - if self.initial_genomes[..index] - .iter() - .any(|other| other[..] == genome[..]) - { + self.representation.validate(genome)?; + if self.initial_genomes[..index].contains(genome) { return invalid( "initial_genomes", format!("genome {index} is given twice: {genome:?}"), @@ -894,6 +1830,7 @@ impl BoBuilder { ); } crate::operator::check_size("hyperparameter_starts", self.hyperparameter_starts)?; + check_batch(self.batch)?; let seed = self .seed .unwrap_or_else(|| StreamRng::from_entropy().next_u64()); @@ -901,10 +1838,18 @@ impl BoBuilder { let missing = initial_points - design.len(); if missing > 0 { let mut rng = StreamRng::seed_from_u64(seed).derive(streams::DESIGN); - design.extend(self.real.latin_hypercube(missing, &mut rng)?); + let sample = self.representation.design(missing, &mut rng)?; + // a point the design has already (as on a small lattice): a random one instead + for genome in sample { + let mut genome = genome; + while design.contains(&genome) { + genome = self.representation.random_genome(&mut rng); + } + design.push(genome); + } } Ok(Bo { - real: self.real, + representation: self.representation, initial_points, acquisition: self.acquisition, kernel: self.kernel, @@ -913,11 +1858,16 @@ impl BoBuilder { raw_samples: self.raw_samples, acquisition_starts: self.acquisition_starts, hyperparameter_starts: self.hyperparameter_starts, + batch: self.batch, + fantasy: self.fantasy, objective: self.objective, seed, design, observations: Population::default(), + constraints: None, + constraint_values: Vec::new(), warm: None, + constraint_warm: Vec::new(), pending: Vec::new(), indices: Vec::new(), asked: false, @@ -926,97 +1876,15 @@ impl BoBuilder { evaluations: 0, best: None, best_generation: 0, - model: None, - model_best: f64::NAN, + best_evaluation: 0, + proposed: Vec::new(), + requeued: Vec::new(), + design_proposed: 0, + proposals: 0, + surrogate: None, }) } } #[cfg(test)] -mod tests { - use super::*; - - // a model of a smooth function of 2 genes, from 10 points - fn model() -> (GaussianProcess, f64) { - let real = Real::new([-1.0..=2.0, 0.0..=3.0]).unwrap(); - let points = real - .latin_hypercube(10, &mut StreamRng::seed_from_u64(2)) - .unwrap(); - let values: Vec = points - .iter() - .map(|x| (2.0 * x[0]).sin() + 0.5 * x[1] * x[1] - 0.3 * x[0] * x[1]) - .collect(); - let scaling = Scaling::new(&real); - let mut x = vec![0.0; 20]; - for (point, unit) in points.iter().zip(x.as_chunks_mut::<2>().0) { - scaling.to_unit(point, unit); - } - let settings = gp::Settings { - kernel: Kernel::Matern52, - noise: Noise::default(), - starts: 3, - seed: 1, - }; - let model = GaussianProcess::fit_unit(settings, scaling, x, &values, None).unwrap(); - let (mean, scale) = model.standardization(); - let best = values.iter().fold(f64::INFINITY, |a, &b| a.min(b)); - (model, (best - mean) / scale) - } - - #[test] - fn the_acquisitions_gradients_match_central_differences() { - let (model, best) = model(); - let acquisitions = [ - Acquisition::ExpectedImprovement, - Acquisition::LogExpectedImprovement, - Acquisition::ProbabilityOfImprovement { xi: 0.01 }, - Acquisition::UpperConfidenceBound { beta: 2.0 }, - ]; - for acquisition in acquisitions { - let search = Search { - model: &model, - best, - acquisition, - scale: model.standardization().1, - }; - for u in [[0.3, 0.6], [0.9, 0.1], [0.55, 0.45], [0.05, 0.95]] { - let mut gradient = [0.0; 2]; - let value = search.value(&u, Some(&mut gradient)); - assert_eq!(value.to_bits(), search.value(&u, None).to_bits()); - for i in 0..2 { - let h = 1e-6; - let (mut plus, mut minus) = (u, u); - plus[i] += h; - minus[i] -= h; - let numeric = - (search.value(&plus, None) - search.value(&minus, None)) / (2.0 * h); - assert!( - (gradient[i] - numeric).abs() <= 1e-5 * numeric.abs().max(1.0), - "{acquisition:?} at {u:?}, gene {i}: {} against {numeric}", - gradient[i] - ); - } - } - } - } - - #[test] - fn the_log_transform_keeps_the_order_and_imputes_the_worst() { - let values = [Some(3.0), None, Some(10.0), Some(3.5), Some(1e6), Some(4.0)]; - let targets = transform(&values, Output::Log).unwrap(); - // δ is the first quartile of the distances above the best, 0.5 here: the ⌊5/4⌋-th of - // 0, 0.5, 1, 7 and 999997 - assert_eq!(targets[0], 0.5_f64.ln()); - assert_eq!(targets[3], 1.0_f64.ln()); - assert!(targets[0] < targets[3] && targets[3] < targets[5] && targets[5] < targets[2]); - assert_eq!(targets[1], targets[4]); - let targets = transform(&values, Output::Standardize).unwrap(); - assert_eq!(targets, [3.0, 1e6, 10.0, 3.5, 1e6, 4.0]); - assert_eq!(transform(&[None, None], Output::Log), None); - // every value the best: δ is 1 - assert_eq!( - transform(&[Some(2.0), Some(2.0)], Output::Log), - Some(vec![0.0, 0.0]) - ); - } -} +mod tests; diff --git a/src/algorithm/bo/space.rs b/src/algorithm/bo/space.rs new file mode 100644 index 00000000..5d6ec5ea --- /dev/null +++ b/src/algorithm/bo/space.rs @@ -0,0 +1,276 @@ +//! The representations Bayesian optimization searches: [`Real`] genomes, and [`Integer`] genomes +//! with their genes rounded inside the kernel. + +// the sealed trait's methods take the crate's own unit-cube map: unnameable outside the crate +#![allow(private_interfaces)] + +use crate::genome::{Integer, Integers, Real, Reals, Representation}; +use crate::model::gp::Scaling; +use crate::{Error, Result, StreamRng}; + +/// A representation that [`Bo`](super::Bo) searches: [`Real`], or [`Integer`]. Sealed: its +/// methods are genoxide's own. +/// +/// - **[`Real`]**: the model's inputs are the genes, scaled to the unit cube by the bounds, and +/// the acquisition function is maximized by L-BFGS-B with its gradient. +/// - **[`Integer`]**: each gene's values are scaled to the unit cube by the bounds alike, and +/// every point the model is given or asked about is a point of the integer lattice: Garrido- +/// Merchán and Hernández-Lobato's (2020) transformation, the genes rounded to the nearest +/// integer inside the kernel, so that the model is constant between integers and certain at +/// an evaluated point (their eq. 7). The acquisition function is then a function of the lattice, +/// maximized there: at the raw samples, random lattice points (or every point, for a lattice of +/// at most that many), then by a hill climb from the best of them and from the best point +/// evaluated, each step to the best of the neighbors one away in one gene, until none is +/// better. Once every point of the lattice is evaluated, the search +/// [has finished](crate::algorithm::Algorithm::is_finished). +/// +/// Reference: Garrido-Merchán, E. C. and Hernández-Lobato, D. (2020). Dealing with categorical and +/// integer-valued variables in Bayesian optimization with Gaussian processes. *Neurocomputing* +/// 380: 20-35. +pub trait Space: sealed::Sealed {} + +impl Space for Real {} +impl Space for Integer {} + +pub(crate) mod sealed { + use super::*; + + /// What Bayesian optimization needs of a representation. + pub trait Sealed: Representation { + // the lower and upper bounds of each gene, as reals + fn real_bounds(&self) -> Vec<(f64, f64)>; + + // the map between genomes and the model's unit cube + fn scaling(&self) -> Scaling { + Scaling::from_bounds(&self.real_bounds()) + } + + // the unit-cube coordinates of `genome` + fn to_unit(scaling: &Scaling, genome: &Self::Genome, unit: &mut [f64]); + + // the genome at the unit-cube coordinates `unit`, in the bounds: on the lattice, the + // nearest point + fn genome_at(&self, scaling: &Scaling, unit: &[f64]) -> Self::Genome; + + // the genes of `genome` as reals, for the model's predictions + fn gene_values(genome: &Self::Genome) -> Vec; + + // `n` genomes of a Latin hypercube + fn design(&self, n: usize, rng: &mut StreamRng) -> Result>; + + // the number of points of an integer lattice (saturating), None for reals + fn lattice(&self) -> Option; + + // every point of the lattice, if there are at most `limit` + fn enumerate(&self, limit: usize) -> Option>; + + // the lattice's neighbors of `genome`: one away in one gene, within the bounds + fn neighbors(&self, genome: &Self::Genome, neighbors: &mut Vec); + } +} + +impl sealed::Sealed for Real { + fn real_bounds(&self) -> Vec<(f64, f64)> { + self.bounds() + .iter() + .map(|range| (*range.start(), *range.end())) + .collect() + } + + fn scaling(&self) -> Scaling { + Scaling::new(self) + } + + fn to_unit(scaling: &Scaling, genome: &Reals, unit: &mut [f64]) { + scaling.to_unit(genome, unit); + } + + fn genome_at(&self, scaling: &Scaling, unit: &[f64]) -> Reals { + scaling.to_genome(unit) + } + + fn gene_values(genome: &Reals) -> Vec { + genome.to_vec() + } + + fn design(&self, n: usize, rng: &mut StreamRng) -> Result> { + self.latin_hypercube(n, rng) + } + + fn lattice(&self) -> Option { + None + } + + fn enumerate(&self, _limit: usize) -> Option> { + None + } + + fn neighbors(&self, _genome: &Reals, _neighbors: &mut Vec) {} +} + +// the value of an integer gene nearest to `x`, in [lower, upper] +fn nearest(x: f64, lower: i64, upper: i64) -> i64 { + // `as` saturates, and a NaN becomes 0, which the clamp moves into the bounds + (x.round() as i64).clamp(lower, upper) +} + +impl sealed::Sealed for Integer { + fn real_bounds(&self) -> Vec<(f64, f64)> { + self.bounds() + .iter() + .map(|range| (*range.start() as f64, *range.end() as f64)) + .collect() + } + + fn to_unit(scaling: &Scaling, genome: &Integers, unit: &mut [f64]) { + scaling.to_unit_by(|i| genome[i] as f64, unit); + } + + fn genome_at(&self, scaling: &Scaling, unit: &[f64]) -> Integers { + let bounds = self.bounds(); + let mut genome: Vec = bounds.iter().map(|range| *range.start()).collect(); + for (k, &i) in scaling.variable().iter().enumerate() { + let (low, high) = (*bounds[i].start(), *bounds[i].end()); + genome[i] = nearest(scaling.gene_at(k, unit[k]), low, high); + } + Integers::from(genome) + } + + fn gene_values(genome: &Integers) -> Vec { + genome.iter().map(|&gene| gene as f64).collect() + } + + fn design(&self, n: usize, rng: &mut StreamRng) -> Result> { + if n == 0 { + return Err(Error::InvalidSetting { + setting: "n", + reason: "a Latin hypercube needs at least 1 genome".to_string(), + }); + } + crate::operator::check_size("n", n)?; + let mut genomes = vec![Vec::with_capacity(self.genome_len()); n]; + let mut strata: Vec = Vec::with_capacity(n); + for range in self.bounds() { + let (low, high) = (*range.start(), *range.end()); + if low == high { + for genome in &mut genomes { + genome.push(low); + } + continue; + } + strata.clear(); + strata.extend(0..n); + for last in (1..n).rev() { + let other = rng.below(last + 1); + strata.swap(last, other); + } + // the values low..=high as the intervals [low − ½, high + ½), stratified like reals + let values = (high - low) as f64 + 1.0; + for (genome, &stratum) in genomes.iter_mut().zip(&strata) { + let u = (stratum as f64 + rng.unit_f64()) / n as f64; + let x = low as f64 - 0.5 + u * values; + genome.push(nearest(x, low, high)); + } + } + Ok(genomes.into_iter().map(Integers::from).collect()) + } + + fn lattice(&self) -> Option { + let mut points: u128 = 1; + for range in self.bounds() { + let values = u128::from(range.end().abs_diff(*range.start())) + 1; + points = points.saturating_mul(values); + } + Some(points) + } + + fn enumerate(&self, limit: usize) -> Option> { + let points = self.lattice()?; + if points > limit as u128 { + return None; + } + let bounds = self.bounds(); + let mut genome: Vec = bounds.iter().map(|range| *range.start()).collect(); + let mut all = Vec::with_capacity(points as usize); + loop { + all.push(Integers::from(genome.clone())); + // the next point, the last gene fastest + let mut gene = genome.len(); + loop { + if gene == 0 { + return Some(all); + } + gene -= 1; + if genome[gene] < *bounds[gene].end() { + genome[gene] += 1; + break; + } + genome[gene] = *bounds[gene].start(); + } + } + } + + fn neighbors(&self, genome: &Integers, neighbors: &mut Vec) { + neighbors.clear(); + for (gene, range) in self.bounds().iter().enumerate() { + let value = genome[gene]; + if value > *range.start() { + let mut neighbor = genome.clone(); + neighbor[gene] = value - 1; + neighbors.push(neighbor); + } + if value < *range.end() { + let mut neighbor = genome.clone(); + neighbor[gene] = value + 1; + neighbors.push(neighbor); + } + } + } +} + +#[cfg(test)] +mod tests { + use super::sealed::Sealed; + use super::*; + + #[test] + fn integer_genomes_round_trip_through_the_unit_cube() { + let integer = Integer::new([-3..=4, 7..=7, 0..=1]).unwrap(); + let scaling = integer.scaling(); + assert_eq!(scaling.dims(), 2); + for genome in integer.enumerate(100).unwrap() { + let mut unit = [0.0; 2]; + Integer::to_unit(&scaling, &genome, &mut unit); + assert!(unit.iter().all(|u| (0.0..=1.0).contains(u))); + assert_eq!(integer.genome_at(&scaling, &unit), genome); + } + assert_eq!(integer.lattice(), Some(16)); + assert_eq!(integer.enumerate(15), None); + // the nearest lattice point of any unit-cube point + let genome = integer.genome_at(&scaling, &[0.5, 0.49]); + assert_eq!(genome[..], [1, 7, 0]); + } + + #[test] + fn an_integer_latin_hypercube_stratifies_each_gene() { + let integer = Integer::new([0..=9, 5..=5]).unwrap(); + let design = integer + .design(10, &mut StreamRng::seed_from_u64(3)) + .unwrap(); + let mut first: Vec = design.iter().map(|genome| genome[0]).collect(); + first.sort_unstable(); + assert_eq!(first, (0..=9).collect::>()); + assert!(design.iter().all(|genome| genome[1] == 5)); + } + + #[test] + fn neighbors_stay_in_the_bounds() { + let integer = Integer::new([0..=2, -1..=1]).unwrap(); + let mut neighbors = Vec::new(); + integer.neighbors(&Integers::from(vec![0, 1]), &mut neighbors); + assert_eq!( + neighbors, + [Integers::from(vec![1, 1]), Integers::from(vec![0, 0])] + ); + } +} diff --git a/src/algorithm/bo/tests.rs b/src/algorithm/bo/tests.rs new file mode 100644 index 00000000..9ddca350 --- /dev/null +++ b/src/algorithm/bo/tests.rs @@ -0,0 +1,216 @@ +use super::*; +use crate::genome::Integer; + +// a model of a smooth function of 2 genes, from 10 points, with its best value standardized +fn model(f: impl Fn(&[f64]) -> f64, seed: u64) -> (GaussianProcess, f64) { + let real = Real::new([-1.0..=2.0, 0.0..=3.0]).unwrap(); + let points = real + .latin_hypercube(10, &mut StreamRng::seed_from_u64(2)) + .unwrap(); + let values: Vec = points.iter().map(|x| f(x)).collect(); + let scaling = Scaling::new(&real); + let mut x = vec![0.0; 20]; + for (point, unit) in points.iter().zip(x.as_chunks_mut::<2>().0) { + scaling.to_unit(point, unit); + } + let settings = gp::Settings { + kernel: Kernel::Matern52, + noise: Noise::default(), + starts: 3, + seed, + }; + let model = GaussianProcess::fit_unit(settings, scaling, x, &values, None).unwrap(); + let (mean, scale) = model.standardization(); + let best = values.iter().fold(f64::INFINITY, |a, &b| a.min(b)); + (model, (best - mean) / scale) +} + +fn smooth(x: &[f64]) -> f64 { + (2.0 * x[0]).sin() + 0.5 * x[1] * x[1] - 0.3 * x[0] * x[1] +} + +// a constraint g(x) = x₀ + x₁ − 2.5, feasible below the diagonal, standardized +fn constraint() -> (GaussianProcess, f64) { + let (model, _) = model(|x| x[0] + x[1] - 2.5 + 0.1 * (3.0 * x[0]).cos(), 4); + let (mean, scale) = model.standardization(); + (model, (0.0 - mean) / scale) +} + +// the gradient of `search` at `u` against central differences +fn check_gradient(search: &Search<'_>, u: [f64; 2], name: &str) { + let mut gradient = [0.0; 2]; + let value = search.value(&u, Some(&mut gradient)); + assert_eq!(value.to_bits(), search.value(&u, None).to_bits()); + for i in 0..2 { + let h = 1e-6; + let (mut plus, mut minus) = (u, u); + plus[i] += h; + minus[i] -= h; + let numeric = (search.value(&plus, None) - search.value(&minus, None)) / (2.0 * h); + assert!( + (gradient[i] - numeric).abs() <= 1e-5 * numeric.abs().max(1.0), + "{name} at {u:?}, gene {i}: {} against {numeric}", + gradient[i] + ); + } +} + +const POINTS: [[f64; 2]; 4] = [[0.3, 0.6], [0.9, 0.1], [0.55, 0.45], [0.05, 0.95]]; + +fn unconstrained(objective: GaussianProcess, best: f64) -> Surrogate { + Surrogate { + objective, + constraints: Vec::new(), + thresholds: Vec::new(), + best: Some(best), + } +} + +#[test] +fn the_acquisitions_gradients_match_central_differences() { + let (objective, best) = model(smooth, 1); + let acquisitions = [ + Acquisition::ExpectedImprovement, + Acquisition::LogExpectedImprovement, + Acquisition::ProbabilityOfImprovement { xi: 0.01 }, + Acquisition::UpperConfidenceBound { beta: 2.0 }, + ]; + let surrogate = unconstrained(objective.clone(), best); + for acquisition in acquisitions { + let search = Search { + surrogate: &surrogate, + acquisition, + scale: objective.standardization().1, + }; + for u in POINTS { + check_gradient(&search, u, &format!("{acquisition:?}")); + } + } +} + +#[test] +fn the_probability_of_feasibility_and_its_gradient() { + let (objective, best) = model(smooth, 1); + let (constraint, threshold) = constraint(); + let mut surrogate = Surrogate { + objective: objective.clone(), + constraints: vec![constraint.clone(), constraint.clone()], + thresholds: vec![threshold, threshold + 0.3], + best: Some(best), + }; + let scale = objective.standardization().1; + // weighing EI by P, and log-EI and log-PI by ln P, and before a feasible point, ln P alone + for acquisition in [ + Acquisition::ExpectedImprovement, + Acquisition::LogExpectedImprovement, + Acquisition::ProbabilityOfImprovement { xi: 0.01 }, + ] { + for best in [Some(best), None] { + surrogate.best = best; + let search = Search { + surrogate: &surrogate, + acquisition, + scale, + }; + for u in POINTS { + check_gradient(&search, u, &format!("{acquisition:?} with {best:?}")); + } + } + } + // the value is the unconstrained one times P, P = Π Φ((tᵢ − μᵢ)/σᵢ) + surrogate.best = Some(best); + let plain = unconstrained(objective, best); + for u in POINTS { + let mut p = 1.0; + for &t in &surrogate.thresholds { + let (mean, variance) = constraint.predict_unit(&u, None); + let sd = variance.max(VARIANCE_FLOOR).sqrt(); + p *= 0.5 * crate::math::erfc(-(t - mean) / sd / std::f64::consts::SQRT_2); + } + let ei = |surrogate| { + Search { + surrogate, + acquisition: Acquisition::ExpectedImprovement, + scale, + } + .value(&u, None) + }; + let weighed = ei(&surrogate); + assert!((weighed - ei(&plain) * p).abs() <= 1e-12 * weighed.abs().max(1e-300)); + } +} + +#[test] +fn a_believer_keeps_the_mean_and_removes_the_uncertainty() { + let (objective, best) = model(smooth, 1); + let base = unconstrained(objective, best); + let u = [0.37, 0.71]; + let (mean, variance) = base.objective.predict_unit(&u, None); + assert!(variance > 1e-4); + let mut fantasies = Fantasies::default(); + fantasies.push(&u, mean, &[]); + let believed = condition(&base, &fantasies).unwrap(); + // the mean is the same everywhere (the model conditioned on its own prediction), and the + // variance at the point is gone + for v in POINTS.iter().chain([&u]) { + let (before, _) = base.objective.predict_unit(v, None); + let (after, _) = believed.objective.predict_unit(v, None); + assert!( + (before - after).abs() < 1e-9, + "{v:?}: {before} against {after}" + ); + } + assert!(believed.objective.predict_unit(&u, None).1 < 1e-9); + // a lie above the mean pulls the mean up around the point, and lowers nothing below the best + let mut lie = Fantasies::default(); + lie.push(&u, mean + 2.0, &[]); + let lied = condition(&base, &lie).unwrap(); + assert!((lied.objective.predict_unit(&u, None).0 - (mean + 2.0)).abs() < 1e-6); + assert_eq!(lied.best, Some(best)); + // a fantasy below the best is the best + let mut low = Fantasies::default(); + low.push(&u, best - 1.0, &[]); + assert_eq!(condition(&base, &low).unwrap().best, Some(best - 1.0)); +} + +#[test] +fn the_log_transform_keeps_the_order_and_imputes_the_worst() { + let values = [Some(3.0), None, Some(10.0), Some(3.5), Some(1e6), Some(4.0)]; + let targets = transform(&values, Output::Log).unwrap(); + // δ is the first quartile of the distances above the best, 0.5 here: the ⌊5/4⌋-th of + // 0, 0.5, 1, 7 and 999997 + assert_eq!(targets[0], 0.5_f64.ln()); + assert_eq!(targets[3], 1.0_f64.ln()); + assert!(targets[0] < targets[3] && targets[3] < targets[5] && targets[5] < targets[2]); + assert_eq!(targets[1], targets[4]); + let targets = transform(&values, Output::Standardize).unwrap(); + assert_eq!(targets, [3.0, 1e6, 10.0, 3.5, 1e6, 4.0]); + assert_eq!(transform(&[None, None], Output::Log), None); + // every value the best: δ is 1 + assert_eq!( + transform(&[Some(2.0), Some(2.0)], Output::Log), + Some(vec![0.0, 0.0]) + ); +} + +#[test] +fn a_small_lattice_is_searched_exhaustively() { + // 12 points: the design takes 8, then every point once, then the search has finished + let integer = Integer::new([0..=3, -1..=1]).unwrap(); + let mut bo = Bo::builder(integer).minimize().seed(1).build().unwrap(); + let f = |x: &crate::genome::Integers| ((x[0] - 2) * (x[0] - 2) + x[1] * x[1]) as f64; + let mut seen = Vec::new(); + while !bo.is_finished() { + let asked: Vec<_> = bo.ask().iter().cloned().collect(); + assert!(!asked.is_empty()); + for genome in &asked { + assert!(!seen.contains(genome)); + seen.push(genome.clone()); + } + let fitness: Vec = asked.iter().map(|x| Fitness::new(f(x))).collect(); + bo.tell(&fitness).unwrap(); + } + assert_eq!(seen.len(), 12); + assert_eq!(bo.best().unwrap().genome()[..], [2, 0]); + assert!(bo.ask().is_empty()); +} diff --git a/src/algorithm/steady.rs b/src/algorithm/steady.rs index 59b7a4c4..1596563e 100644 --- a/src/algorithm/steady.rs +++ b/src/algorithm/steady.rs @@ -1,7 +1,7 @@ //! A steady-state genetic algorithm that takes results one at a time and in any order, for //! asynchronous evaluation. -use crate::engine::GenomeHashing; +use crate::engine::{Evaluations, GenomeHashing, Provided, Wanted}; use crate::genome::{Genome, Representation}; use crate::operator::{Crossover, Mutate, Select}; use crate::rng::Chance; @@ -53,6 +53,54 @@ pub trait Incremental { /// The number of evaluations when the best individual so far was received. fn best_evaluation(&self) -> u64; + + /// Called by the [`AsyncEngine`](crate::engine::AsyncEngine) once at the start of each run, + /// with what the fitness function [provides](crate::engine::FitnessFunction::provides) + /// besides the fitness, as [`Algorithm::prepare`](super::Algorithm::prepare). Nothing by + /// default. + /// + /// # Errors + /// + /// [`Error::InvalidSetting`](crate::Error::InvalidSetting) for an extra the algorithm needs + /// and the fitness function doesn't provide, with the fix in the reason. + #[inline] + fn prepare(&mut self, provided: Provided) -> Result<()> { + let _ = provided; + Ok(()) + } + + /// What the evaluations of the next [proposed](Incremental::propose) genome want besides the + /// fitness, such as the values of the constraints. Nothing by default: the engine then + /// evaluates as it would without extras. + #[inline] + fn wants(&self) -> Wanted { + Wanted::NOTHING + } + + /// The fitness of a genome and the [wanted](Incremental::wants) extras of its evaluation, a + /// single one: what the [`AsyncEngine`](crate::engine::AsyncEngine) gives an algorithm that + /// wants extras, instead of [`receive`](Incremental::receive). By default, + /// `receive(genome, evaluation.fitness()[0])`. + /// + /// # Errors + /// + /// As [`receive`](Incremental::receive), and + /// [`Error::FitnessCount`](crate::Error::FitnessCount) for an evaluation without exactly one + /// fitness. + #[inline] + fn receive_evaluation( + &mut self, + genome: Self::Genome, + evaluation: &Evaluations<'_>, + ) -> Result>> { + match evaluation.fitness() { + [fitness] => self.receive(genome, *fitness), + other => Err(crate::Error::FitnessCount { + expected: 1, + got: other.len(), + }), + } + } } /// A steady-state genetic algorithm for asynchronous evaluation, from diff --git a/src/bin/genoxide/config.rs b/src/bin/genoxide/config.rs index 2c3f3452..aa0eb87e 100644 --- a/src/bin/genoxide/config.rs +++ b/src/bin/genoxide/config.rs @@ -128,8 +128,8 @@ pub struct Fitness { /// per gene on each line. #[serde(default)] pub gradient: bool, - /// The number of inequality constraints g(x) <= 0 whose values, then Jacobian, the program - /// writes after the gradient. + /// The number of inequality constraints g(x) <= 0 whose values the program writes after the + /// value (then, with `gradient`, after the gradient, and their Jacobian). #[serde(default)] pub constraints: usize, } @@ -228,20 +228,7 @@ pub enum Algorithm { /// Random restarts after convergence; none if unset. restarts: Option, }, - Bo { - seed: Option, - /// The points of the initial design; 2(n + 1) if unset. - initial_points: Option, - #[serde(default, deserialize_with = "bo_acquisition")] - acquisition: Option, - kernel: Option, - #[serde(default, deserialize_with = "bo_noise")] - noise: Option, - output: Option, - raw_samples: Option, - acquisition_starts: Option, - hyperparameter_starts: Option, - }, + Bo(Bo), NelderMead { seed: Option, #[serde(default, deserialize_with = "nelder_mead_coefficients")] @@ -502,6 +489,44 @@ pub enum BoNoise { Learned { learned: f64 }, } +/// Bayesian optimization, on a real or an integer genome. +#[derive(Debug, Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +pub struct Bo { + pub seed: Option, + /// The points of the initial design; 2(n + 1) if unset. + pub initial_points: Option, + #[serde(default, deserialize_with = "bo_acquisition")] + pub acquisition: Option, + pub kernel: Option, + #[serde(default, deserialize_with = "bo_noise")] + pub noise: Option, + pub output: Option, + pub raw_samples: Option, + pub acquisition_starts: Option, + pub hyperparameter_starts: Option, + /// The points of each generation after the initial design; 1 if unset. + pub batch: Option, + /// What a point not evaluated yet is taken to be worth; the Kriging believer if unset. + pub fantasy: Option, + /// An asynchronous run: each worker gets a new point as soon as it's done, chosen with the + /// points still being evaluated fantasized. Off if unset: a generation of `batch` points at a + /// time. + #[serde(default)] + pub asynchronous: bool, +} + +/// What Bayesian optimization takes a point not evaluated yet to be worth: `"believer"`, +/// `"liar-min"`, `"liar-mean"` or `"liar-max"`. +#[derive(Clone, Copy, Debug, Deserialize, Serialize)] +#[serde(rename_all = "kebab-case")] +pub enum BoFantasy { + Believer, + LiarMin, + LiarMean, + LiarMax, +} + /// What Bayesian optimization's model fits: `"standardize"` or `"log"`. #[derive(Clone, Copy, Debug, Deserialize, Serialize)] #[serde(rename_all = "kebab-case")] diff --git a/src/bin/genoxide/main.rs b/src/bin/genoxide/main.rs index de045c21..0cff4f09 100644 --- a/src/bin/genoxide/main.rs +++ b/src/bin/genoxide/main.rs @@ -23,9 +23,10 @@ The fitness program (`fitness.command` in the run file) runs once per worker, in directory. It reads a genome per line on stdin, the genes separated by spaces (bits as 0 and 1), and writes a line per genome on stdout: its objective values, then optionally a constraint violation (0 when feasible), separated by spaces, and flushes. `nan` marks a genome that can't -be scored. With `gradient = true` in `[fitness]`, a line is the value, then a derivative per -gene (`genoxide fitness --gradient` for the built-in smooth ones); with `constraints = m` -too, then the values of m constraints g(x) <= 0 and their Jacobian, a row per constraint. +be scored. With `constraints = m` in `[fitness]`, a line is the value, then the values of m +constraints g(x) <= 0. With `gradient = true`, a line is the value, then a derivative per gene +(`genoxide fitness --gradient` for the built-in smooth ones); with `constraints = m` too, +then the values of m constraints g(x) <= 0 and their Jacobian, a row per constraint. A minimal run file: diff --git a/src/bin/genoxide/process.rs b/src/bin/genoxide/process.rs index b3bd4339..00db067c 100644 --- a/src/bin/genoxide/process.rs +++ b/src/bin/genoxide/process.rs @@ -353,6 +353,8 @@ impl Pool { values, extras, ) + } else if self.constraints > 0 { + parse_constrained(&process.line, self.constraints, values, extras) } else { parse(&process.line, values) }; @@ -426,6 +428,48 @@ pub fn parse(line: &str, values: &mut [f64]) -> Result { } } +/// Parses an answer with constraints and without a gradient: the value, then the values of +/// `constraints` inequality constraints g(x) <= 0, into `values` (one value) and the buffer of +/// `extras` if it's wanted. Returns the violation: the sum of the positive constraint values. +pub fn parse_constrained( + line: &str, + constraints: usize, + values: &mut [f64], + extras: Option<&mut Extras<'_>>, +) -> Result { + let mut inequalities = extras.and_then(Extras::inequalities); + let mut count = 0; + let mut violation = 0.0; + for word in line.split_whitespace() { + let number: f64 = word + .parse() + .map_err(|_| format!("`{word}` isn't a number"))?; + match count { + 0 => values[0] = number, + k if k <= constraints => { + violation += at_most(number, 0.0); + if let Some(inequalities) = inequalities.as_deref_mut() { + inequalities[k - 1] = number; + } + } + _ => {} + } + count += 1; + } + if count != 1 + constraints { + let what = if constraints == 1 { + "the constraint's value".to_string() + } else { + format!("the {constraints} constraints' values") + }; + return Err(format!( + "expected {} numbers (the value and {what}), got {count}", + 1 + constraints + )); + } + Ok(violation) +} + /// Parses an answer of the gradient protocol: the value, then a derivative per gene, then the /// values of `constraints` inequality constraints g(x) <= 0 and their Jacobian, a row of a /// derivative per gene for each, into `values` (one value) and the buffers of `extras` that are @@ -555,11 +599,11 @@ impl FitnessFunction for Single<'_> { (value[0], violation) } - // the gradient, with `fitness.gradient`, and the constraints' values and Jacobian, with - // `fitness.constraints` + // the gradient, with `fitness.gradient`, and the constraints' values (and with the gradient, + // their Jacobian), with `fitness.constraints` fn provides(&self) -> Provided { match (self.0.gradient, self.0.constraints) { - (false, _) => Provided::NOTHING, + (false, m) => Provided::NOTHING.with_inequalities(m), (true, 0) => Provided::GRADIENT, (true, m) => Provided::GRADIENT .with_inequalities(m) @@ -621,6 +665,28 @@ mod tests { ); } + #[test] + fn constrained_answers_parse() { + let (mut value, mut g) = ([0.0], [0.0; 2]); + let mut extras = Extras::new(None, Some(&mut g), None); + assert_eq!( + parse_constrained("1.5 -0.5 2\n", 2, &mut value, Some(&mut extras)), + Ok(2.0) + ); + assert_eq!((value, g), ([1.5], [-0.5, 2.0])); + assert_eq!(parse_constrained("4 1 1", 2, &mut value, None), Ok(2.0)); + assert!( + parse_constrained("4 1", 2, &mut value, None) + .unwrap_err() + .contains("expected 3 numbers (the value and the 2 constraints' values), got 2") + ); + assert!( + parse_constrained("4", 1, &mut value, None) + .unwrap_err() + .contains("the value and the constraint's value") + ); + } + #[test] fn gradient_answers_parse() { let (mut value, mut gradient) = ([0.0], [0.0; 2]); diff --git a/src/bin/genoxide/run.rs b/src/bin/genoxide/run.rs index fa93c463..6443adcc 100644 --- a/src/bin/genoxide/run.rs +++ b/src/bin/genoxide/run.rs @@ -76,13 +76,6 @@ pub fn run(run: config::Run, path: &Path, options: Options) -> Result { let directory = path.parent().map(Path::to_path_buf).unwrap_or_default(); let gradient = run.fitness.gradient; let constraints = run.fitness.constraints; - if constraints > 0 && !gradient { - return Err( - "`fitness.constraints` needs `fitness.gradient = true`: the program writes the \ - constraints' values and Jacobian after the gradient" - .to_string(), - ); - } let command = match (run.fitness.command, run.fitness.builtin) { (Some(command), None) if !command.is_empty() => command, (None, Some(name)) => { @@ -102,6 +95,12 @@ pub fn run(run: config::Run, path: &Path, options: Options) -> Result { )); } let builtin = crate::builtin::constraints_of(&name); + if constraints > 0 && !gradient { + return Err(format!( + "`fitness.constraints`: the built-in fitness `{name}` writes its \ + constraints' values with its gradient: set `fitness.gradient = true`" + )); + } if gradient && constraints != builtin { return Err(format!( "`fitness.constraints`: the built-in fitness `{name}` writes {builtin} \ @@ -241,13 +240,17 @@ pub fn run(run: config::Run, path: &Path, options: Options) -> Result { config::Genome::Integer { length, bounds } => { let bounds = bounds.per_gene(length)?; let integer = setting(Integer::new(bounds.map(|[low, high]| low..=high)))?; - with_operators( - integer, - run.algorithm, - &context, - |crossover| ListCrossover::new(crossover, "integer"), - integer_mutation, - ) + match run.algorithm { + // Bayesian optimization on the integer lattice + config::Algorithm::Bo(settings) => bayesian(integer, settings, &context), + algorithm => with_operators( + integer, + algorithm, + &context, + |crossover| ListCrossover::new(crossover, "integer"), + integer_mutation, + ), + } } config::Genome::Real { length, bounds } => { let bounds = bounds.per_gene(length)?; @@ -264,6 +267,96 @@ pub fn run(run: config::Run, path: &Path, options: Options) -> Result { } } +// Bayesian optimization on a real or an integer genome: a generation of `batch` points evaluated +// in parallel, or an asynchronous run +fn bayesian(representation: R, settings: config::Bo, context: &Context) -> Result +where + R: bo::Space + Serialize + DeserializeOwned, + R::Genome: Genes + Serialize + DeserializeOwned + Send + Sync, +{ + let config::Bo { + seed, + initial_points, + acquisition, + kernel, + noise, + output, + raw_samples, + acquisition_starts, + hyperparameter_starts, + batch, + fantasy, + asynchronous: run_asynchronously, + } = settings; + let mut builder = Bo::builder(representation).objective(context.single_objective()?); + if let Some(seed) = seed { + builder = builder.seed(seed); + } + if let Some(points) = initial_points { + builder = builder.initial_points(points); + } + if let Some(acquisition) = acquisition { + builder = builder.acquisition(match acquisition { + config::BoAcquisition::Named(config::BoAcquisitionName::LogEi) => { + bo::Acquisition::LogExpectedImprovement + } + config::BoAcquisition::Named(config::BoAcquisitionName::Ei) => { + bo::Acquisition::ExpectedImprovement + } + config::BoAcquisition::Table(config::BoAcquisitionTable::Pi { xi }) => { + bo::Acquisition::ProbabilityOfImprovement { xi } + } + config::BoAcquisition::Table(config::BoAcquisitionTable::Ucb { beta }) => { + bo::Acquisition::UpperConfidenceBound { beta } + } + }); + } + if let Some(kernel) = kernel { + builder = builder.kernel(match kernel { + config::Kernel::Matern52 => gp::Kernel::Matern52, + config::Kernel::SquaredExponential => gp::Kernel::SquaredExponential, + }); + } + if let Some(noise) = noise { + builder = builder.noise(match noise { + config::BoNoise::Fixed(variance) => gp::Noise::Fixed(variance), + config::BoNoise::Learned { learned } => gp::Noise::Learned { min: learned }, + }); + } + if let Some(output) = output { + builder = builder.output(match output { + config::BoOutput::Standardize => bo::Output::Standardize, + config::BoOutput::Log => bo::Output::Log, + }); + } + if let Some(samples) = raw_samples { + builder = builder.raw_samples(samples); + } + if let Some(starts) = acquisition_starts { + builder = builder.acquisition_starts(starts); + } + if let Some(starts) = hyperparameter_starts { + builder = builder.hyperparameter_starts(starts); + } + if let Some(batch) = batch { + builder = builder.batch(batch); + } + if let Some(fantasy) = fantasy { + builder = builder.fantasy(match fantasy { + config::BoFantasy::Believer => bo::Fantasy::KrigingBeliever, + config::BoFantasy::LiarMin => bo::Fantasy::ConstantLiar(bo::Lie::Min), + config::BoFantasy::LiarMean => bo::Fantasy::ConstantLiar(bo::Lie::Mean), + config::BoFantasy::LiarMax => bo::Fantasy::ConstantLiar(bo::Lie::Max), + }); + } + let bo = setting(builder.build())?; + if run_asynchronously { + asynchronous(bo, context) + } else { + generational(bo, context) + } +} + fn de_strategy(strategy: config::DeStrategy) -> de::Strategy { match strategy { config::DeStrategy::Named(config::DeStrategyName::Rand1) => de::Strategy::Rand1, @@ -595,69 +688,7 @@ fn real_algorithm(real: Real, algorithm: config::Algorithm, context: &Context) - } generational(setting(builder.build())?, context) } - config::Algorithm::Bo { - seed, - initial_points, - acquisition, - kernel, - noise, - output, - raw_samples, - acquisition_starts, - hyperparameter_starts, - } => { - let mut builder = Bo::builder(real).objective(context.single_objective()?); - if let Some(seed) = seed { - builder = builder.seed(seed); - } - if let Some(points) = initial_points { - builder = builder.initial_points(points); - } - if let Some(acquisition) = acquisition { - builder = builder.acquisition(match acquisition { - config::BoAcquisition::Named(config::BoAcquisitionName::LogEi) => { - bo::Acquisition::LogExpectedImprovement - } - config::BoAcquisition::Named(config::BoAcquisitionName::Ei) => { - bo::Acquisition::ExpectedImprovement - } - config::BoAcquisition::Table(config::BoAcquisitionTable::Pi { xi }) => { - bo::Acquisition::ProbabilityOfImprovement { xi } - } - config::BoAcquisition::Table(config::BoAcquisitionTable::Ucb { beta }) => { - bo::Acquisition::UpperConfidenceBound { beta } - } - }); - } - if let Some(kernel) = kernel { - builder = builder.kernel(match kernel { - config::Kernel::Matern52 => gp::Kernel::Matern52, - config::Kernel::SquaredExponential => gp::Kernel::SquaredExponential, - }); - } - if let Some(noise) = noise { - builder = builder.noise(match noise { - config::BoNoise::Fixed(variance) => gp::Noise::Fixed(variance), - config::BoNoise::Learned { learned } => gp::Noise::Learned { min: learned }, - }); - } - if let Some(output) = output { - builder = builder.output(match output { - config::BoOutput::Standardize => bo::Output::Standardize, - config::BoOutput::Log => bo::Output::Log, - }); - } - if let Some(samples) = raw_samples { - builder = builder.raw_samples(samples); - } - if let Some(starts) = acquisition_starts { - builder = builder.acquisition_starts(starts); - } - if let Some(starts) = hyperparameter_starts { - builder = builder.hyperparameter_starts(starts); - } - generational(setting(builder.build())?, context) - } + config::Algorithm::Bo(settings) => bayesian(real, settings, context), config::Algorithm::NelderMead { seed, coefficients, @@ -841,7 +872,7 @@ where config::Algorithm::NelderMead { .. } => { Err("`nelder-mead` needs a real genome".to_string()) } - config::Algorithm::Bo { .. } => Err("`bo` needs a real genome".to_string()), + config::Algorithm::Bo(_) => Err("`bo` needs a real or an integer genome".to_string()), config::Algorithm::Lbfgsb { .. } => Err("`lbfgsb` needs a real genome".to_string()), config::Algorithm::Mma { .. } => Err("`mma` needs a real genome".to_string()), config::Algorithm::FirstOrder { .. } => { diff --git a/src/engine/asynchronous.rs b/src/engine/asynchronous.rs index 2cbe6805..b74e63b5 100644 --- a/src/engine/asynchronous.rs +++ b/src/engine/asynchronous.rs @@ -1,10 +1,11 @@ //! Asynchronous evaluation: every worker gets a new genome as soon as it's done. use super::{ - Checkpoint, FitnessFunction, Info, InfoStore, IntoFitness, NanPolicy, Outcome, Progress, Stop, - StopReason, checkpoint, trace, validate_checkpoint, + BatchExtras, Checkpoint, Evaluations, Extras, FitnessFunction, Info, InfoStore, IntoFitness, + NanPolicy, Outcome, Progress, Stop, StopReason, Wanted, checkpoint, trace, validate_checkpoint, }; use crate::algorithm::Incremental; +use crate::genome::Genome; use crate::observer::{Observer, Snapshot}; use crate::{Error, Fitness, Individual, Result}; use std::any::Any; @@ -47,6 +48,11 @@ pub const MAX_WORKERS: usize = 4096; /// [`Engine`](super::Engine): for the population, the individuals discarded since the last /// generation and the best. /// +/// An algorithm that [wants](Incremental::wants) extras, such as the values of the constraints +/// for a [`Bo`](crate::algorithm::Bo) with constraints, gets them from the fitness function's +/// [`evaluate_with`](FitnessFunction::evaluate_with), each worker with buffers of its own, through +/// [`receive_evaluation`](Incremental::receive_evaluation). +/// /// ``` /// use genoxide::prelude::*; /// @@ -77,13 +83,100 @@ pub struct AsyncEngine<'o, A: Incremental, F> { infos: InfoStore, } -// a finished evaluation: the genome and its fitness with its info, or the panic of the fitness -// function +// a finished evaluation: the genome and its fitness with its info and extras, or the panic of +// the fitness function type Done = ( G, - std::result::Result<(Result, Option), Box>, + std::result::Result<(Result, Option, Rows), Box>, ); +// the extras of one evaluation, as wanted: empty for one not wanted +#[derive(Default)] +struct Rows { + gradient: Vec, + inequalities: Vec, + jacobian: Vec, +} + +impl Rows { + // buffers for a genome of `genes` genes with `constraints` inequality constraints + fn new(wanted: Wanted, genes: usize, constraints: usize) -> Self { + let length = |wanted: bool, length: usize| vec![0.0; if wanted { length } else { 0 }]; + Self { + gradient: length(wanted.gradient, genes), + inequalities: length(wanted.inequalities, constraints), + jacobian: length(wanted.constraint_jacobian, constraints * genes), + } + } + + // whether a wanted extra has a NaN + fn has_nan(&self) -> bool { + [&self.gradient, &self.inequalities, &self.jacobian] + .iter() + .any(|row| row.iter().any(|value| value.is_nan())) + } +} + +// evaluates `genome` with the `wanted` extras of `constraints` inequality constraints +fn evaluate_with>( + fitness: &F, + genome: &G, + wanted: Wanted, + constraints: usize, +) -> (Result, Option, Rows) { + if wanted.is_empty() { + let (result, info) = if fitness.is_batch() { + batch_of_one(fitness.evaluate_batch(&[genome])) + } else { + fitness.evaluate(genome).into_evaluation() + }; + return (result, info, Rows::default()); + } + let genes = genome.len(); + let mut rows = Rows::new(wanted, genes, constraints); + let Rows { + gradient, + inequalities, + jacobian, + } = &mut rows; + let (result, info) = if fitness.is_batch() { + let mut extras = BatchExtras::new( + wanted.gradient.then_some(gradient.as_mut_slice()), + wanted.inequalities.then_some(inequalities.as_mut_slice()), + wanted + .constraint_jacobian + .then_some(jacobian.as_mut_slice()), + genes, + constraints, + ); + batch_of_one(fitness.evaluate_batch_with(&[genome], &mut extras)) + } else { + let mut extras = Extras::new( + wanted.gradient.then_some(gradient.as_mut_slice()), + wanted.inequalities.then_some(inequalities.as_mut_slice()), + wanted + .constraint_jacobian + .then_some(jacobian.as_mut_slice()), + ); + fitness.evaluate_with(genome, &mut extras).into_evaluation() + }; + (result, info, rows) +} + +// the score of a batch of one, which must give one score +fn batch_of_one(mut scores: Vec) -> (Result, Option) { + match (scores.pop(), scores.len()) { + (Some(score), 0) => score.into_evaluation(), + (score, rest) => { + let count = Error::FitnessCount { + expected: 1, + got: rest + usize::from(score.is_some()), + }; + (Err(count), None) + } + } +} + impl<'o, A, F> AsyncEngine<'o, A, F> where A: Incremental, @@ -176,8 +269,11 @@ where /// [`Error::InvalidFitness`] for a negative constraint violation. /// - [`Error::FitnessCount`] if a [`Batch`](super::Batch) doesn't return one score for one /// genome. - /// - The errors of the algorithm's [`receive`](Incremental::receive) and of the checkpoint - /// closure. + /// - [`Error::InvalidSetting`] if the algorithm wants an extra that the fitness function + /// doesn't [provide](FitnessFunction::provides). + /// - The errors of the algorithm's [`prepare`](Incremental::prepare), its + /// [`receive`](Incremental::receive) or + /// [`receive_evaluation`](Incremental::receive_evaluation), and of the checkpoint closure. /// /// The run stops at the first error, once the evaluations in flight are done. If the algorithm /// has run before and a stop condition is already met, or its limit of evaluations was @@ -207,6 +303,19 @@ where }); } validate_checkpoint(&self.checkpoint)?; + let provided = self.fitness.provides(); + self.algorithm.prepare(provided)?; + let wanted = self.algorithm.wants(); + if let Some(missing) = wanted.missing_from(provided) { + return Err(Error::InvalidSetting { + setting: "fitness", + reason: format!( + "the algorithm wants the {missing}, which the fitness function doesn't \ + provide: supply them, e.g. with `Constrained`" + ), + }); + } + let constraints = provided.inequalities; let _span = trace::run::(); let Self { algorithm, @@ -233,6 +342,7 @@ where notified: None, discarded: Vec::new(), infos, + wanted, }; // a run that continues: its stop condition may already be met, or its budget of // evaluations spent before the initial population was complete @@ -280,21 +390,7 @@ where }; let Ok(genome) = job else { return }; let evaluated = panic::catch_unwind(AssertUnwindSafe(|| { - if !fitness.is_batch() { - return fitness.evaluate(&genome).into_evaluation(); - } - // a batch of one, which must give one score - let mut scores = fitness.evaluate_batch(&[&genome]); - match (scores.pop(), scores.len()) { - (Some(score), 0) => score.into_evaluation(), - (score, rest) => { - let count = Error::FitnessCount { - expected: 1, - got: rest + usize::from(score.is_some()), - }; - (Err(count), None) - } - } + evaluate_with(fitness, &genome, wanted, constraints) })); if done.send((genome, evaluated)).is_err() { return; @@ -329,14 +425,14 @@ where if panicked.is_some() || failure.is_some() { continue; } - let (fitness, info) = match evaluated { + let (fitness, info, rows) = match evaluated { Ok(evaluation) => evaluation, Err(payload) => { panicked = Some(payload); continue; } }; - if let Err(error) = driver.accept(genome, fitness, info) { + if let Err(error) = driver.accept(genome, fitness, info, &rows) { failure = Some(error); continue; } @@ -390,6 +486,8 @@ struct Driver<'a, 'o, A: Incremental> { // the info of the population, the discarded individuals, the best and the results since the // last notification infos: &'a mut InfoStore, + // the extras the algorithm wants of each evaluation + wanted: Wanted, } impl Driver<'_, '_, A> { @@ -407,20 +505,49 @@ impl Driver<'_, '_, A> { } } - // gives a result to the algorithm, after the NaN policy, and keeps its info + // gives a result and its extras to the algorithm, after the NaN policy (a NaN in an extra + // makes the whole evaluation invalid, or an error), and keeps its info fn accept( &mut self, genome: A::Genome, fitness: Result, info: Option, + rows: &Rows, ) -> Result<()> { + let fitness = match fitness { + Ok(fitness) if fitness.is_valid() && rows.has_nan() => Err(Error::NanFitness), + result => result, + }; let fitness = match fitness { Ok(fitness) => fitness, Err(Error::NanFitness) if self.nan_policy == NanPolicy::Invalid => Fitness::invalid(), Err(error) => return Err(error), }; let info = info.map(|info| (genome.clone(), info)); - if let Some(individual) = self.algorithm.receive(genome, fitness)? + let discarded = if self.wanted.is_empty() { + self.algorithm.receive(genome, fitness)? + } else { + let fitness = [fitness]; + let wanted = self.wanted; + let genes = genome.len(); + let constraints = if wanted.inequalities { + rows.inequalities.len() + } else { + rows.jacobian.len() / genes.max(1) + }; + let evaluation = Evaluations::with_extras( + &fitness, + wanted.gradient.then_some(rows.gradient.as_slice()), + wanted.inequalities.then_some(rows.inequalities.as_slice()), + wanted + .constraint_jacobian + .then_some(rows.jacobian.as_slice()), + genes, + constraints, + )?; + self.algorithm.receive_evaluation(genome, &evaluation)? + }; + if let Some(individual) = discarded && !self.observers.is_empty() { self.discarded.push(individual); diff --git a/src/lib.rs b/src/lib.rs index b25961ba..da34517b 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -10,7 +10,7 @@ //! | Real numbers in a box, no gradient | [`Cmaes`](algorithm::Cmaes), [`De`](algorithm::De), [`Pso`](algorithm::Pso), [`Es`](algorithm::Es), [`NelderMead`](algorithm::NelderMead) | //! | Smooth functions with a gradient | [`Lbfgsb`](algorithm::Lbfgsb), and [`FirstOrder`](algorithm::FirstOrder) (Adam, momentum) for millions of variables | //! | Many variables, few inequality constraints, with gradients | [`Mma`](algorithm::Mma) (MMA and GCMMA) | -//! | An expensive function: tens to a few hundred evaluations | [`Bo`](algorithm::Bo), Bayesian optimization, with the Gaussian processes of [`model::gp`] | +//! | An expensive function: tens to a few hundred evaluations, in batches or asynchronously, with constraints, of real or integer genes | [`Bo`](algorithm::Bo), Bayesian optimization, with the Gaussian processes of [`model::gp`] | //! | A smooth problem solved in stages (a smoothing, sharpness or penalty changed step by step) | [`Continuation`](algorithm::Continuation) around a local method, its state kept | //! | Several objectives at once | [`Nsga2`](multi::Nsga2), [`Nsga3`](multi::Nsga3), [`Moead`](multi::Moead), [`SmsEmoa`](multi::SmsEmoa) | //! | Programs and formulas | [`gp`]: tree genetic programming | diff --git a/src/model/gp.rs b/src/model/gp.rs index a69e1397..57cdc41e 100644 --- a/src/model/gp.rs +++ b/src/model/gp.rs @@ -4,7 +4,7 @@ //! //! **Unstable for one release.** This module is public from genoxide 0.13 so that a Gaussian //! process can be fitted and queried on its own, but its API and the bits of its fits may still -//! change in 0.14, as the batch and constrained Bayesian optimization of that release use it. +//! change in 0.14, as the surrogate-assisted methods planned after it use it. //! //! A [`GaussianProcess`] models a function `f` of the genes of a [`Real`] genome as //! `f(x) ~ GP(m, σ_f² k(x, x′))` (Rasmussen and Williams, 2006, eq. 2.37), observed with @@ -264,13 +264,25 @@ pub(crate) struct Scaling { impl Scaling { pub(crate) fn new(real: &Real) -> Self { - let bounds = real.bounds(); - let variable = real.variable_genes().to_vec(); - let lower: Vec = variable.iter().map(|&i| *bounds[i].start()).collect(); - let upper: Vec = variable.iter().map(|&i| *bounds[i].end()).collect(); + let bounds: Vec<(f64, f64)> = real + .bounds() + .iter() + .map(|range| (*range.start(), *range.end())) + .collect(); + Self::from_bounds(&bounds) + } + + // the map of genes with the bounds `(lower, upper)`, one pair per gene: the genes whose bounds + // differ are the model's dimensions + pub(crate) fn from_bounds(bounds: &[(f64, f64)]) -> Self { + let variable: Vec = (0..bounds.len()) + .filter(|&i| bounds[i].0 < bounds[i].1) + .collect(); + let lower: Vec = variable.iter().map(|&i| bounds[i].0).collect(); + let upper: Vec = variable.iter().map(|&i| bounds[i].1).collect(); let width = lower.iter().zip(&upper).map(|(l, u)| u - l).collect(); Self { - template: bounds.iter().map(|range| *range.start()).collect(), + template: bounds.iter().map(|&(lower, _)| lower).collect(), variable, lower, upper, @@ -278,6 +290,30 @@ impl Scaling { } } + // the genes of the model's dimensions + pub(crate) fn variable(&self) -> &[usize] { + &self.variable + } + + // the unit-cube coordinates of the genes that `gene` gives by index + pub(crate) fn to_unit_by(&self, gene: impl Fn(usize) -> f64, unit: &mut [f64]) { + for (k, &i) in self.variable.iter().enumerate() { + unit[k] = (gene(i) - self.lower[k]) / self.width[k]; + } + } + + // the value of the model's dimension `k` at the unit-cube coordinate `u`, in its bounds: the + // ends map to the bounds exactly + pub(crate) fn gene_at(&self, k: usize, u: f64) -> f64 { + if u >= 1.0 { + self.upper[k] + } else if u <= 0.0 { + self.lower[k] + } else { + (self.lower[k] + u * self.width[k]).clamp(self.lower[k], self.upper[k]) + } + } + // the model's dimensions: the genes with more than one value pub(crate) fn dims(&self) -> usize { self.variable.len() @@ -298,14 +334,7 @@ impl Scaling { pub(crate) fn to_genome(&self, unit: &[f64]) -> Reals { let mut genome = self.template.clone(); for (k, &i) in self.variable.iter().enumerate() { - let u = unit[k]; - genome[i] = if u >= 1.0 { - self.upper[k] - } else if u <= 0.0 { - self.lower[k] - } else { - (self.lower[k] + u * self.width[k]).clamp(self.lower[k], self.upper[k]) - }; + genome[i] = self.gene_at(k, unit[k]); } Reals::from(genome) } @@ -351,6 +380,8 @@ pub struct GaussianProcess { // the points in the unit cube, count × dims, and the standardization of the values x: Vec, count: usize, + // the standardized values + y: Vec, y_mean: f64, y_scale: f64, // the hyperparameters in the scaled units, and their logarithms (the fitted parameters) @@ -547,6 +578,55 @@ impl GaussianProcess { (self.y_mean, self.y_scale) } + // the standardized values the model was fitted to, in order + pub(crate) fn targets(&self) -> &[f64] { + &self.y + } + + // The model with the points `units` (unit-cube coordinates, a row of `dims` per point) added + // with the standardized `values`, as if they had been observed: the kernel's hyperparameters + // and the standardization kept, the constant mean estimated again (its generalized least + // squares estimate, as at every fit), K_y factored again. None if no jitter factors it. + pub(crate) fn with_points(&self, units: &[f64], values: &[f64]) -> Option { + let dims = self.scaling.dims(); + let mut x = self.x.clone(); + x.extend_from_slice(units); + let mut y = self.y.clone(); + y.extend_from_slice(values); + let count = y.len(); + debug_assert_eq!(x.len(), count * dims); + let data = Data { + x: &x, + y: &y, + count, + dims, + kernel: self.kernel, + noise: Noise::Fixed(self.noise), + }; + let mut workspace = Workspace::new(count); + let jitter = data.factor(&self.length_scales, self.signal, self.noise, &mut workspace)?; + let (mean, log_likelihood) = data.solve(&mut workspace, None); + let Workspace { l, b, .. } = workspace; + Some(GaussianProcess { + kernel: self.kernel, + scaling: self.scaling.clone(), + x, + count, + y, + y_mean: self.y_mean, + y_scale: self.y_scale, + mean, + length_scales: self.length_scales.clone(), + signal: self.signal, + noise: self.noise, + log: self.log.clone(), + jitter, + factor: l, + alpha: b, + log_likelihood, + }) + } + // fits the model to the unit-cube points `x` (count × dims) and `values`, all finite, from // the warm start `warm` (the logarithms of an earlier fit's hyperparameters) or the fixed // first start @@ -579,8 +659,10 @@ impl GaussianProcess { let Some(jitter) = data.factor(&length_scales, signal, noise, &mut workspace) else { return Err(Error::InvalidSetting { setting: "values", - reason: "the kernel matrix of the points doesn't factor, even with a jitter of its diagonal's scale" - .to_string(), + reason: + "the kernel matrix of the points doesn't factor, even with a jitter of its \ + diagonal's scale" + .to_string(), }); }; let (mean, log_likelihood) = data.solve(&mut workspace, None); @@ -590,6 +672,7 @@ impl GaussianProcess { scaling, x, count, + y, y_mean, y_scale, mean, @@ -1101,6 +1184,7 @@ impl GaussianProcessBuilder { scaling, x, count, + y, y_mean, y_scale, mean, diff --git a/tests/algorithms.rs b/tests/algorithms.rs index 20b7069d..1f7a8324 100644 --- a/tests/algorithms.rs +++ b/tests/algorithms.rs @@ -247,7 +247,7 @@ fn portable_run>(algorithm: A) -> Vec { // the same for the methods that need a constraint's values and the gradients: Rosenbrock's // function, with the genes' sum at most 1 -fn portable_constrained_run(algorithm: Mma) -> Vec { +fn portable_constrained_run>(algorithm: A) -> Vec { let rosenbrock = Constrained::differentiable( 1, |x: &Reals, gradient: &mut [f64], g: &mut [f64], jacobian: &mut [f64]| { @@ -428,8 +428,31 @@ fn portable_runs() { .build() .unwrap(), ), + // batches: the points after the first, chosen with the earlier ones fantasized + portable_run( + Bo::builder(real()) + .batch(3) + .fantasy(bo::Fantasy::KrigingBeliever) + .minimize() + .seed(1) + .build() + .unwrap(), + ), + portable_run( + Bo::builder(real()) + .batch(2) + .fantasy(bo::Fantasy::ConstantLiar(bo::Lie::Mean)) + .acquisition(bo::Acquisition::ExpectedImprovement) + .minimize() + .seed(1) + .build() + .unwrap(), + ), + // a model per constraint, the probability of feasibility + portable_constrained_run(Bo::builder(real()).minimize().seed(1).build().unwrap()), ]; - let expected: [[f64; 4]; 16] = [ + + let expected: [[f64; 4]; 19] = [ // L-SHADE [ 0.5886518163542276, @@ -541,10 +564,55 @@ fn portable_runs() { 0.7794047534046049, 0.6827343441068123, ], + // Bayesian optimization in batches of 3, the Kriging believer + [ + 0.5850147650315565, + 0.4831239937078218, + 0.17793827600675627, + -0.01072745023849997, + ], + // Bayesian optimization in batches of 2, EI, the constant liar with the mean + [ + -0.8772643061279055, + 1.5450479136340656, + 2.0635451232757465, + 4.5196908819175645, + ], + // constrained Bayesian optimization, the genes' sum at most 1 + [ + 0.6539353361599387, + 0.41472093863190285, + 0.10930157616421621, + -0.17840095840143988, + ], ]; for (run, expected) in runs.iter().zip(expected) { assert_eq!(run[..], expected, "{run:?}"); } + // Bayesian optimization of integer genes: the last 4 points of 30 generations + let quadratic = |x: &Integers| { + let x: Vec = x.iter().map(|&v| v as f64).collect(); + let (a, b) = (x[1] - x[0] * x[0] / 8.0, 3.0 - x[0]); + 10.0 * a * a + b * b + 0.5 * (x[2] - x[3]) * (x[2] + 1.0) + }; + let integer = Bo::builder(Integer::uniform(4, -12..=12).unwrap()) + .minimize() + .seed(1) + .build() + .unwrap(); + let mut engine = Engine::new(integer, quadratic).stop_when(Stop::generations(30)); + engine.run().unwrap(); + let points: Vec> = engine.algorithm().population().as_slice()[36..] + .iter() + .map(|individual| individual.genome().to_vec()) + .collect(); + let expected = [ + [6, 4, -2, -12], + [8, 8, -5, -12], + [7, 6, -5, -12], + [3, 1, -5, -12], + ]; + assert_eq!(points, expected, "{points:?}"); } #[test] diff --git a/tests/bo.rs b/tests/bo.rs index 234ac180..091d5ef9 100644 --- a/tests/bo.rs +++ b/tests/bo.rs @@ -1,7 +1,9 @@ //! Bayesian optimization: the classic problems of Jones, Schonlau and Welch (1998) reached in tens //! of evaluations, the guarantees of every algorithm, and how invalid points are handled. -use genoxide::algorithm::bo::{Acquisition, Output}; +use genoxide::algorithm::Incremental; +use genoxide::algorithm::bo::{Acquisition, Fantasy, Lie, Output}; +use genoxide::constraint::Constrained; use genoxide::model::gp::Kernel; use genoxide::prelude::*; use genoxide::problems::{Branin, GoldsteinPrice, Hartmann3, Problem, SixHumpCamel}; @@ -334,7 +336,7 @@ fn invalid_settings_are_errors() { other => panic!("not an invalid setting: {other:?}"), }; let fixed = Real::uniform(2, 1.0..=1.0).unwrap(); - assert_eq!(setting(Bo::builder(fixed).build()), "real"); + assert_eq!(setting(Bo::builder(fixed).build()), "representation"); assert_eq!( setting(Bo::builder(real()).initial_points(0).build()), "initial_points" @@ -405,3 +407,415 @@ fn a_fixed_gene_takes_no_part() { let length_scales = engine.algorithm().model().unwrap().hyperparameters(); assert_eq!(length_scales.length_scales()[2], f64::INFINITY); } + +// ---- batches ------------------------------------------------------------------------------------- + +// a Bo of Branin told its initial design, with a batch of `q` points and the `fantasy` +fn after_the_design(seed: u64, q: usize, fantasy: Fantasy) -> Bo { + let mut bo = Bo::builder(Branin.representation()) + .batch(q) + .fantasy(fantasy) + .minimize() + .seed(seed) + .build() + .unwrap(); + let fitness: Vec = bo + .ask() + .iter() + .map(|x| Fitness::new(Branin.evaluate(x))) + .collect(); + bo.tell(&fitness).unwrap(); + bo +} + +const FANTASIES: [Fantasy; 4] = [ + Fantasy::KrigingBeliever, + Fantasy::ConstantLiar(Lie::Min), + Fantasy::ConstantLiar(Lie::Mean), + Fantasy::ConstantLiar(Lie::Max), +]; + +#[test] +fn a_batch_is_new_points_chosen_after_the_fantasies() { + for seed in 1..=3 { + let mut single = after_the_design(seed, 1, Fantasy::default()); + let first: Vec = single.ask().iter().cloned().collect(); + let mut batches: Vec> = Vec::new(); + for fantasy in FANTASIES { + let mut bo = after_the_design(seed, 4, fantasy); + let observed: Vec = bo.population().iter().map(|o| o.genome().clone()).collect(); + let batch: Vec = bo.ask().iter().cloned().collect(); + assert_eq!(batch.len(), 4); + // the first point is the one a batch of 1 asks: the same model, the same streams + assert_eq!(batch[0], first[0], "seed {seed}, {fantasy:?}"); + for (i, point) in batch.iter().enumerate() { + assert!(!observed.contains(point) && !batch[..i].contains(point)); + } + // the model is that of the evaluations, without the fantasies + assert_eq!(bo.model().unwrap().len(), observed.len()); + batches.push(batch); + } + // the lies lead the later points elsewhere + assert_ne!(batches[1][1..], batches[3][1..], "seed {seed}"); + } +} + +#[test] +fn a_batch_counts_rounds_and_evaluations() { + let bo = Bo::builder(Branin.representation()) + .batch(4) + .minimize() + .seed(1) + .build() + .unwrap(); + assert_eq!((bo.batch(), bo.fantasy()), (4, Fantasy::KrigingBeliever)); + let outcome = Engine::new(bo, Branin) + .stop_when(Stop::generations(10)) + .run() + .unwrap(); + // the design of 6, then 10 rounds of 4 + assert_eq!(outcome.evaluations(), 46); + assert!(outcome.best_fitness().score().unwrap() < 0.397_887 + 1e-2); + // the batch changes between generations + let outcome = Engine::new(branin_bo(1), Branin) + .stop_when(Stop::generations(4)) + .control(|bo: &mut Bo, progress| bo.set_batch(2 + progress.generation() as usize)) + .run() + .unwrap(); + assert_eq!(outcome.evaluations(), 6 + 2 + 3 + 4 + 5); + let mut bo = branin_bo(1); + assert!(matches!(bo.set_batch(0), Err(Error::InvalidSetting { .. }))); + assert_eq!(bo.batch(), 1); +} + +#[cfg(feature = "parallel")] +#[test] +fn a_batch_gives_the_same_bits_on_any_number_of_threads() { + let run = |parallel: bool| { + let bo = Bo::builder(Branin.representation()) + .batch(4) + .fantasy(Fantasy::KrigingBeliever) + .minimize() + .seed(3) + .build() + .unwrap(); + let mut engine = Engine::new(bo, Branin) + .parallel(parallel) + .stop_when(Stop::evaluations(30)); + engine.run().unwrap(); + let genomes: Vec> = engine + .algorithm() + .population() + .iter() + .map(|o| o.genome().iter().map(|x| x.to_bits()).collect()) + .collect(); + genomes + }; + let sequential = run(false); + assert_eq!(sequential.len(), 30); + for threads in [1, 2, 8] { + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .unwrap(); + assert_eq!(pool.install(|| run(true)), sequential, "{threads} threads"); + } +} + +// ---- asynchronous evaluation ------------------------------------------------------------------- + +#[test] +fn proposals_fantasize_the_points_being_evaluated() { + let mut bo = branin_bo(2); + // the design first, one point at a time + let design: Vec = (0..6).map(|_| bo.propose()).collect(); + assert_eq!(bo.proposed(), &design[..]); + // results in any order + for x in design.iter().rev() { + let fitness = Fitness::new(Branin.evaluate(x)); + assert!(bo.receive(x.clone(), fitness).unwrap().is_none()); + } + assert!(bo.proposed().is_empty()); + assert_eq!(bo.evaluations(), 6); + // three points while none of them is evaluated: distinct, and new + let pending: Vec = (0..3).map(|_| bo.propose()).collect(); + for (i, x) in pending.iter().enumerate() { + assert!(!design.contains(x) && !pending[..i].contains(x)); + } + assert_eq!(bo.proposed(), &pending[..]); + // a result that is in already is returned, not added + let again = bo + .receive(design[0].clone(), Fitness::new(1.0)) + .unwrap() + .expect("evaluated already"); + assert_eq!(again.genome(), &design[0]); + assert_eq!(bo.evaluations(), 6); + // an invalid genome is an error, and changes nothing + assert!(matches!( + bo.receive(Reals::from(vec![20.0, 0.0]), Fitness::new(1.0)), + Err(Error::InvalidGenome { .. }) + )); + assert_eq!(bo.proposed().len(), 3); +} + +#[test] +fn one_worker_gives_the_same_asynchronous_run_every_time() { + let run = || { + let mut engine = AsyncEngine::new(branin_bo(4), Branin) + .workers(1) + .stop_when(Stop::evaluations(25)); + let outcome = engine.run().unwrap(); + let population = engine.algorithm().population().clone(); + (outcome.into_best(), population) + }; + let (best, population) = run(); + assert_eq!(run(), (best.clone(), population.clone())); + assert_eq!(population.len(), 25); + assert!(best.fitness().unwrap().score().unwrap() < 0.397_887 + 0.05); +} + +#[test] +fn workers_keep_busy_and_reach_the_minimum() { + // results arrive in the order their evaluations end, so runs differ: each reaches the target + for workers in [2, 4] { + let outcome = AsyncEngine::new(branin_bo(1), Branin) + .workers(workers) + .stop_when(Stop::target(0.397_887 + 1e-2).or(Stop::evaluations(80))) + .run() + .unwrap(); + assert_eq!( + outcome.stop_reason(), + StopReason::Target, + "{workers} workers" + ); + } +} + +// ---- constraints --------------------------------------------------------------------------------- + +// Gramacy et al.'s (2016) toy problem: x₁ + x₂ on [0, 1]² with two constraints, the first active +// at the minimum (tests/reference/gramacy_toy.py) +const TOY_MINIMUM: f64 = 0.599_788_052_010_067_6; + +fn toy(x: &Reals, g: &mut [f64]) -> f64 { + let wave = genoxide::math::sin(2.0 * std::f64::consts::PI * (x[0] * x[0] - 2.0 * x[1])); + g[0] = 1.5 - x[0] - 2.0 * x[1] - 0.5 * wave; + g[1] = x[0] * x[0] + x[1] * x[1] - 1.5; + x[0] + x[1] +} + +fn toy_bo(seed: u64, batch: usize) -> Bo { + Bo::builder(Real::uniform(2, 0.0..=1.0).unwrap()) + .batch(batch) + .minimize() + .seed(seed) + .build() + .unwrap() +} + +#[test] +fn constrained_bayesian_optimization_reaches_the_feasible_minimum() { + // every seed of 20 within 1e-5 in at most 39 evaluations, a median of 27 + for seed in 1..=3 { + let mut engine = Engine::new(toy_bo(seed, 1), Constrained::new(2, toy)) + .stop_when(Stop::target(TOY_MINIMUM + 1e-5).or(Stop::evaluations(50))); + let outcome = engine.run().unwrap(); + assert_eq!(outcome.stop_reason(), StopReason::Target, "seed {seed}"); + assert!(outcome.best_fitness().is_feasible()); + let bo = engine.algorithm(); + assert_eq!(bo.constraints(), Some(2)); + assert_eq!(bo.constraint_models().len(), 2); + // the values are kept with each point + let mut g = [0.0; 2]; + toy(bo.population().as_slice()[3].genome(), &mut g); + assert_eq!(bo.constraint_values(3), &g[..]); + // the models interpolate the values: at a point evaluated, clearly feasible or not, the + // probability of feasibility is about 1 or 0 + for (index, individual) in bo.population().iter().enumerate() { + let g = bo.constraint_values(index); + let p = bo + .probability_of_feasibility_at(individual.genome()) + .unwrap(); + if g.iter().all(|&g| g < -0.05) { + assert!(p > 0.99, "{g:?}: {p}"); + } else if g.iter().any(|&g| g > 0.05) { + assert!(p < 0.01, "{g:?}: {p}"); + } + } + assert!( + bo.acquisition_at(outcome.best_genome()) + .unwrap() + .is_finite() + ); + } + // a batch of 3 + let outcome = Engine::new(toy_bo(1, 3), Constrained::new(2, toy)) + .stop_when(Stop::target(TOY_MINIMUM + 1e-4).or(Stop::evaluations(60))) + .run() + .unwrap(); + assert_eq!(outcome.stop_reason(), StopReason::Target); + // and asynchronously, one worker: the values come with each result + let outcome = AsyncEngine::new(toy_bo(1, 1), Constrained::new(2, toy)) + .workers(1) + .stop_when(Stop::target(TOY_MINIMUM + 1e-4).or(Stop::evaluations(60))) + .run() + .unwrap(); + assert_eq!(outcome.stop_reason(), StopReason::Target); +} + +#[test] +fn without_the_constraints_values_the_search_ignores_them() { + // the same problem as a (score, violation) closure: the search goes for the infeasible + // corner, and the best by Deb's rules stays far from the minimum + let tuple = |x: &Reals| { + let mut g = [0.0; 2]; + let score = toy(x, &mut g); + (score, g.iter().map(|&g| g.max(0.0)).sum::()) + }; + let mut engine = Engine::new(toy_bo(1, 1), tuple).stop_when(Stop::evaluations(40)); + let outcome = engine.run().unwrap(); + assert_eq!(engine.algorithm().constraints(), Some(0)); + assert!(outcome.best_fitness().score().unwrap() > TOY_MINIMUM + 1e-3); +} + +#[test] +fn constrained_settings_and_tells() { + // the upper confidence bound can't be weighed by a probability + let ucb = Bo::builder(Real::uniform(2, 0.0..=1.0).unwrap()) + .acquisition(Acquisition::UpperConfidenceBound { beta: 2.0 }) + .seed(1) + .build() + .unwrap(); + let result = Engine::new(ucb, Constrained::new(2, toy)) + .stop_when(Stop::evaluations(10)) + .run(); + assert!(matches!( + result, + Err(Error::InvalidSetting { + setting: "acquisition", + .. + }) + )); + // after a run with constraints, a tell without their values is an error that changes nothing + let mut engine = + Engine::new(toy_bo(1, 1), Constrained::new(2, toy)).stop_when(Stop::evaluations(8)); + engine.run().unwrap(); + let mut bo = engine.into_algorithm(); + assert!(matches!( + bo.set_acquisition(Acquisition::UpperConfidenceBound { beta: 1.0 }), + Err(Error::InvalidSetting { .. }) + )); + let asked = bo.ask().len(); + let fitness = vec![Fitness::new(1.0); asked]; + assert!(matches!( + bo.tell(&fitness), + Err(Error::InvalidSetting { .. }) + )); + assert_eq!(bo.evaluations(), 8); + let values = vec![-1.0; 2 * asked]; + let evaluations = + genoxide::engine::Evaluations::with_extras(&fitness, None, Some(&values), None, 2, 2) + .unwrap(); + bo.tell_evaluations(&evaluations).unwrap(); + assert_eq!(bo.evaluations(), 8 + asked as u64); +} + +#[test] +fn a_cec_2006_problem_by_its_constraint_values() { + use genoxide::problems::cec2006::G24; + // G24: two genes, two constraints, its minimum on the boundary of the feasible region + let minimum = G24.optimum().unwrap().value(); + let bo = Bo::builder(G24.representation()) + .minimize() + .seed(1) + .build() + .unwrap(); + let outcome = Engine::new(bo, G24) + .stop_when(Stop::target(minimum + 1e-4).or(Stop::evaluations(60))) + .run() + .unwrap(); + assert_eq!(outcome.stop_reason(), StopReason::Target); +} + +// ---- integer genes ------------------------------------------------------------------------------- + +#[test] +fn integer_genes_are_searched_on_their_lattice() { + // a quadratic of 4 integer genes in [−10, 10]⁴, 194,481 points, whose continuous minimum + // (2.6, −1.3, 0.4, 3.5) isn't on the lattice: its integer minimum, found by enumeration + let f = |x: &Integers| { + let x: Vec = x.iter().map(|&v| v as f64).collect(); + (x[0] - 2.6) * (x[0] - 2.6) + + 2.0 * (x[1] + 1.3) * (x[1] + 1.3) + + (x[2] - 0.4) * (x[2] - 0.4) + + 0.5 * (x[3] - 3.5) * (x[3] - 3.5) + + 0.3 * (x[0] - 2.6) * (x[1] + 1.3) + }; + let integer = Integer::uniform(4, -10..=10).unwrap(); + let mut minimum = f64::INFINITY; + for a in -10..=10 { + for b in -10..=10 { + for c in -10..=10 { + for d in -10..=10 { + minimum = minimum.min(f(&Integers::from(vec![a, b, c, d]))); + } + } + } + } + for seed in 1..=3 { + let bo = Bo::builder(integer.clone()) + .minimize() + .seed(seed) + .build() + .unwrap(); + let mut engine = + Engine::new(bo, f).stop_when(Stop::target(minimum).or(Stop::evaluations(50))); + let outcome = engine.run().unwrap(); + assert_eq!(outcome.stop_reason(), StopReason::Target, "seed {seed}"); + let population = engine.algorithm().population().as_slice(); + for (i, individual) in population.iter().enumerate() { + assert!(integer.validate(individual.genome()).is_ok()); + let earlier = &population[..i]; + assert!(!earlier.iter().any(|o| o.genome() == individual.genome())); + } + } +} + +#[test] +fn integer_settings() { + let small = Integer::new([0..=1, 0..=2]).unwrap(); + // the default design, 6 points, fits the 6 points of the lattice; 7 don't + assert_eq!( + Bo::builder(small.clone()).build().unwrap().initial_points(), + 6 + ); + let result = Bo::builder(small).initial_points(7).build(); + assert!(matches!( + result, + Err(Error::InvalidSetting { + setting: "initial_points", + .. + }) + )); + // no gene with more than one value is an error + let fixed = Integer::new([3..=3, 3..=3]).unwrap(); + assert!(Bo::builder(fixed).build().is_err()); +} + +#[test] +fn evaluations_without_constraint_values_cant_go_on_with_them() { + // a design told by hand, without the constraints' values: a run with them is an error + let mut bo = toy_bo(1, 1); + let fitness: Vec = bo.ask().iter().map(|x| Fitness::new(x[0] + x[1])).collect(); + bo.tell(&fitness).unwrap(); + let result = Engine::new(bo, Constrained::new(2, toy)) + .stop_when(Stop::evaluations(10)) + .run(); + assert!(matches!( + result, + Err(Error::InvalidSetting { + setting: "fitness", + .. + }) + )); +} diff --git a/tests/checkpoint.rs b/tests/checkpoint.rs index f538336b..70355ff6 100644 --- a/tests/checkpoint.rs +++ b/tests/checkpoint.rs @@ -1197,3 +1197,94 @@ mod continuation { })); } } + +#[test] +fn bayesian_optimization_in_batches_resumes() { + use genoxide::algorithm::bo::{Fantasy, Lie}; + use genoxide::problems::{Branin, Problem}; + for fantasy in [Fantasy::KrigingBeliever, Fantasy::ConstantLiar(Lie::Max)] { + let batch = || { + Bo::builder(Branin.representation()) + .batch(3) + .fantasy(fantasy) + .minimize() + .seed(4) + .build() + .unwrap() + }; + resumes(batch, branin, 3, 7); + } +} + +// Gramacy et al.'s (2016) toy problem, its two constraints' values one by one +fn toy(x: &Reals, g: &mut [f64]) -> f64 { + let wave = genoxide::math::sin(TAU * (x[0] * x[0] - 2.0 * x[1])); + g[0] = 1.5 - x[0] - 2.0 * x[1] - 0.5 * wave; + g[1] = x[0] * x[0] + x[1] * x[1] - 1.5; + x[0] + x[1] +} + +#[test] +fn constrained_bayesian_optimization_resumes() { + use genoxide::constraint::Constrained; + // the constraints' values of every point and their models' warm starts are saved + let toy_bo = || { + Bo::builder(Real::uniform(2, 0.0..=1.0).unwrap()) + .batch(2) + .minimize() + .seed(5) + .build() + .unwrap() + }; + resumes(toy_bo, Constrained::new(2, toy), 3, 8); +} + +#[test] +fn asynchronous_bayesian_optimization_resumes_with_its_pending_points() { + use genoxide::algorithm::Incremental; + use genoxide::engine::Provided; + // prepared at the start of its run, as an engine does + let mut bo = bo(6); + Incremental::prepare(&mut bo, Provided::NOTHING).unwrap(); + let design: Vec = (0..6).map(|_| bo.propose()).collect(); + for x in &design { + bo.receive(x.clone(), Fitness::new(branin(x))).unwrap(); + } + // three points being evaluated, then one result + let pending: Vec = (0..3).map(|_| bo.propose()).collect(); + bo.receive(pending[0].clone(), Fitness::new(branin(&pending[0]))) + .unwrap(); + // the checkpoint holds the two points still being evaluated + let mut resumed: Bo = checkpoint::load(bytes(&bo).as_slice()).unwrap(); + assert_eq!(resumed.proposed(), &pending[1..]); + // a new run's engine has lost their evaluations: they are proposed again first + Incremental::prepare(&mut resumed, Provided::NOTHING).unwrap(); + assert!(resumed.proposed().is_empty()); + assert_eq!(resumed.propose(), pending[1]); + assert_eq!(resumed.propose(), pending[2]); + // then the run goes on as the uninterrupted one, with the same points fantasized + assert_eq!(resumed.propose(), bo.propose()); + assert_eq!(bytes(&resumed), bytes(&bo)); + for x in &pending[1..] { + bo.receive(x.clone(), Fitness::new(branin(x))).unwrap(); + resumed.receive(x.clone(), Fitness::new(branin(x))).unwrap(); + } + assert_eq!(resumed.propose(), bo.propose()); + assert_eq!(bytes(&resumed), bytes(&bo)); +} + +#[test] +fn integer_bayesian_optimization_resumes() { + let quadratic = |x: &Integers| { + let x: Vec = x.iter().map(|&v| v as f64).collect(); + (x[0] - 2.6) * (x[0] - 2.6) + 2.0 * (x[1] + 1.3) * (x[1] + 1.3) + x[2] * x[2] + }; + let integer_bo = || { + Bo::builder(Integer::uniform(3, -5..=5).unwrap()) + .minimize() + .seed(2) + .build() + .unwrap() + }; + resumes(integer_bo, quadratic, 3, 9); +} diff --git a/tests/cli.rs b/tests/cli.rs index a5ca6cd2..449d0dcd 100644 --- a/tests/cli.rs +++ b/tests/cli.rs @@ -849,7 +849,70 @@ length = 4", ) .replace("builtin = \"sphere\"", "builtin = \"one-max\""), ); - assert!(error.contains("`bo` needs a real genome"), "{error}"); + assert!( + error.contains("`bo` needs a real or an integer genome"), + "{error}" + ); + std::fs::remove_dir_all(&directory).unwrap(); +} + +#[test] +fn bayesian_optimization_in_batches_asynchronously_and_on_integers() { + let directory = directory("bo-batch"); + let run = |text: &str| untimed(run(&directory, "run.toml", text, &[]).unwrap()); + let default = run(BO); + // batches of 3, evaluated by the 2 workers: whole generations, past the 25 evaluations + for fantasy in ["believer", "liar-min", "liar-mean", "liar-max"] { + let batch = run(&bo(&format!("batch = 3\nfantasy = \"{fantasy}\""))); + assert_eq!(batch["evaluations"], 27, "{fantasy}"); + assert!( + batch["fitness"].as_f64().unwrap() < 0.1, + "{fantasy}: {batch}" + ); + assert_ne!(batch, default, "{fantasy}"); + } + // asynchronously, each worker given a point as soon as it's done: exactly the budget + let asynchronous = run(&bo("asynchronous = true")); + assert_eq!(asynchronous["evaluations"], 25); + assert!( + asynchronous["fitness"].as_f64().unwrap() < 0.1, + "{asynchronous}" + ); + // on an integer genome, to the sphere's minimum 0 at the origin + let integer = run(&BO.replace( + "type = \"real\"\nlength = 2\nbounds = [-5.0, 5.0]", + "type = \"integer\"\nlength = 3\nbounds = [-6, 6]", + )); + assert_eq!(integer["fitness"], 0.0, "{integer}"); + assert_eq!(integer["genome"], serde_json::json!([0, 0, 0])); + // a program's constraints: the built-in `volume`, read without its gradient by `bo` + let volume = MMA + .replace("length = 100", "length = 2") + .replace("type = \"mma\"", "type = \"bo\"") + .replace("evaluations = 1000", "evaluations = 30"); + let constrained = run(&volume); + assert_eq!(constrained["violation"], 0.0, "{constrained}"); + // the minimum of 1/x₀ + 2/x₁ subject to x₀ + x₁ <= 2: (1 + √2)² / 2 + let minimum = (1.0 + 2f64.sqrt()).powi(2) / 2.0; + let found = constrained["fitness"].as_f64().unwrap(); + assert!(found - minimum < 1e-2, "{constrained}"); + // invalid settings + for (settings, message) in [ + ("batch = 0", "batch"), + ("fantasy = \"liar\"", "`liar-min`"), + ( + "acquisition = { type = \"ucb\", beta = 1.0 }", + "acquisition", + ), + ] { + let text = if message == "acquisition" { + volume.replace("type = \"bo\"", &format!("type = \"bo\"\n{settings}")) + } else { + bo(settings) + }; + let error = run_error(&directory, &text); + assert!(error.contains(message), "{settings}: {error}"); + } std::fs::remove_dir_all(&directory).unwrap(); } @@ -1465,7 +1528,7 @@ fn mma_settings_and_constraints_are_checked() { ); expect( &MMA.replace("gradient = true\n", ""), - "`fitness.constraints` needs `fitness.gradient = true`", + "writes its constraints' values with its gradient: set `fitness.gradient = true`", ); expect( &MMA.replace("constraints = 1\n", ""), diff --git a/tests/reference/gramacy_toy.py b/tests/reference/gramacy_toy.py new file mode 100644 index 00000000..8725119e --- /dev/null +++ b/tests/reference/gramacy_toy.py @@ -0,0 +1,60 @@ +"""The global minimum of Gramacy et al.'s (2016) toy problem, to 30 significant digits with mpmath, +independently of genoxide: the tests and the bo_constrained example embed it, rounded to the +nearest f64. + + pip install mpmath + python tests/reference/gramacy_toy.py + +Gramacy, R. B., Gray, G. A., Le Digabel, S., Lee, H. K. H., Ranjan, P., Wells, G. and Wild, S. M. +(2016). Modeling an augmented Lagrangian for blackbox constrained optimization. Technometrics +58(1): 1-11, section 1: minimize f(x) = x1 + x2 on [0, 1]^2 subject to + + c1(x) = 3/2 - x1 - 2 x2 - sin(2 pi (x1^2 - 2 x2)) / 2 <= 0 + c2(x) = x1^2 + x2^2 - 3/2 <= 0. + +The paper gives the minimizer as about (0.1954, 0.4044) with f about 0.5998, c1 active there and c2 +strictly satisfied, and two local minima, about (0.7197, 0.1411) with f about 0.8609, and (0, 0.75) +on the bound x1 = 0. With c1 active and c2 inactive, a minimum inside the box satisfies c1 = 0 and +the Lagrange condition grad f = -lambda grad c1, that is dc1/dx1 = dc1/dx2: two equations, solved +here by Newton's method from the paper's points. A scan of the box on a grid of 1/2000 then checks +that the first is the global minimum. +""" + +from mpmath import cos, findroot, mp, mpf, nstr, pi, sin + +mp.dps = 30 + + +def c1(x1, x2): + return mpf(3) / 2 - x1 - 2 * x2 - sin(2 * pi * (x1**2 - 2 * x2)) / 2 + + +def c2(x1, x2): + return x1**2 + x2**2 - mpf(3) / 2 + + +def equations(x1, x2): + angle = 2 * pi * (x1**2 - 2 * x2) + dc1_dx1 = -1 - 2 * pi * x1 * cos(angle) + dc1_dx2 = -2 + 2 * pi * cos(angle) + return [c1(x1, x2), dc1_dx1 - dc1_dx2] + + +for start in [(mpf("0.1954"), mpf("0.4044")), (mpf("0.7197"), mpf("0.1411"))]: + x1, x2 = findroot(equations, start) + print(f"x = ({nstr(x1, 20)}, {nstr(x2, 20)}), f = {nstr(x1 + x2, 20)} ({float(x1 + x2)!r}), " + f"c2 = {nstr(c2(x1, x2), 6)}") + +# the least x1 + x2 over the feasible points of a grid of 1/2000 +n = 2000 +least = None +for i in range(n + 1): + x1 = mpf(i) / n + for j in range(n + 1): + x2 = mpf(j) / n + if least is not None and x1 + x2 >= least: + break + if c1(x1, x2) <= 0 and c2(x1, x2) <= 0: + least = x1 + x2 + break +print(f"the grid's least feasible f: {nstr(least, 6)}")