From a8feee91eefb097a4bbe893d9732d83af01094f8 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sun, 24 May 2026 19:31:43 +0200 Subject: [PATCH 01/65] Refresh emcee plan for post-merge surface --- docs/dev/plans/emcee-minimizer.md | 574 +++++++++++++++++++++--------- 1 file changed, 398 insertions(+), 176 deletions(-) diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index b8f056a2d..0aa3cc1de 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -6,98 +6,172 @@ ## Prerequisite -This plan depends on the -[`minimizer-category-consolidation`](../adrs/accepted/minimizer-category-consolidation.md) -ADR and the -[`switchable-category-owned-selectors`](../adrs/accepted/switchable-category-owned-selectors.md) -ADR. Both are accepted and the implementing work is merged; the -`Analysis.minimizer` + `Analysis.minimizer.type` surface emcee builds on -is in place. +This plan depends on three accepted ADRs, all merged on `develop`: + +- [`minimizer-category-consolidation`](../adrs/accepted/minimizer-category-consolidation.md) + — unified `minimizer` category with `BayesianMinimizerBase` and the + `BumpsDreamMinimizer` Bayesian precedent. +- [`switchable-category-owned-selectors`](../adrs/accepted/switchable-category-owned-selectors.md) + — `analysis.minimizer.type = 'X'` is the writable surface; no + owner-level `._type` shims. +- [`minimizer-input-output-split`](../adrs/accepted/minimizer-input-output-split.md) + — fit-filled outputs live on a paired `analysis.fit_result` + instance (`LeastSquaresFitResult` or `BayesianFitResult`), not on + the minimizer. + +emcee inherits the entire paired surface for free: +`BayesianMinimizerBase._fit_result_class = BayesianFitResult`, so +`Analysis._swap_minimizer` instantiates both +`EmceeMinimizer` (inputs) and `BayesianFitResult` (outputs) +atomically. ## ADR -Implements the emcee follow-on described in §1, §5, §6 and §9 of -[`docs/dev/adrs/accepted/minimizer-category-consolidation.md`](../adrs/accepted/minimizer-category-consolidation.md) -(after the prerequisite plan promotes it). No new ADR is required; this -plan is a direct application of the rules already accepted there. +This plan implements the emcee follow-on described in §1, §5, §6 and +§9 of +[`docs/dev/adrs/accepted/minimizer-category-consolidation.md`](../adrs/accepted/minimizer-category-consolidation.md). +No new ADR is required. -If implementation uncovers a design question not covered by the ADR (for -example, resume semantics on parameter-set mismatch), stop and ask +If implementation uncovers a design question not covered by the +ADRs (e.g. resume semantics on parameter-set mismatch, or +`InitializationMethodEnum` ↔ +`DreamPopulationInitializationEnum` reconciliation), stop and ask before proceeding. ## Branch and PR -- Branch: `feature/emcee-minimizer`. Do not push unless asked. +- Branch: `emcee-minimizer`. Do not push unless asked. - Each step in §"Implementation steps (Phase 1)" must be staged with explicit paths and committed locally **before** moving to the next - step. -- After P1.7, stop and wait for the user review gate before starting + step. See `.github/copilot-instructions.md` → **Commits**. +- After P1.8, stop and wait for the user review gate before starting Phase 2. -## Decisions already made (from the ADR) - -1. emcee is exposed as a new concrete `minimizer` class - (`EmceeMinimizer`) registered under - `MinimizerTypeEnum.EMCEE = 'emcee'`. -2. Sampler settings reuse the verbose attribute names from ADR §5 - (`sampling_steps`, `burn_in_steps`, `thinning_interval`, - `population_size`, `parallel_workers`, `initialization_method`, - `random_seed`) with an emcee-specific addition: `proposal_moves`. -3. Resume uses emcee's `HDFBackend` against the `/emcee_chain` group of - the same `analysis/results.h5` file used by the snapshot writer. No - separate sidecar file. A non-resume `fit()` follows the prerequisite - plan's lifecycle and **truncates** `results.h5` (after the standard - warning); resume opens it in append mode. -4. The `fit()` action accepts an explicit `resume=True, extra_steps=N` - pair when the active minimizer supports incremental sampling. For +## Decisions already made (from the accepted ADRs) + +1. emcee is a new concrete Bayesian minimizer registered under + `MinimizerTypeEnum.EMCEE = 'emcee'`. Selected via + `project.analysis.minimizer.type = 'emcee'`. +2. emcee inherits two paired surfaces from `BayesianMinimizerBase`: + - **Settings** (writable inputs): `sampling_steps`, + `burn_in_steps`, `thinning_interval`, `population_size`, + `parallel_workers`, `initialization_method`, `random_seed` — all + declared by `BayesianMinimizerBase.__init__`. emcee may override + class-level defaults (e.g. `sampling_steps=5000`, + `population_size=32`) and adds one emcee-specific input + `proposal_moves`. + - **Outputs** (fit-filled, internal `_set_*`): `acceptance_rate_mean`, + `gelman_rubin_max`, `effective_sample_size_min`, + `best_log_posterior`, `point_estimate_name`, + `sampler_completed`, `credible_interval_inner/outer` — already + on `BayesianFitResult`, not on the minimizer. emcee's projection + writer calls `self.fit_result._set_*` (not `self.minimizer._set_*`). +3. Verbose attribute names map to emcee native kwargs via + `EmceeMinimizer._native_key_map` (overrides the DREAM-style + defaults on `BayesianMinimizerBase._native_key_map`): + + | Verbose | emcee native | + | --- | --- | + | `sampling_steps` | `nsteps` | + | `burn_in_steps` | `nburn` | + | `thinning_interval` | `thin` | + | `population_size` | `nwalkers` | + | `parallel_workers` | `pool` | + | `initialization_method` | (custom — see §6) | + | `random_seed` | `random_seed` | +4. Resume uses emcee's `HDFBackend` against the `/emcee_chain` group + of the same `analysis/results.h5` file used by the snapshot + writer. No separate sidecar file. A non-resume `fit()` follows + the prerequisite plan's lifecycle and **truncates** `results.h5` + (after the standard warning); resume opens it in append mode. +5. `Analysis.fit()` gains an optional `resume=True, extra_steps=N` + call shape for minimizers that support incremental sampling. For other minimizers, passing `resume=True` raises immediately. -5. emcee outputs translate to the existing `BayesianFitResults` shape - exactly as DREAM does — same `PosteriorSamples`, - `PosteriorParameterSummary`, etc. — so plotting and display code - needs no specialization. +6. emcee outputs translate to the existing `BayesianFitResults` + runtime shape (plural — note the singular `BayesianFitResult` is + the persistence category, not the runtime object) exactly as + DREAM does — same `PosteriorSamples`, `PosteriorParameterSummary`, + etc. Plotting and display code needs no specialization. +7. Two `EmceeMinimizer` classes coexist with the same DREAM + precedent: + - `src/easydiffraction/analysis/categories/minimizer/emcee.py` + — the persisted **category** class (`BayesianMinimizerBase` + subclass) used for CIF persistence and the user-facing setter + surface. + - `src/easydiffraction/analysis/minimizers/emcee.py` — the live + **engine** class registered with the engine + `MinimizerFactory`. Holds the `emcee.EnsembleSampler` instance + and runs the sampler. + This mirrors the existing `BumpsDreamMinimizer` split between + `categories/minimizer/bumps_dream.py` and + `minimizers/bumps_dream.py`. ## Open questions -- **Resume after parameter-set change.** If the user fits, then edits - which parameters are free, then calls `fit(resume=True, ...)`, emcee's - HDFBackend will fail because the dimensionality changed. Plan default: - detect mismatch and raise with a clear message asking the user to - start a fresh run. Confirm during P1.4. -- **Resume after a non-emcee fit.** If the user runs DREAM, then sets - `minimizer_type = 'emcee'`, then calls `fit(resume=True, ...)`, the - `/emcee_chain` group will be missing. Plan default: raise a clear - `ValueError` pointing at the prerequisite-plan lifecycle rule ("a new - fit overwrites the file"). -- **Move-mix semantics.** emcee supports proposal-move mixtures (e.g. 70 - % stretch + 30 % differential evolution). The ADR exposes - `proposal_moves` as a single string. Plan default: limit - `proposal_moves` to single-move strings for v1 (`stretch`, `de`, - `de_snooker`, `walk`). Mixtures deferred to a later plan. Record this - in the descriptor's `description=`. +- **Resume after parameter-set change.** If the user fits, then + edits which parameters are free, then calls + `fit(resume=True, ...)`, emcee's `HDFBackend.shape` mismatches the + current parameter count. Plan default: detect mismatch in P1.5 and + raise `ValueError` with a clear "start a fresh run" message. +- **Resume after a non-emcee fit.** If the user runs DREAM, switches + to emcee, and calls `fit(resume=True, ...)`, the `/emcee_chain` + group will be missing. Plan default: raise `ValueError` pointing + at the prerequisite ADR's lifecycle rule ("a new fit overwrites + the file"). +- **`InitializationMethodEnum` ↔ `DreamPopulationInitializationEnum` + reconciliation** (deferred from review-8 F6 of the consolidation + work). emcee uses different init methods (`ball`, `uniform`, + `prior`); DREAM exposes a broader engine-level enum with `EPS`, + `COV`, `LHS`, `RANDOM`. The persisted user-facing enum + (`InitializationMethodEnum`) is narrower + (`LATIN_HYPERCUBE`, `BALL`, `UNIFORM`, `PRIOR`). P1.3 must decide + whether to narrow DREAM's engine enum to match, or accept the + asymmetry. Recommend: keep DREAM's broader engine enum (legacy + DREAM users may pass `EPS`/`COV`/`RANDOM` directly to the engine) + but document that only `LATIN_HYPERCUBE` is persistable for DREAM; + emcee accepts `BALL`/`UNIFORM`/`PRIOR` only. +- **Move-mix semantics.** emcee supports proposal-move mixtures + (e.g. 70 % stretch + 30 % differential evolution). The + consolidation ADR §5 exposes `proposal_moves` as a single string + descriptor. Plan default: limit `proposal_moves` to single-move + strings for v1 (`stretch`, `de`, `de_snooker`, `walk`). Mixtures + deferred to a follow-on plan. Record this in the descriptor's + `description=` text. ## Cleanup opportunities inherited from earlier work -The consolidation work left four cleanup opportunities tracked in -[`docs/dev/issues/open.md`](../issues/open.md) that touch code this plan -will modify. Fold them in while the surrounding code is already being -edited, rather than queuing a separate refactor PR. - -- **F1 — Collapse duplicate predictive-cache-key helpers.** - `Analysis._predictive_cache_key` and - `Plotter._posterior_predictive_key` build the identical string; keep - one canonical helper. Tracked as [open-issue 100](../issues/open.md). -- **F4 — Drop dead branch in `Analysis._fit_state_categories`.** Both - branches return the same list since the Bayesian categories were - absorbed. Tracked as [open-issue 101](../issues/open.md). -- **F7 — Drop compute-and-ignore `result_kind` validation in - `_restore_persisted_fit_state`.** Replace with a validator helper or - move the warning into `fit_result.result_kind` setter. Tracked as - [open-issue 102](../issues/open.md). -- **F10 — Make `_sync_engine_from_minimizer_category` skip-keys +The input/output-split work left four cleanup items still open in +[`docs/dev/issues/open.md`](../issues/open.md) that touch code this +plan modifies. Fold them in opportunistically while the surrounding +code is already being edited; the plan does not block on them. + +- **#100 — Collapse duplicate predictive-cache-key helpers.** + `Analysis._predictive_cache_key` + ([analysis.py:528](../../../src/easydiffraction/analysis/analysis.py)) + and `Plotter._posterior_predictive_key` + ([plotting.py:3823](../../../src/easydiffraction/display/plotting.py)) + build the identical string; keep one canonical helper. P1.6 may + touch the predictive plotting path while validating emcee + posterior emission. +- **#101 — Remove dead branch in + `Analysis._fit_state_categories`.** Both branches return the same + list since the Bayesian categories were absorbed. One-line fix at + [analysis.py:1184-1205](../../../src/easydiffraction/analysis/analysis.py). +- **#102 — Drop compute-and-ignore `result_kind` validation.** + `_restore_persisted_fit_state` + ([serialize.py:590-606](../../../src/easydiffraction/io/cif/serialize.py)) + calls `FitResultKindEnum(result_kind_value)` for its side effect + only. Move the warning into the descriptor setter, or extract a + validator helper. +- **#103 — Make `_sync_engine_from_minimizer_category` skip-keys declarative.** This plan adds `proposal_moves` as a second - engine-level "ambient" key; introduce the `_engine_sync_skip_keys` - frozenset on `MinimizerCategoryBase` before adding the second member. - Tracked as [open-issue 103](../issues/open.md). + engine-level "ambient" key (alongside `random_seed`). The current + magic-string skip at + [analysis.py:1138](../../../src/easydiffraction/analysis/analysis.py) + should become a class-level + `_engine_sync_skip_keys: ClassVar[frozenset[str]] = frozenset(...)` + on `MinimizerCategoryBase` before the second member lands. + Recommend addressing as part of P1.5. When the matching open-issue is fully resolved, move it to [`closed.md`](../issues/closed.md) and update @@ -105,115 +179,256 @@ When the matching open-issue is fully resolved, move it to ## Concrete files likely to change -Created: +### Created -- `src/easydiffraction/analysis/categories/minimizer/emcee.py` (concrete - `EmceeMinimizer` class). +- `src/easydiffraction/analysis/categories/minimizer/emcee.py` — + persisted **category** class `EmceeMinimizer(BayesianMinimizerBase)` + with class-level defaults, `_engine_metadata`, and the overridden + `_native_key_map`. +- `src/easydiffraction/analysis/minimizers/emcee.py` — live + **engine** class `EmceeMinimizer(BayesianMinimizerEngineBase or + equivalent)` registered with `MinimizerFactory`. Holds + `emcee.EnsembleSampler` and the `HDFBackend`. - `tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py`. -- `tests/integration/fitting/test_emcee.py` (cross-check vs DREAM on a - shared toy fit; assert posterior medians agree to within tolerance). +- `tests/unit/easydiffraction/analysis/minimizers/test_emcee.py`. +- `tests/integration/fitting/test_emcee.py` (cross-check vs DREAM on + a shared toy fit; assert posterior medians agree to within + tolerance). - `docs/docs/tutorials/ed-23.py` (emcee + resume tutorial). -Modified: - -- `src/easydiffraction/analysis/minimizers/enums.py` (add - `MinimizerTypeEnum.EMCEE`). -- `src/easydiffraction/analysis/categories/minimizer/__init__.py` (add - the explicit `EmceeMinimizer` import to trigger registration). -- `src/easydiffraction/analysis/categories/minimizer/factory.py` (the - factory may need no change if registration uses `@Factory.register`). -- `src/easydiffraction/analysis/analysis.py` (`fit()` signature gains - `resume: bool = False, extra_steps: int | None = None`; route to the - live engine appropriately). -- `src/easydiffraction/io/results_sidecar.py` (read path: when - `/emcee_chain` is present, expose a small helper to construct an - `emcee.backends.HDFBackend(path, name='emcee_chain', read_only=...)`). -- `pyproject.toml` and `pixi.toml` (add `emcee>=3.1` dependency). +### Modified + +- `src/easydiffraction/analysis/minimizers/enums.py` — add + `MinimizerTypeEnum.EMCEE = 'emcee'`. +- `src/easydiffraction/analysis/categories/minimizer/__init__.py` + — add explicit `EmceeMinimizer` (category) import so registration + fires. +- `src/easydiffraction/analysis/minimizers/__init__.py` (or the + factory's package init) — add explicit `EmceeMinimizer` (engine) + import so engine registration fires. +- `src/easydiffraction/analysis/categories/minimizer/bayesian_base.py` + — only if review-8 F6 reconciliation (open question above) calls + for narrowing the persisted enum surface; otherwise unchanged. +- `src/easydiffraction/analysis/analysis.py` — `fit()` signature + gains `resume: bool = False, extra_steps: int | None = None`; + validation + dispatch to engine. Wire `_engine_sync_skip_keys` + (#103) before adding `proposal_moves` to the ambient set. +- `src/easydiffraction/io/results_sidecar.py` — read path: when + `/emcee_chain` is present, expose a helper to construct an + `emcee.backends.HDFBackend(path, name='emcee_chain', + read_only=True)` for inspection/visualisation. +- `pyproject.toml` and `pixi.toml` — add `emcee>=3.1` dependency. + +### Deleted + +- None. ## Implementation steps (Phase 1) +Mark `[x]` as each step lands. + - [ ] **P1.1 — Add emcee dependency.** Add `emcee>=3.1` to - `pyproject.toml` and `pixi.toml`. Run `pixi install` locally to - verify resolution. Commit: `Add emcee dependency` + `pyproject.toml` and `pixi.toml`. Run `pixi install` locally + to verify resolution. Commit: `Add emcee dependency` -- [ ] **P1.2 — Register `MinimizerTypeEnum.EMCEE`.** Add the enum member - with value `'emcee'`. No other code wiring yet. Commit: +- [ ] **P1.2 — Register `MinimizerTypeEnum.EMCEE`.** Add the enum + member with value `'emcee'` to + `src/easydiffraction/analysis/minimizers/enums.py`. No other + code wiring yet. Commit: `Register emcee minimizer enum value` -- [ ] **P1.3 — Add `EmceeMinimizer` concrete class.** Descriptor setup - follows the prerequisite plan's accepted helper pattern: - class-level defaults for emcee-specific values - (`sampling_steps=5000`, `population_size=32`, …, - `proposal_moves='stretch'`) and instance descriptors constructed - from the Bayesian minimizer helpers. Before wiring emcee, decide - whether DREAM's direct-engine `DreamPopulationInitializationEnum` - remains broader than the persisted `InitializationMethodEnum` - subset or is narrowed to match it. Implement `_native_kwargs()` - mapping to emcee's - `EnsembleSampler.run_mcmc(nsteps=..., progress=..., ...)`. Update - `src/easydiffraction/analysis/categories/minimizer/__init__.py` to - import `EmceeMinimizer` (registration trigger). Commit: - `Add EmceeMinimizer concrete class` - -- [ ] **P1.4 — Implement run + resume via HDFBackend.** In the live - solver layer (the new `Analysis._engine` path introduced in the - prerequisite plan), instantiate - `emcee.backends.HDFBackend(project.analysis_dir / 'results.h5', name='emcee_chain')`. - Lifecycle: - - **New fit** (`fit()` without `resume`): the prerequisite plan's - `Analysis.fit()` truncates `results.h5` _before_ the engine is asked - to sample (P1.10 in that plan). After truncation, the `HDFBackend` - is instantiated against the freshly recreated file and - `EnsembleSampler.run_mcmc(...)` is called. - - **Resume** (`fit(resume=True, extra_steps=N)`): - - require the active minimizer's `MinimizerTypeEnum` to support - resume (currently only `EMCEE`); - - require `results.h5` to exist and contain a `/emcee_chain` group - (raise `FileNotFoundError` / `ValueError` with a clear message - otherwise); - - reload the backend, validate `backend.shape` matches the current - parameter count (raise `ValueError` on mismatch with a clear - message and recommend starting a fresh fit); - - bypass the truncate-and-warn step; - - call - `run_mcmc(initial_state=None, nsteps=N, progress=True, skip_initial_state_check=True)` - to extend the chain. Translate the sampler's state to - `BayesianFitResults` exactly like DREAM, populating - `Parameter.posterior` via the existing helpers. Commit: - `Implement emcee run and resume via HDFBackend` - -- [ ] **P1.5 — Plug emcee outputs into existing posterior pipeline.** - Verify the existing sidecar writer for `/posterior`, - `/distribution_cache`, `/pair_cache`, `/predictive` correctly - picks up emcee results. Adjust only where emcee surfaces data - differently from DREAM (e.g. - `EnsembleSampler.get_chain(flat=False, discard=burn, thin=thin)` - vs the DREAM extraction helper). Cache derivations (KDE, pair - grids) must match the existing format. Commit: - `Route emcee posterior through sidecar pipeline` - -- [ ] **P1.6 — Add `ed-23.py` tutorial.** New notebook source at - `docs/docs/tutorials/ed-23.py`: demonstrate - `analysis.minimizer_type = 'emcee'`, a short run, save, resume - with `extra_steps=`, and a posterior plot. Run - `pixi run notebook-prepare` to generate the `.ipynb`. Commit: - `Add ed-23 emcee tutorial` - -- [ ] **P1.7 — Phase 1 review gate.** Stop and request user review - before Phase 2. +- [ ] **P1.3 — Add `EmceeMinimizer` category class.** New file + `src/easydiffraction/analysis/categories/minimizer/emcee.py`. + `EmceeMinimizer(BayesianMinimizerBase)` declares: + - `type_info` with `tag=MinimizerTypeEnum.EMCEE` and a description. + - `_engine_metadata: ClassVar[dict[str, str]] = {'optimizer_name': + 'emcee', 'method_name': 'stretch'}` (matching the + `BumpsDreamMinimizer` precedent for the + `_restore_fit_results_from_projection` lookup). + - `_native_key_map` override mapping the verbose names to emcee's + native kwargs (see §"Decisions already made" point 3). + - Class-level defaults for emcee-specific values: + `sampling_steps=5000`, `burn_in_steps=1000`, + `thinning_interval=5`, `population_size=32`, + `parallel_workers=0`, `proposal_moves='stretch'`. + - `__init__` constructs descriptors via the inherited helpers + (`_sampling_steps_descriptor(default)`, etc. from + `BayesianMinimizerBase`) and adds a new `proposal_moves` + descriptor with a `MembershipValidator` over the single-move + set (`stretch`, `de`, `de_snooker`, `walk`). + - **Decide the `InitializationMethodEnum` reconciliation** (open + question above) before wiring. `EmceeMinimizer._supported_initialization_methods` + should list `(BALL, UNIFORM, PRIOR)` regardless of the DREAM + decision. + + Update + `src/easydiffraction/analysis/categories/minimizer/__init__.py` + to import `EmceeMinimizer` (registration trigger via + `@MinimizerCategoryFactory.register`). + + The paired `BayesianFitResult` flows automatically because + `BayesianMinimizerBase._fit_result_class = BayesianFitResult`; no + wiring needed in this step. + + Commit: `Add EmceeMinimizer category class` + +- [ ] **P1.4 — Add `EmceeMinimizer` engine class.** New file + `src/easydiffraction/analysis/minimizers/emcee.py`. The engine + class is registered with `MinimizerFactory` and holds the + `emcee.EnsembleSampler` plus an `HDFBackend` attribute. Mirror + the shape of + [`bumps_dream.py BumpsDreamMinimizer`](../../../src/easydiffraction/analysis/minimizers/bumps_dream.py) + — descriptor attributes (`burn`, `thin`, `pop`, `init`, …) + that `Analysis._sync_engine_from_minimizer_category` writes to + from the category's `_native_kwargs()`. The descriptor names + on the engine must match the keys returned by + `EmceeMinimizer._native_kwargs()` (i.e. emcee's native names — + `nsteps`, `nburn`, `nwalkers`, `pool`). + + Engine method shape (sketch): + + ```python + class EmceeMinimizer(BayesianMinimizerEngineBase): + name = MinimizerTypeEnum.EMCEE + method = 'stretch' + + def fit(self, *, structures, experiments, analysis, + resume=False, extra_steps=None, ...): + backend = emcee.backends.HDFBackend( + path=analysis.project.info.path + / 'analysis' / 'results.h5', + name='emcee_chain', + read_only=False, + ) + sampler = emcee.EnsembleSampler( + nwalkers=self.nwalkers, + ndim=len(free_params), + log_prob_fn=log_prob, + pool=self.pool, + moves=..., # from proposal_moves + backend=backend, + ) + if resume: + self._validate_resume(backend, free_params) + sampler.run_mcmc(None, nsteps=extra_steps, + skip_initial_state_check=True, + progress=True) + else: + initial_state = self._initial_state(...) + sampler.run_mcmc(initial_state, + nsteps=self.nsteps, + progress=True) + return self._build_results(sampler, ...) + ``` + + Register with the engine `MinimizerFactory` and update + `src/easydiffraction/analysis/minimizers/__init__.py` (or the + relevant package init) to import the engine class. + + Commit: `Add EmceeMinimizer engine class` + +- [ ] **P1.5 — Wire `fit(resume=True, extra_steps=N)` on + `Analysis`.** In + `src/easydiffraction/analysis/analysis.py`: + - `Analysis.fit()` gains keyword args `resume: bool = False, + extra_steps: int | None = None`. Default behaviour (no resume) + is unchanged. + - When `resume=True`: + - Validate that `self.minimizer.type == + MinimizerTypeEnum.EMCEE.value` (only emcee supports resume in + v1). Raise `ValueError` with a clear message otherwise. + - Require `extra_steps` to be a positive integer. + - **Bypass** the `_warn_results_sidecar_overwrite` step and the + `_clear_persisted_fit_state` reset — both would clobber the + backend. + - Forward `resume=True, extra_steps=N` to the live engine via + the existing `Fitter` plumbing. + + Also address open issue #103: introduce + `_engine_sync_skip_keys: ClassVar[frozenset[str]] = frozenset({'random_seed'})` + on `MinimizerCategoryBase`, and update + `_sync_engine_from_minimizer_category` + ([analysis.py:1134-1146](../../../src/easydiffraction/analysis/analysis.py)) + to use it. `BayesianMinimizerBase` (or `EmceeMinimizer` + directly) overrides the frozenset to include `'proposal_moves'` + if/when the engine consumes that key under a different name than + the category attribute. + + Commit: `Wire emcee resume into Analysis.fit` + +- [ ] **P1.6 — Route emcee outputs into the existing fit_result and + sidecar pipeline.** Verify the existing + `_store_posterior_fit_projection` + ([analysis.py](../../../src/easydiffraction/analysis/analysis.py)) + writes to `self.fit_result._set_*` (the + `BayesianFitResult` instance auto-paired with the + `EmceeMinimizer`) and that the `/posterior`, + `/distribution_cache`, `/pair_cache`, `/predictive` groups in + `results.h5` receive emcee output without modification. + Adjust only where emcee surfaces data differently from + DREAM (e.g. `EnsembleSampler.get_chain(flat=False, + discard=burn, thin=thin)` vs the DREAM extraction helper). + Cache derivations (KDE, pair grids) reuse the existing + pipeline. + + Opportunistic cleanup: address open issue #100 if the + predictive plotting path is being touched anyway. Collapse + `Analysis._predictive_cache_key` and + `Plotter._posterior_predictive_key` into one canonical helper. + + Commit: `Route emcee posterior through fit_result and sidecar` + +- [ ] **P1.7 — Add `ed-23.py` tutorial.** New notebook source at + `docs/docs/tutorials/ed-23.py` covering: + - `project.analysis.minimizer.type = 'emcee'` (post-switchable + syntax). + - `project.analysis.minimizer.sampling_steps = 1000` (small for + tutorial speed). + - `project.analysis.fit()` and a posterior plot. + - `project.save()`. + - `project.analysis.fit(resume=True, extra_steps=500)` continues + the chain. + - Final posterior plot after resume. + + Run `pixi run notebook-prepare` to regenerate the `.ipynb`. + + Verification grep (must return empty against + `docs/docs/tutorials/ed-23.py`): + + ``` + git grep -nE '\banalysis\.minimizer_type\b|\bminimizer\.runtime_seconds\b|\bminimizer\.gelman_rubin_max\b' docs/docs/tutorials/ed-23.py + ``` + + Commit: `Add ed-23 emcee tutorial` + +- [ ] **P1.8 — Phase 1 review gate.** No code change. Stop and + request user review. After approval, proceed to Phase 2. ## Verification (Phase 2) -Same log-capture pattern as the prerequisite plan; commands repeated for -completeness. +Each command captures its log with a zsh-safe exit-code variable as +required by `.github/copilot-instructions.md` → **Workflow**. - [ ] **P2.1 — Add unit + integration tests.** - `tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py`: - descriptor defaults, native-key mapping, swap behavior, resume + category-class descriptor defaults; `_native_key_map` override; + pairing with `BayesianFitResult`; swap behavior; resume parameter-set-mismatch error path (no real sampler). - - `tests/integration/fitting/test_emcee.py`: end-to-end fit on a small - synthetic problem; resume; assert posterior medians agree with a - DREAM run within tolerance. + - `tests/unit/easydiffraction/analysis/minimizers/test_emcee.py`: + engine-class registration, descriptor defaults, native kwargs + plumbing. + - `tests/integration/fitting/test_emcee.py`: end-to-end fit on a + small synthetic problem; resume; assert posterior medians + agree with a DREAM run within tolerance. + + Layout check: + + ``` + pixi run test-structure-check > /tmp/easydiffraction-test-structure-check.log 2>&1; \ + test_structure_check_exit_code=$?; \ + tail -n 200 /tmp/easydiffraction-test-structure-check.log; \ + exit $test_structure_check_exit_code + ``` - [ ] **P2.2 — Auto-fixes and static checks.** @@ -224,6 +439,8 @@ completeness. exit $fix_exit_code ``` + Then: + ``` pixi run check > /tmp/easydiffraction-check.log 2>&1; \ check_exit_code=$?; \ @@ -250,6 +467,7 @@ completeness. ``` - [ ] **P2.5 — Script tests.** + ``` pixi run script-tests > /tmp/easydiffraction-script-tests.log 2>&1; \ script_tests_exit_code=$?; \ @@ -263,23 +481,27 @@ completeness. **Description (user-facing):** -EasyDiffraction adds emcee — a widely-used affine-invariant MCMC sampler -— as a second Bayesian fitter. It is selected exactly like the existing -samplers: +EasyDiffraction adds emcee — a widely-used affine-invariant MCMC +sampler — as a second Bayesian fitter. It is selected exactly like +the existing samplers, via the uniform switchable-category surface: -- `project.analysis.minimizer_type = 'emcee'` -- `project.analysis.minimizer.sampling_steps = 5000` -- `project.analysis.fit()` +```python +project.analysis.minimizer.type = 'emcee' +project.analysis.minimizer.sampling_steps = 5000 +project.analysis.fit() +``` Long runs can be **resumed** without starting over: -- `project.analysis.fit(resume=True, extra_steps=2000)` +```python +project.analysis.fit(resume=True, extra_steps=2000) +``` -emcee's chain state lives inside the same `analysis/results.h5` file as -the other posterior data, so saving and reopening a project is a +emcee's chain state lives inside the same `analysis/results.h5` file +as the other posterior data, so saving and reopening a project is a single-file affair. Plots, parameter posteriors, and tables work the -same as for the existing DREAM sampler, so switching between samplers to -cross-check results is straightforward. +same as for DREAM, so switching between samplers to cross-check +results is straightforward. -A new tutorial (`ed-23`) walks through a short run, saving the project, -and resuming for additional steps. +A new tutorial (`ed-23`) walks through a short run, saving the +project, and resuming for additional steps. From a5e832cb6dd08323fb0b3a87ab9ebf4bbeab76af Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sun, 24 May 2026 19:50:58 +0200 Subject: [PATCH 02/65] Document /draft-adr /review-adr /draft-plan /review-plan shortcuts --- .github/copilot-instructions.md | 137 ++++++++++++++++++++++++++++++++ 1 file changed, 137 insertions(+) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 1bf989d5c..9b0de7264 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -251,3 +251,140 @@ When asked to create a plan: the plan, and a pointer to the affected plan section. After updating the plan, also update the reply if a numbered step shifts so that cross-references stay accurate. + +## Agent Shortcuts + +When the user enters one of these literal keywords at the start of a +message, execute the matching task instead of asking for the full +instructions every time. The keyword is the entire trigger; arguments +follow on the same line. + +Common preamble for every shortcut (run once at task start): + +- Ask the user, in one batch, for any permission grants needed to + run unattended for the full task. At minimum: `Bash` with + `run_in_background` for polling, `Edit`/`Write` on `docs/`, the + shell primitives `git`, `until`, `sleep`, `ls`, `grep`. Cite this + section so the user knows why. +- Stay on the current branch. Do not switch or create branches. +- Polling cadence is 60 s. Use Bash with `run_in_background` and an + `until [ -f ]; do sleep 60; done` body so the harness + notifies you when the awaited file appears. Do not poll inline. +- Filename suffixes follow the existing convention: + `_review-N.md` and `_reply-N.md` next to the parent + ADR or plan, where `N` is one greater than the highest existing + number for that stem (starting at 1). +- Each shortcut runs autonomously. Do not pause for confirmation + between rounds; auto-apply every finding. Only stop when the + termination condition for that shortcut is met, or the user sends + an explicit message asking you to stop or change direction. + +### `/draft-adr ` + +Act as the ADR author. Draft an ADR suggestion, then respond to +incoming reviews in a polling loop. + +**Setup (once):** + +1. Run the common preamble. +2. Pick a flat lowercase-dash slug from ``. Save the ADR at + `docs/dev/adrs/suggestions/.md` using the project's ADR + template (Status: Proposed; Context; Decision; Consequences; + Alternatives Considered; Deferred Work as needed). +3. Start polling for `_review-1.md` next to the ADR. + +**Loop (per tick):** + +- When `_review-N.md` appears: read it, update the ADR to + address every finding, and write `_reply-N.md` with one + section per finding (verdict + action taken + pointer to the + affected ADR section). +- Start polling for `_review-(N+1).md`. +- **Do not commit.** Leave every edit in the worktree as modified + or untracked. The reviewer side (`/review-adr`) is responsible + for the final commit. + +**Termination:** only on an explicit user message asking you to +stop or change direction. The loop never self-terminates. + +### `/review-adr []` + +Act as the ADR reviewer. Review an ADR suggestion in a polling loop +until all findings are addressed. + +**Setup (once):** + +1. Run the common preamble. +2. Identify the target ADR. If `` is given, target + `docs/dev/adrs/suggestions/.md`. Otherwise monitor + `docs/dev/adrs/suggestions/` for a `.md` with no existing + `_review-*.md`. If none exists, report "no ADR has + appeared yet" and start polling for it. +3. When the ADR appears, run a static review per + [`.github/copilot-instructions.md`](.github/copilot-instructions.md) + → **Change Discipline** plan-review rule. **Do not run tests, + lint, build, formatters, or any `pixi` command — ADR reviews are + static reads only.** Save the review at `_review-1.md` + next to the ADR. + +**Loop (per tick):** + +- Poll for `_reply-N.md` matching the most recent review. +- When the reply appears, re-read the ADR against the new reply + and every prior review/reply. Pick exactly one branch: + - **Findings remain:** write `_review-(N+1).md` listing + only the open findings, then poll for the next reply. + - **All findings addressed:** write a final + `_review-(N+1).md` stating "no findings; ADR is ready", + then run the termination cleanup below and stop. + +**Termination cleanup (only when the final clean review is +written):** + +1. `git rm` every `_review-*.md` and `_reply-*.md` + next to the ADR (including the final clean review just + written). +2. `git add` the cleaned-up ADR. +3. Commit with message `Add ADR suggestion` (or an + equivalent imperative ≤72 chars). +4. Report the commit hash and stop. + +### `/draft-plan []` + +Act as the implementation-plan author. Same loop as `/draft-adr`, +applied to an implementation plan instead of an ADR. + +- Source ADR: if `` is given, target the ADR at + `docs/dev/adrs/accepted/.md` (or `suggestions/` if the ADR + is not yet promoted). Otherwise pick the newest accepted ADR + matching the most recent `/review-adr` cycle. Use the same slug + for the plan. +- Save the plan at `docs/dev/plans/.md` using the project's + plan template per + [`.github/copilot-instructions.md`](.github/copilot-instructions.md) + → **Planning**: ADR cross-reference, branch + PR notes, + Decisions, Open questions, Concrete files, Phase 1 steps with + status checklist, Phase 2 verification commands using the + zsh-safe log-capture pattern, and a Suggested Pull Request + section. +- Loop behaviour identical to `/draft-adr`: poll + `docs/dev/plans/_review-N.md`; write + `_reply-N.md`; **no commits in the loop**. + +**Termination:** only on an explicit user message asking you to +stop or change direction. + +### `/review-plan []` + +Act as the implementation-plan reviewer. Same loop as `/review-adr`, +applied to an implementation plan. + +- Target: if `` is given, target + `docs/dev/plans/.md`. Otherwise monitor `docs/dev/plans/` + for the newest plan matching the most recent `/draft-plan` + cycle. +- Static reads only — same "no tests / lint / build / formatters / + pixi" rule applies during plan reviews. +- Loop and termination identical to `/review-adr`, with the final + commit message `Add implementation plan` (or an + equivalent imperative ≤72 chars). From 421bfa28b0c90d069c317c6d8c971d5c43f4cef1 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sun, 24 May 2026 20:04:07 +0200 Subject: [PATCH 03/65] Sync copilot-instructions with current API and branch conventions --- .github/copilot-instructions.md | 40 ++++++++++++++++++++++++--------- 1 file changed, 30 insertions(+), 10 deletions(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 9b0de7264..e6da15b9c 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -60,13 +60,27 @@ - Concrete classes use `@Factory.register`. Each package's `__init__.py` must explicitly import every concrete class to trigger registration — always update it when adding a class. -- Switchable categories (factory-swappable at runtime) follow this fixed - API on the owner (experiment / structure / analysis): `` - (read-only), `_type` (getter+setter), - `show_supported__types()`, `show_current__type()`. - The owner owns the type setter and show methods; show methods delegate - to `Factory.show_supported(...)`. Required even if only one - implementation exists. +- Switchable categories (factory-swappable at runtime) follow the + category-owned selector contract from + [`switchable-category-owned-selectors.md`](../docs/dev/adrs/accepted/switchable-category-owned-selectors.md): + the owner exposes `` (read-only attribute on the owner), + and the category itself exposes `.type` (writable + property) and `.show_supported()`. There are no + owner-level `._type` setters and no owner-level + `show_supported__types()` / `show_current__type()` + methods. The owner provides a private `_swap_` hook that + the category's `type` setter calls through a back-reference; + inside the hook the owner replaces the category instance + (Family A), rebinds the live engine (Family B), or activates + sibling categories (Family C) — the user-facing surface stays + uniform. Required even if only one implementation exists. +- Result-output categories paired with a switchable input category + (today: `analysis.fit_result` paired with `analysis.minimizer` + via `._fit_result_class`) are **internal pairs**, + not user-facing switchables: they do not expose `type` or + `show_supported()`; the owner swaps them in lockstep with the + paired input category. See + [`minimizer-input-output-split.md`](../docs/dev/adrs/accepted/minimizer-input-output-split.md). - Categories are flat siblings within their owner. Never nest a category as a child of another category of a different type; cross-reference via IDs instead. @@ -143,6 +157,11 @@ etc.). - Each change is atomic and single-commit-sized: make one change, suggest the commit message, then stop and wait for confirmation. + Exception: when the user invokes an **Agent Shortcut** (see that + section), the matching loop runs autonomously per its own + termination rule — neither a per-commit pause nor a per-tick + pause applies inside that loop. The default applies again as soon + as the shortcut terminates. - When in doubt, ask. ## Commits @@ -207,9 +226,10 @@ When asked to create a plan: example, `docs/dev/adrs/suggestions/foo.md` maps to `docs/dev/plans/foo.md`. If a plan has no corresponding ADR or spans multiple ADRs, choose a concise feature slug and list all related ADRs - in the plan. Use the same `` for the implementation - branch (`feature/`). Do not push the branch unless - asked. + in the plan. Use the same `` as a **flat-slug + implementation branch** off `develop` (no `feature/` prefix — + e.g. `emcee-minimizer`, not `feature/emcee-minimizer`). PRs target + `develop`, not `master`. Do not push the branch unless asked. - Include a status checklist with `[ ]` items; mark `[x]` as completed during implementation. - Apply the two-phase workflow (Phase 1 implementation, Phase 2 From 82cca793727608079f426d1b929c71550768e55c Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sun, 24 May 2026 20:33:49 +0200 Subject: [PATCH 04/65] Refine shortcut triggers and loop behavior --- .github/copilot-instructions.md | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index e6da15b9c..575444140 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -274,10 +274,13 @@ When asked to create a plan: ## Agent Shortcuts -When the user enters one of these literal keywords at the start of a -message, execute the matching task instead of asking for the full -instructions every time. The keyword is the entire trigger; arguments -follow on the same line. +When the user's message starts with one of these literal keywords, +treat it as an operational command, not as ordinary prose. Execute the +matching task instead of asking for the full instructions every time. +The first non-whitespace token is the trigger; arguments follow on the +same line. This applies in future turns and after context compaction: +if the newest user message begins with `/draft-adr`, `/review-adr`, +`/draft-plan`, or `/review-plan`, enter that shortcut's stateful loop. Common preamble for every shortcut (run once at task start): @@ -290,6 +293,10 @@ Common preamble for every shortcut (run once at task start): - Polling cadence is 60 s. Use Bash with `run_in_background` and an `until [ -f ]; do sleep 60; done` body so the harness notifies you when the awaited file appears. Do not poll inline. + If the current harness exposes a native recurring automation or + heartbeat mechanism instead of background Bash, use that mechanism + with the same cadence and file target. Do not downgrade the shortcut + into a one-shot review/reply because background Bash is unavailable. - Filename suffixes follow the existing convention: `_review-N.md` and `_reply-N.md` next to the parent ADR or plan, where `N` is one greater than the highest existing @@ -298,6 +305,12 @@ Common preamble for every shortcut (run once at task start): between rounds; auto-apply every finding. Only stop when the termination condition for that shortcut is met, or the user sends an explicit message asking you to stop or change direction. +- Before starting any poll, first check whether the file being awaited + already exists. If it does, process it immediately, then continue the + loop from the next expected suffix. +- Never stop after writing only the first review, first reply, or first + draft. After every loop action, immediately arrange the next poll + unless the shortcut's termination condition has been reached. ### `/draft-adr ` @@ -408,3 +421,7 @@ applied to an implementation plan. - Loop and termination identical to `/review-adr`, with the final commit message `Add implementation plan` (or an equivalent imperative ≤72 chars). +- A bare `/review-plan` is still enough to start the full reviewer + loop. If a target plan already exists, write the first static review + and then immediately wait for `_reply-1.md`; do not return a + final answer that implies the task is complete after the first review. From 2c9462a6413713b88c557f251f538bd92cefedec Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sun, 24 May 2026 20:49:11 +0200 Subject: [PATCH 05/65] Add emcee minimizer implementation plan --- docs/dev/plans/emcee-minimizer.md | 464 +++++++++++++++++++++++++----- 1 file changed, 398 insertions(+), 66 deletions(-) diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index 0aa3cc1de..277d74b3c 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -70,20 +70,63 @@ before proceeding. `EmceeMinimizer._native_key_map` (overrides the DREAM-style defaults on `BayesianMinimizerBase._native_key_map`): - | Verbose | emcee native | + | Verbose | emcee native engine attribute | | --- | --- | | `sampling_steps` | `nsteps` | | `burn_in_steps` | `nburn` | | `thinning_interval` | `thin` | | `population_size` | `nwalkers` | - | `parallel_workers` | `pool` | + | `parallel_workers` | `parallel_workers` (engine integer; **not** mapped directly to emcee's `pool`) | | `initialization_method` | (custom — see §6) | | `random_seed` | `random_seed` | + + **`parallel_workers` semantics.** The category persists an + integer; the engine class holds it under the same name. emcee's + `EnsembleSampler` takes a `pool=` argument that is a pool object + (anything with a `.map` method) or `None` for serial execution — + not an integer. The engine builds the actual pool around + `run_mcmc`: + + | `parallel_workers` value | Engine behaviour | + | --- | --- | + | `1` | `pool=None` (serial); single process | + | `0` | `multiprocessing.Pool(os.cpu_count())` | + | `N > 1` | `multiprocessing.Pool(N)` | + + The pool is closed in a `finally:` block after `run_mcmc` returns. + `Analysis._sync_engine_from_minimizer_category` must therefore + skip `parallel_workers` from the native-attribute sync (it + already skips `random_seed` for the same "engine handles it" + reason; add `parallel_workers` to the + `_engine_sync_skip_keys` frozenset introduced in P1.5). 4. Resume uses emcee's `HDFBackend` against the `/emcee_chain` group of the same `analysis/results.h5` file used by the snapshot writer. No separate sidecar file. A non-resume `fit()` follows the prerequisite plan's lifecycle and **truncates** `results.h5` (after the standard warning); resume opens it in append mode. + + **`Project.save()` must not delete `/emcee_chain`.** The current + `write_analysis_results_sidecar` + ([`results_sidecar.py:282`](../../../src/easydiffraction/io/results_sidecar.py)) + opens `results.h5` with mode `'w'` (full truncate). Every save + after a fit will erase the emcee chain unless this writer is + modified. After this plan, the writer opens the file in append + mode (`'a'`) and replaces only the EasyDiffraction-canonical + groups (`/posterior`, `/distribution_cache`, `/pair_cache`, + `/predictive`) by deleting them first if present, leaving any + other top-level groups (currently only `/emcee_chain`) untouched. + + **Non-resume truncate must still happen — and explicitly.** The + current `_warn_results_sidecar_overwrite` + ([`analysis.py:943-953`](../../../src/easydiffraction/analysis/analysis.py)) + / + [`warn_analysis_results_sidecar_overwrite`](../../../src/easydiffraction/io/results_sidecar.py) + only **warns**; it does not delete. With the writer switched to + append mode, a fresh non-resume emcee fit after an older emcee + run would otherwise leave the stale `/emcee_chain` group in + `results.h5`. This plan therefore adds a real preparation step + that warns **and removes** the file before the engine starts. See + P1.5a; the resume path bypasses it. 5. `Analysis.fit()` gains an optional `resume=True, extra_steps=N` call shape for minimizers that support incremental sampling. For other minimizers, passing `resume=True` raises immediately. @@ -194,7 +237,11 @@ When the matching open-issue is fully resolved, move it to - `tests/integration/fitting/test_emcee.py` (cross-check vs DREAM on a shared toy fit; assert posterior medians agree to within tolerance). -- `docs/docs/tutorials/ed-23.py` (emcee + resume tutorial). +- `docs/docs/tutorials/ed-25.py` (emcee + resume tutorial). The + next free tutorial slot — `ed-23` is already the "Co2SiO4 + Sequential Fit" tutorial and `ed-24` is the "LBCO Bayesian + Display" tutorial. Verify `ed-25.py` is unused before creating + it at P1.7 start; bump if a newer slot is already occupied. ### Modified @@ -217,7 +264,10 @@ When the matching open-issue is fully resolved, move it to `/emcee_chain` is present, expose a helper to construct an `emcee.backends.HDFBackend(path, name='emcee_chain', read_only=True)` for inspection/visualisation. -- `pyproject.toml` and `pixi.toml` — add `emcee>=3.1` dependency. +- `pyproject.toml`, `pixi.toml`, and `pixi.lock` — add `emcee>=3.1` + as a direct runtime dependency and refresh the lockfile via + `pixi lock` (CI installs from the lockfile, not from the manifest + files alone). ### Deleted @@ -227,9 +277,19 @@ When the matching open-issue is fully resolved, move it to Mark `[x]` as each step lands. -- [ ] **P1.1 — Add emcee dependency.** Add `emcee>=3.1` to - `pyproject.toml` and `pixi.toml`. Run `pixi install` locally - to verify resolution. Commit: `Add emcee dependency` +- [ ] **P1.1 — Add emcee dependency and refresh the lockfile.** + - Add `emcee>=3.1` to `pyproject.toml` (runtime dependencies, not + just the `doc` extra — the existing lockfile carries emcee only + as `extra == 'doc'` which CI does not install for runtime). + - Add the same dependency to `pixi.toml` (runtime feature). + - Run `pixi lock` to regenerate `pixi.lock` with `emcee` as a + direct runtime dependency. The refreshed `pixi.lock` is the + artifact CI consumes; `pixi install` is a local sanity check + only. + - Stage `pyproject.toml`, `pixi.toml`, and `pixi.lock` together. + + Files modified by this step: `pyproject.toml`, `pixi.toml`, + `pixi.lock`. Commit: `Add emcee runtime dependency` - [ ] **P1.2 — Register `MinimizerTypeEnum.EMCEE`.** Add the enum member with value `'emcee'` to @@ -282,43 +342,163 @@ Mark `[x]` as each step lands. that `Analysis._sync_engine_from_minimizer_category` writes to from the category's `_native_kwargs()`. The descriptor names on the engine must match the keys returned by - `EmceeMinimizer._native_kwargs()` (i.e. emcee's native names — - `nsteps`, `nburn`, `nwalkers`, `pool`). + `EmceeMinimizer._native_kwargs()`, which are: `nsteps`, + `nburn`, `thin`, `nwalkers`, `parallel_workers`, + `random_seed`, plus `initialization_method` and + `proposal_moves` handled by custom hooks (see §3 for the + mapping table and the `parallel_workers` semantics — the + engine attribute is an integer; the actual emcee `pool` + object is built and torn down inside `EmceeMinimizer.fit`, + not by the native-key sync). + + **Engine-facing contract.** `EmceeMinimizer.fit` matches the + existing + [`MinimizerBase.fit`](../../../src/easydiffraction/analysis/minimizers/base.py) + contract — `Fitter.fit` (the layer that owns `structures`, + `experiments`, `weights`, and `analysis`) calls every engine + uniformly. The engine receives only `parameters` and the + already-built `objective_function`: + + ```python + MinimizerBase.fit( + parameters: list[Parameter], + objective_function: Callable[[dict[str, object]], np.ndarray], + verbosity: VerbosityEnum = VerbosityEnum.FULL, + *, + finalize_tracking: bool = True, + use_physical_limits: bool = False, + random_seed: int | None = None, + resume: bool = False, # added by P1.5 + extra_steps: int | None = None, # added by P1.5 + ) -> FitResults + ``` + + The base implementation raises `NotImplementedError` only when + `resume=True` (see P1.5). `EmceeMinimizer.fit` overrides with + the same signature and honours `resume` / `extra_steps`. + + **Sidecar path.** emcee's `HDFBackend` needs a file path, but + the engine signature deliberately does not carry one. The + engine reads it from a private attribute + `self._sidecar_path: Path | None` that `Fitter.fit` sets on + the engine before calling `engine.fit(...)`, derived from + `analysis.project.info.path / 'analysis' / 'results.h5'`. + Engines that do not need it ignore the attribute; the + attribute defaults to `None` and `EmceeMinimizer.fit` raises + `RuntimeError` if it is `None` when needed. + + **Residual-to-log-probability adapter.** emcee expects a + scalar log probability from a flat walker coordinate + (`np.ndarray` shape `(ndim,)`), but the `objective_function` + `Fitter.fit` passes us returns a residual array from an + `engine_params` dict. The engine class builds an adapter + `log_prob(theta)` that: + + 1. Maps `theta` (the walker vector) onto an `engine_params` + dict using a fixed ordered list of free-parameter unique + names captured **once** at sampler construction (from the + `parameters` argument to `fit`). + 2. Rejects values outside + `[parameter.fit_min, parameter.fit_max]` by returning + `-np.inf` immediately — does not call the calculator for + invalid proposals. + 3. Calls the **passed-in** `objective_function(engine_params)` + (do **not** call `Fitter._build_objective_function` from + the engine — that is a `Fitter` helper, not engine API). + 4. Returns Gaussian log likelihood + `-0.5 * np.sum(r**2)`. The current fit weights are already + folded into the residuals by the objective function; this + step does **not** re-weight. + 5. Returns `-np.inf` on calculator exceptions (rare; emcee + re-proposes). + 6. Treats the prior as flat over the box-bounded parameter + volume — no informative priors in v1; deferred to a + follow-on plan. + + The adapter must **not** mutate live `Parameter.value` state + on `-np.inf` returns. Implement by passing an + `engine_params` dict to `objective_function` (the existing + objective writes the values into live parameters internally; + that mutation only happens once `objective_function` is + invoked, so the bounds check above must guard every call). Engine method shape (sketch): ```python - class EmceeMinimizer(BayesianMinimizerEngineBase): + class EmceeMinimizer(MinimizerBase): name = MinimizerTypeEnum.EMCEE method = 'stretch' - def fit(self, *, structures, experiments, analysis, - resume=False, extra_steps=None, ...): + # Set by Fitter.fit before this fit() call: + _sidecar_path: Path | None = None + + def fit( + self, + parameters: list[Parameter], + objective_function: Callable[..., object], + verbosity: VerbosityEnum = VerbosityEnum.FULL, + *, + finalize_tracking: bool = True, + use_physical_limits: bool = False, + random_seed: int | None = None, + resume: bool = False, + extra_steps: int | None = None, + ) -> FitResults: + if self._sidecar_path is None: + msg = ('emcee engine requires Fitter.fit to set ' + '_sidecar_path; was Analysis configured?') + raise RuntimeError(msg) + + free_param_names = [p.unique_name for p in parameters] + param_by_name = {p.unique_name: p for p in parameters} + + def log_prob(theta): + for name, value in zip(free_param_names, theta): + p = param_by_name[name] + if not (p.fit_min <= value <= p.fit_max): + return -np.inf + engine_params = dict(zip(free_param_names, theta)) + try: + r = objective_function(engine_params) + except Exception: + return -np.inf + return -0.5 * float(np.sum(np.asarray(r) ** 2)) + backend = emcee.backends.HDFBackend( - path=analysis.project.info.path - / 'analysis' / 'results.h5', - name='emcee_chain', + self._sidecar_path, name='emcee_chain', read_only=False, ) - sampler = emcee.EnsembleSampler( - nwalkers=self.nwalkers, - ndim=len(free_params), - log_prob_fn=log_prob, - pool=self.pool, - moves=..., # from proposal_moves - backend=backend, - ) - if resume: - self._validate_resume(backend, free_params) - sampler.run_mcmc(None, nsteps=extra_steps, - skip_initial_state_check=True, - progress=True) - else: - initial_state = self._initial_state(...) - sampler.run_mcmc(initial_state, - nsteps=self.nsteps, - progress=True) - return self._build_results(sampler, ...) + pool = self._build_pool(self.parallel_workers) + try: + sampler = emcee.EnsembleSampler( + nwalkers=self.nwalkers, + ndim=len(free_param_names), + log_prob_fn=log_prob, + pool=pool, + moves=self._resolve_moves(self.proposal_moves), + backend=backend, + ) + if resume: + self._validate_resume(backend, free_param_names) + sampler.run_mcmc( + None, nsteps=extra_steps, + skip_initial_state_check=True, + progress=True, + ) + else: + initial_state = self._initial_state( + parameters, self.nwalkers, + self.init, random_seed, + ) + sampler.run_mcmc( + initial_state, nsteps=self.nsteps, + progress=True, + ) + finally: + if pool is not None: + pool.close() + pool.join() + return self._build_results(sampler, parameters) ``` Register with the engine `MinimizerFactory` and update @@ -327,34 +507,154 @@ Mark `[x]` as each step lands. Commit: `Add EmceeMinimizer engine class` -- [ ] **P1.5 — Wire `fit(resume=True, extra_steps=N)` on - `Analysis`.** In - `src/easydiffraction/analysis/analysis.py`: - - `Analysis.fit()` gains keyword args `resume: bool = False, - extra_steps: int | None = None`. Default behaviour (no resume) - is unchanged. - - When `resume=True`: - - Validate that `self.minimizer.type == - MinimizerTypeEnum.EMCEE.value` (only emcee supports resume in - v1). Raise `ValueError` with a clear message otherwise. - - Require `extra_steps` to be a positive integer. - - **Bypass** the `_warn_results_sidecar_overwrite` step and the - `_clear_persisted_fit_state` reset — both would clobber the - backend. - - Forward `resume=True, extra_steps=N` to the live engine via - the existing `Fitter` plumbing. - - Also address open issue #103: introduce - `_engine_sync_skip_keys: ClassVar[frozenset[str]] = frozenset({'random_seed'})` +- [ ] **P1.5 — Wire `fit(resume=True, extra_steps=N)` end-to-end.** + The current fit stack does not accept `resume` / `extra_steps` + anywhere. Every signature and call site listed below must be + updated in this step. Each item is one short edit; the step + lands as a single commit because the signatures must change in + lockstep. + + **`Fitter` stays the layer that owns `structures`, `experiments`, + `weights`, parameter collection, and objective construction.** + Engines receive only `parameters` and `objective_function` + (already-built) via the existing `MinimizerBase.fit` shape. This + plan adds `resume` and `extra_steps` to the same shape. + + **Signatures (add `resume: bool = False, extra_steps: int | None = None`):** + + - `Analysis.fit` + ([analysis.py:929](../../../src/easydiffraction/analysis/analysis.py)). + User-facing entry point. + - `Analysis._run_single`, `Analysis._run_joint`, + `Analysis._prepare_fit_run`, `Analysis._fit_single`, + `Analysis._fit_joint` (every internal helper that takes the + fit through to `Fitter.fit`). + - `Fitter.fit` + ([fitting.py:140-150](../../../src/easydiffraction/analysis/fitting.py)) + — adds the keyword pair to its existing signature + (`structures`, `experiments`, `weights`, `analysis`, + `verbosity`, `use_physical_limits`, `random_seed`, **+ + `resume`, `extra_steps`**). Forwards the pair to + `self.minimizer.fit(...)`. + - `MinimizerBase.fit` + ([base.py:351-360](../../../src/easydiffraction/analysis/minimizers/base.py)) + — adds the keyword pair to its existing engine-facing shape + (`parameters`, `objective_function`, `verbosity`, *keyword*: + `finalize_tracking`, `use_physical_limits`, `random_seed`, + **+ `resume`, `extra_steps`**). The base implementation + handles non-resume calls unchanged and raises + `NotImplementedError(f"Minimizer '{self.name}' does not + support resume.")` when `resume=True`. `EmceeMinimizer.fit` + overrides with the same signature and honours both args. + + **Sidecar-path plumbing.** `Fitter.fit` resolves the sidecar + path from `analysis.project.info.path` (when both are non-None) + and sets `self.minimizer._sidecar_path` on the engine before + calling `self.minimizer.fit(...)`. Engines that do not need it + ignore the attribute; `EmceeMinimizer.fit` reads it (and raises + `RuntimeError` if `None` as defence in depth — but normal users + never hit that path because of the upfront save-required guard + in the next bullet). + + **Behaviour rules:** + + - **Single mode only for v1.** `Analysis._run_joint` raises + `ValueError('Resume is supported in single fit mode only')` + when `resume=True`. Joint-mode resume is deferred; recorded + explicitly in §"Open questions". + - **Validate the active minimizer.** `Analysis.fit` raises + `ValueError` when `resume=True` and + `self.minimizer.type != MinimizerTypeEnum.EMCEE.value`. Match + the clear-error pattern used elsewhere. + - **Require a saved project for emcee.** Unlike DREAM (which + keeps the chain in memory), emcee's `HDFBackend` is the + sampler's live chain store — it needs a real file path. + `Analysis.fit` raises `ValueError` when + `self.minimizer.type == MinimizerTypeEnum.EMCEE.value` and + `self.project.info.path is None`, with a clear scientist-facing + message that points at the fix: + `"emcee requires a saved project; call project.save_as() + before analysis.fit()."` The check fires for both the initial + run and `resume=True`. The engine's `_sidecar_path is None` + `RuntimeError` (per P1.4) remains as defence-in-depth for + direct engine calls outside the `Analysis` flow but is not + reachable from the user-facing path. + - **Validate `extra_steps`.** Require positive integer when + `resume=True`. Raise on `None`, `0`, or negative. + - **Bypass reset.** Skip the new + `prepare_analysis_results_sidecar_for_new_fit` helper (see + P1.5a) and `_clear_persisted_fit_state` when `resume=True` — + both would clobber the chain and the persisted fit state. + - **Defence in depth.** `MinimizerBase.fit`'s + `NotImplementedError` on `resume=True` only matters if an + engine is called directly outside the `Analysis` flow — the + `Analysis.fit` guard above fires first in normal use. + + **Open issue #103 cleanup.** Introduce + `_engine_sync_skip_keys: ClassVar[frozenset[str]] = frozenset({'random_seed', 'parallel_workers'})` on `MinimizerCategoryBase`, and update `_sync_engine_from_minimizer_category` ([analysis.py:1134-1146](../../../src/easydiffraction/analysis/analysis.py)) - to use it. `BayesianMinimizerBase` (or `EmceeMinimizer` - directly) overrides the frozenset to include `'proposal_moves'` - if/when the engine consumes that key under a different name than - the category attribute. - - Commit: `Wire emcee resume into Analysis.fit` + to use it. `EmceeMinimizer` (category) overrides the frozenset to + add `'proposal_moves'` if the engine consumes that key + differently from the category attribute (sketch in P1.4). + + Commit: `Wire emcee resume through fit stack` + +- [ ] **P1.5a — Make `results.h5` append-on-save and add an + explicit truncate-on-new-fit prep step.** Two coordinated + changes that land in a single commit because they jointly + preserve the ADR lifecycle (resume keeps `/emcee_chain`; + new fit removes it). + + In + [`src/easydiffraction/io/results_sidecar.py`](../../../src/easydiffraction/io/results_sidecar.py): + + - Replace `h5py.File(sidecar_path, 'w')` (line 282) with + `h5py.File(sidecar_path, 'a')`. Before writing each + EasyDiffraction-canonical group (`/posterior`, + `/distribution_cache`, `/pair_cache`, `/predictive`), delete + that group first if present so the writer's behaviour for those + groups is unchanged. + - Do **not** touch any other top-level group from the writer. + `/emcee_chain` survives every save. + - **Add a new helper** `prepare_analysis_results_sidecar_for_new_fit(*, analysis_dir: Path) -> None` + that takes the same `analysis_dir` shape as + `warn_analysis_results_sidecar_overwrite`, **warns** when the + file exists (matching the current warning text), and then + **removes** the file entirely so a fresh fit starts from a + clean slate. The old `warn_analysis_results_sidecar_overwrite` + becomes a thin wrapper that delegates to the new helper, or is + replaced outright by the new helper at every call site. + + In + [`src/easydiffraction/analysis/analysis.py`](../../../src/easydiffraction/analysis/analysis.py): + + - Replace `_warn_results_sidecar_overwrite` (lines 943-953) with + a call to the new `prepare_analysis_results_sidecar_for_new_fit` + helper. Same call sites in `_run_single` and `_run_joint`. + - **Bypass on resume.** The `resume=True` branch in + `Analysis._prepare_fit_run` (added by P1.5) must **not** call + this helper — that is the whole point of resume keeping the + chain alive. The bypass rule listed in P1.5 "Behaviour rules + → Bypass reset" therefore now also covers the + `prepare_analysis_results_sidecar_for_new_fit` call. + + Add focused unit tests in Phase 2 (P2.1) covering: + + - **Append preserves `/emcee_chain`.** Write a sidecar payload, + create a stub `/emcee_chain` group on the same file, re-write + the sidecar — the `/emcee_chain` group must survive. + - **New-fit prep removes the file.** Create a sidecar with a + stub `/emcee_chain` group; call + `prepare_analysis_results_sidecar_for_new_fit`; assert the + file is gone (or empty) and the stale group is unreachable. + - **Resume bypass.** Set up an `Analysis` with a saved project, + invoke the resume code path (mocked engine), and assert + `prepare_analysis_results_sidecar_for_new_fit` was **not** + called. + + Commit: `Append-on-save plus explicit truncate-on-new-fit prep` - [ ] **P1.6 — Route emcee outputs into the existing fit_result and sidecar pipeline.** Verify the existing @@ -378,8 +678,17 @@ Mark `[x]` as each step lands. Commit: `Route emcee posterior through fit_result and sidecar` -- [ ] **P1.7 — Add `ed-23.py` tutorial.** New notebook source at - `docs/docs/tutorials/ed-23.py` covering: +- [ ] **P1.7 — Add `ed-25.py` tutorial.** Verify first that + `docs/docs/tutorials/ed-25.py` is unused. `ed-23.py` is the + "Co2SiO4 Sequential Fit" tutorial and `ed-24.py` is the + "LBCO Bayesian Display" tutorial — do **not** overwrite + either. If `ed-25.py` already exists by the time this step + runs, pick the next free integer slot and adjust the file + name + references below to match. + + New notebook source at `docs/docs/tutorials/ed-25.py` + covering: + - `project.analysis.minimizer.type = 'emcee'` (post-switchable syntax). - `project.analysis.minimizer.sampling_steps = 1000` (small for @@ -390,16 +699,29 @@ Mark `[x]` as each step lands. the chain. - Final posterior plot after resume. + Update the docs navigation in the same step: + + - Add an entry under "MCMC / Bayesian" (or the appropriate + section) in + [`docs/docs/tutorials/index.md`](../../docs/tutorials/index.md) + pointing at `ed-25.ipynb`. + - Add a navigation entry under the matching section in + [`docs/mkdocs.yml`](../../../docs/mkdocs.yml). + Run `pixi run notebook-prepare` to regenerate the `.ipynb`. - Verification grep (must return empty against - `docs/docs/tutorials/ed-23.py`): + Verification greps: ``` - git grep -nE '\banalysis\.minimizer_type\b|\bminimizer\.runtime_seconds\b|\bminimizer\.gelman_rubin_max\b' docs/docs/tutorials/ed-23.py + test -f docs/docs/tutorials/ed-25.py + git grep -nE '\banalysis\.minimizer_type\b|\bminimizer\.runtime_seconds\b|\bminimizer\.gelman_rubin_max\b' docs/docs/tutorials/ed-25.py + git grep -n 'ed-25' docs/docs/tutorials/index.md docs/mkdocs.yml ``` - Commit: `Add ed-23 emcee tutorial` + The first must be true; the second must be empty; the third must + return at least one hit in each file. + + Commit: `Add ed-25 emcee tutorial` - [ ] **P1.8 — Phase 1 review gate.** No code change. Stop and request user review. After approval, proceed to Phase 2. @@ -417,6 +739,16 @@ required by `.github/copilot-instructions.md` → **Workflow**. - `tests/unit/easydiffraction/analysis/minimizers/test_emcee.py`: engine-class registration, descriptor defaults, native kwargs plumbing. + - `tests/unit/easydiffraction/analysis/test_analysis.py` (or + matching coverage file): assert `Analysis.fit()` raises + `ValueError` with a save-prompt message when emcee is the + active minimizer and `project.info.path is None`, for both + initial fits and `resume=True`. + - `tests/unit/easydiffraction/io/test_results_sidecar.py`: the + save-after-resume invariant from P1.5a — write a sidecar + payload, then write a stub `/emcee_chain` group on the same + file, then re-write the sidecar; the `/emcee_chain` group + must survive. - `tests/integration/fitting/test_emcee.py`: end-to-end fit on a small synthetic problem; resume; assert posterior medians agree with a DREAM run within tolerance. @@ -503,5 +835,5 @@ single-file affair. Plots, parameter posteriors, and tables work the same as for DREAM, so switching between samplers to cross-check results is straightforward. -A new tutorial (`ed-23`) walks through a short run, saving the +A new tutorial (`ed-25`) walks through a short run, saving the project, and resuming for additional steps. From 1f9c44946f99fe94ad0f20a23ad07b946ff62076 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sun, 24 May 2026 20:59:43 +0200 Subject: [PATCH 06/65] Add /implement-plan and switch shortcuts to sentinel-based handoff --- .github/copilot-instructions.md | 189 ++++++++++++++++++++++++++------ 1 file changed, 158 insertions(+), 31 deletions(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 575444140..decf5a0dc 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -311,6 +311,25 @@ Common preamble for every shortcut (run once at task start): - Never stop after writing only the first review, first reply, or first draft. After every loop action, immediately arrange the next poll unless the shortcut's termination condition has been reached. +- **Final-review sentinel.** When a reviewer shortcut decides no + findings remain, it writes a final review file whose body begins + with the literal sentinel line + `**No findings. Ready to commit.**` on its own line (immediately + after the title and any boilerplate header). The matching author + shortcut, before writing a reply, runs + `grep -q "^\*\*No findings\. Ready to commit\.\*\*$" + docs/dev/{adrs/suggestions,plans}/_review-.md` on the + newest review. If the sentinel is found, the author shortcut + **stops the polling loop** without writing a reply and reports + that the review cycle is closed. The sentinel is the only + termination signal between author and reviewer — no other phrase + triggers it. +- **Existing shortcuts never delete files and never commit.** The + four shortcuts in this section + (`/draft-adr`, `/review-adr`, `/draft-plan`, `/review-plan`) + only read existing files and write new ones. They never call + `git rm`, `git add`, or `git commit`. Cleanup and commit happen + in a separate dedicated shortcut (`/implement-plan`). ### `/draft-adr ` @@ -328,17 +347,23 @@ incoming reviews in a polling loop. **Loop (per tick):** -- When `_review-N.md` appears: read it, update the ADR to - address every finding, and write `_reply-N.md` with one - section per finding (verdict + action taken + pointer to the - affected ADR section). +- When `_review-N.md` appears, first check for the + final-review sentinel (see Common preamble). If + `**No findings. Ready to commit.**` is the first body line, + stop the polling loop: report that the review cycle is closed + and that `/implement-plan` (or a manual commit) is the next + step. Do not write a reply. +- Otherwise read the review, update the ADR to address every + finding, and write `_reply-N.md` with one section per + finding (verdict + action taken + pointer to the affected ADR + section). - Start polling for `_review-(N+1).md`. -- **Do not commit.** Leave every edit in the worktree as modified - or untracked. The reviewer side (`/review-adr`) is responsible - for the final commit. +- **Do not commit and do not delete any file.** Leave every edit + in the worktree as modified or untracked. Commit and cleanup + happen later in `/implement-plan`. -**Termination:** only on an explicit user message asking you to -stop or change direction. The loop never self-terminates. +**Termination:** either the sentinel-driven stop above or an +explicit user message asking you to stop or change direction. ### `/review-adr []` @@ -367,20 +392,18 @@ until all findings are addressed. and every prior review/reply. Pick exactly one branch: - **Findings remain:** write `_review-(N+1).md` listing only the open findings, then poll for the next reply. - - **All findings addressed:** write a final - `_review-(N+1).md` stating "no findings; ADR is ready", - then run the termination cleanup below and stop. - -**Termination cleanup (only when the final clean review is -written):** - -1. `git rm` every `_review-*.md` and `_reply-*.md` - next to the ADR (including the final clean review just - written). -2. `git add` the cleaned-up ADR. -3. Commit with message `Add ADR suggestion` (or an - equivalent imperative ≤72 chars). -4. Report the commit hash and stop. + - **All findings addressed:** write the final review file + `_review-(N+1).md` whose body begins with the + final-review sentinel from the Common preamble: + `**No findings. Ready to commit.**` on its own line, followed + by a short paragraph summarising the round (no findings list). + Then **stop the loop**. **Do not delete any review or reply + file. Do not commit anything.** Cleanup and commit happen in + `/implement-plan`. + +**Termination:** either the sentinel-written final review above +or an explicit user message asking you to stop or change +direction. ### `/draft-plan []` @@ -400,12 +423,15 @@ applied to an implementation plan instead of an ADR. status checklist, Phase 2 verification commands using the zsh-safe log-capture pattern, and a Suggested Pull Request section. -- Loop behaviour identical to `/draft-adr`: poll - `docs/dev/plans/_review-N.md`; write - `_reply-N.md`; **no commits in the loop**. +- Loop behaviour identical to `/draft-adr`, including the + sentinel-driven stop: before writing a reply, check the newest + `docs/dev/plans/_review-N.md` for the final-review + sentinel and stop if present. Otherwise read the review, update + the plan, write `_reply-N.md`, poll for the next review. + **No commits and no file deletions in this loop.** -**Termination:** only on an explicit user message asking you to -stop or change direction. +**Termination:** either the sentinel-driven stop or an explicit +user message asking you to stop or change direction. ### `/review-plan []` @@ -418,10 +444,111 @@ applied to an implementation plan. cycle. - Static reads only — same "no tests / lint / build / formatters / pixi" rule applies during plan reviews. -- Loop and termination identical to `/review-adr`, with the final - commit message `Add implementation plan` (or an - equivalent imperative ≤72 chars). +- Loop and termination identical to `/review-adr`, including the + final-review-sentinel rule: when all findings are addressed, + write the final review with `**No findings. Ready to commit.**` + as the first body line and stop. **Do not delete any review or + reply file and do not commit anything** — those steps belong to + `/implement-plan`. - A bare `/review-plan` is still enough to start the full reviewer loop. If a target plan already exists, write the first static review and then immediately wait for `_reply-1.md`; do not return a final answer that implies the task is complete after the first review. + +### `/implement-plan []` + +Clean up the review/reply deliberation history, commit the latest +ADR + plan, then execute the plan's Phase 1 implementation steps +autonomously. This is the only shortcut in this section that +deletes files and commits changes. + +**Setup (once):** + +1. Ask the user, **in one batch**, to grant every permission this + shortcut will need so the implementation phase runs without + per-step prompts. List explicitly: + - `Bash` (general, including `run_in_background`) for `git`, + `ls`, `grep`, `find`, `until`, `sleep`, and any plan-step + command (`pixi run notebook-prepare`, etc.). + - `Read` / `Edit` / `Write` on `src/`, `tests/`, + `docs/dev/adrs/`, `docs/dev/plans/`, `docs/docs/tutorials/`, + `pyproject.toml`, `pixi.toml`, `pixi.lock`, and any other + paths the plan's `Concrete files likely to change` section + enumerates. + - `git rm`, `git add`, `git commit` (subsumed under Bash). + - Cite this section so the user knows the scope. +2. **Do not** run tests, lint, `pixi run fix`, `pixi run check`, + or the integration/script-test suites during this shortcut — + those belong to the plan's Phase 2 verification gate and run + only after the Phase 1 review gate the plan defines. +3. Stay on the current branch. +4. Identify targets: + - **Plan**: if `` is given, target + `docs/dev/plans/.md`. Otherwise pick the most recently + modified plan in `docs/dev/plans/` whose stem matches the + current branch slug. + - **ADR**: derive from the plan's `## ADR` reference. The ADR + usually lives in `docs/dev/adrs/suggestions/.md` at + this point (a Phase 1 plan step typically promotes it to + `accepted/`); accept either location. + +**Phase A — cleanup + commits (two atomic commits):** + +1. **ADR commit.** + - `git rm` every `docs/dev/adrs//_review-*.md` + and `_reply-*.md` next to the ADR. + - `git add docs/dev/adrs//.md`. + - Commit with message `Add ADR suggestion` (or + `Promote ADR to accepted` if the ADR was already + under `accepted/`). +2. **Plan commit.** + - `git rm` every `docs/dev/plans/_review-*.md` and + `_reply-*.md`. + - `git add docs/dev/plans/.md`. + - Commit with message `Add implementation plan`. + +If the ADR or plan has no review/reply siblings (e.g. the user +drafted it directly without running the loop), skip the `git rm` +step but still stage + commit the clean artifact. + +**Phase B — implementation:** + +3. Parse the plan's `## Implementation steps (Phase 1)` section + in order. Each `- [ ] **P1.X — **` item is one + atomic commit. For each step in turn: + 1. Read the step body to identify the files and edits. + 2. Apply the edits exactly as the step prescribes. If the + step says to run a build-step command (e.g. + `pixi run notebook-prepare`, `pixi lock`), run it — those + are part of the plan, not Phase 2 verification. + 3. Stage the files the step enumerates with explicit paths + (per **Commits** rule "Stage only the files modified for + the step"). Do not stage unrelated dirty files. + 4. Commit with the suggested commit message from the step + (the `Commit:` line). Edit the `- [ ]` checkbox to `- [x]` + in the plan file before staging, so the checklist tracks + progress, and include the checklist update in the same + commit. +4. **Phase 1 review gate.** The plan's final P1 step is always + "Phase 1 review gate. No code change. Stop and request user + review." When you reach that step, mark it `[x]`, commit the + checklist update alone, and **stop**. Report: + - How many P1 commits landed. + - The current branch and tip commit hash. + - That Phase 2 verification (`pixi run fix`, `pixi run check`, + `pixi run unit-tests`, `pixi run integration-tests`, + `pixi run script-tests`) is still pending and is the user's + next action. + +**If a step fails:** stop immediately, do not skip ahead, report +the failure with the step ID and the error. Do not run Phase 2 +verification commands as a debugging tool — fix the step or wait +for user input. + +**Permissions / approval flow.** The setup-step permission batch +is the only prompt this shortcut emits during normal operation. +If the plan introduces a step that requires a permission the user +did not pre-approve (e.g. an `npm` invocation when the upfront +batch only covered `pixi`), pause at that step and ask for the +extra grant rather than blocking on a single tool call. Resume +the loop once granted. From 5d213cc4601d47e8dff3f96c6bd76b61d8c74b94 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sun, 24 May 2026 21:07:48 +0200 Subject: [PATCH 07/65] Clean up and formatting --- .github/copilot-instructions.md | 554 --------------- .gitignore | 4 + AGENTS.md | 5 - CLAUDE.md | 1 - .../minimizer-category-consolidation.md | 7 +- .../switchable-category-owned-selectors.md | 14 +- docs/dev/plans/emcee-minimizer.md | 658 +++++++++--------- .../dev/plans/minimizer-input-output-split.md | 9 +- 8 files changed, 331 insertions(+), 921 deletions(-) delete mode 100644 .github/copilot-instructions.md delete mode 100644 AGENTS.md delete mode 100644 CLAUDE.md diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md deleted file mode 100644 index decf5a0dc..000000000 --- a/.github/copilot-instructions.md +++ /dev/null @@ -1,554 +0,0 @@ -# Copilot Instructions for EasyDiffraction - -## Project Context - -- Python library for crystallographic diffraction analysis (refining - structural models against experimental data). -- Domain axes: `sample_form` (powder, single crystal), `beam_mode` - (time-of-flight, constant wavelength), `radiation_probe` (neutron, - x-ray), `scattering_type` (bragg, total). -- Calculation backends: `cryspy` and `crysfml` (Bragg), `pdffit2` (total - scattering). -- CIF maps to `DatablockItem`/`DatablockCollection` and - `CategoryItem`/`CategoryCollection` (loops). Follow CIF naming; - deviate only for a clearly better API. -- Metadata via frozen dataclasses: `TypeInfo`, `Compatibility`, - `CalculatorSupport`. -- Audience is scientists, often non-programmers: prioritize - discoverability, clear errors, and safe defaults over developer - ergonomics. -- Critical-software rigor: every code path tested, edge cases handled - explicitly, no silent failures. - -## Code Style - -- snake_case (functions/vars), PascalCase (classes), UPPER_SNAKE_CASE - (constants). -- `from __future__ import annotations` in every module. Type-annotate - all public signatures. -- Numpy-style docstrings on all public classes/methods (Parameters / - Returns / Raises where applicable). Summary is one line ≤72 chars - (`max-doc-length`); shorten wording rather than wrap. -- Flat over nested, explicit over clever, composition over deep - inheritance. No defensive checks for unlikely edge cases. -- One class per file when substantial; group small related classes. -- No `**kwargs` — use explicit keyword arguments. -- No string-based dispatch (e.g. `getattr(self, f'_{name}')`); write - named methods (`_set_sample_form`, `_set_beam_mode`). Narrow framework - metadata lookups are allowed when the attribute name is a class-level - declaration, is not user input, and is validated in one central place; - for example, `CategoryItem._category_entry_name`. -- Public attrs are either editable (getter+setter property) or read-only - (getter only). For internal mutation of read-only props, use a private - `_set_` method, not a public setter. -- Lint complexity thresholds in `pyproject.toml` (`max-args`, - `max-branches`, `max-statements`, `max-locals`, `max-nested-blocks`, - …) are guardrails. A violation means refactor (extract helpers, - parameter objects, flatten) — do not raise thresholds, add `# noqa`, - or otherwise silence them. For complex refactors touching many lines - or public API, propose a plan and wait for approval. - -## Architecture - -- Eager top-of-module imports by default. Lazy imports only to break - circular deps or to keep `core/` free of heavy imports on rarely- - called paths (e.g. `help()`). -- No `pkgutil`/`importlib` auto-discovery, no background threads, no - monkey-patching or runtime class mutation. -- No `__all__`; control public API via explicit `__init__.py` imports. - No redundant `import X as X` aliases. -- Concrete classes use `@Factory.register`. Each package's `__init__.py` - must explicitly import every concrete class to trigger registration — - always update it when adding a class. -- Switchable categories (factory-swappable at runtime) follow the - category-owned selector contract from - [`switchable-category-owned-selectors.md`](../docs/dev/adrs/accepted/switchable-category-owned-selectors.md): - the owner exposes `` (read-only attribute on the owner), - and the category itself exposes `.type` (writable - property) and `.show_supported()`. There are no - owner-level `._type` setters and no owner-level - `show_supported__types()` / `show_current__type()` - methods. The owner provides a private `_swap_` hook that - the category's `type` setter calls through a back-reference; - inside the hook the owner replaces the category instance - (Family A), rebinds the live engine (Family B), or activates - sibling categories (Family C) — the user-facing surface stays - uniform. Required even if only one implementation exists. -- Result-output categories paired with a switchable input category - (today: `analysis.fit_result` paired with `analysis.minimizer` - via `._fit_result_class`) are **internal pairs**, - not user-facing switchables: they do not expose `type` or - `show_supported()`; the owner swaps them in lockstep with the - paired input category. See - [`minimizer-input-output-split.md`](../docs/dev/adrs/accepted/minimizer-input-output-split.md). -- Categories are flat siblings within their owner. Never nest a category - as a child of another category of a different type; cross-reference - via IDs instead. -- Every finite, closed set of values (factory tags, axes, enumerated - descriptors) is a `(str, Enum)`; compare against members, not raw - strings. -- Keep `core/` free of domain logic (base classes and utilities only). -- Don't introduce abstractions before a concrete second use case. Don't - add dependencies without asking. - -## Testing - -- Every new module, class, or bug fix ships with tests. See - `docs/dev/adrs/accepted/test-strategy.md` for the full strategy. -- Unit tests mirror the source tree: - `src/easydiffraction//.py` → - `tests/unit/easydiffraction//test_.py`. Verify with - `pixi run test-structure-check`. Supplementary tests: - `test__coverage.py`. Category packages with only - `default.py`/`factory.py` may use one parent-level - `test_.py`. -- Tests expecting `log.error()` to raise must `monkeypatch` Logger to - RAISE mode (another test may have leaked WARN mode). -- `@typechecked` setters raise `typeguard.TypeCheckError`, not - `TypeError`. -- No test-ordering dependence, no network, no sleeping, no real - calculation engines in unit tests. - -## Tutorials - -- Notebooks in `docs/docs/tutorials/*.ipynb` are generated artifacts. - Edit only the corresponding `*.py`, then run - `pixi run notebook-prepare`. - -## Change Discipline - -- Before any structural/design change (new categories, factories, - switchable-category wiring, datablocks, CIF serialisation), read - `docs/dev/adrs/index.md` and the relevant accepted ADRs. Localised bug - fixes or test updates need only this file. -- Development documentation lives under `docs/dev/`. Use - `docs/dev/adrs/index.md` as the architecture and decision navigation - surface; there is no separate `architecture.md` source of truth. -- Project is in beta: no legacy shims, no deprecation warnings — update - tests and tutorials to the current API. -- Minimal diffs; don't reformat working code. Fix only what's asked; - flag adjacent issues as comments. Don't add features or refactor - unless asked. Don't remove TODOs or comments unless the change fully - resolves them. -- Never remove or replace existing functionality without explicit - confirmation — highlight every removal and wait for approval. -- When renaming or auditing usages, search the entire project (code, - tests, tutorials, docs). Use `git grep -n` because all contributors - have Git; do not assume `rg` is installed. If `git grep` is - unavailable, fall back to `find ... -type f` plus `grep -n`. -- When asked to review a plan, save the review next to that plan using - `_review-N.md`, where `N` is one greater than the highest - existing review number for that plan. For example, - `docs/dev/plans/background-refactor.md` is reviewed in - `docs/dev/plans/background-refactor_review-1.md`, then - `docs/dev/plans/background-refactor_review-2.md`. A reviewer must not - run tests, `pixi run fix`, `pixi run check`, or any other build or - verification command; reviews are static reads of code, plan, and - documentation only. Note in the review which checks were skipped so - the next implementer knows the gap. -- Writing a review or a reply to a review does **not** require running - any formatter (`prettier`, `pixi run fix`, `ruff format`, …) or any - lint/check/test command on the review/reply file itself or any - surrounding documentation. Review and reply files are markdown-only, - written by hand, and committed as-is. Formatting passes happen later, - during implementation Phase 2 verification — not in the review cycle. - This rule applies to both `_review-N.md` and `_reply-N.md` files - regardless of where they live (`docs/dev/plans/`, `docs/dev/adrs/…/`, - etc.). -- Each change is atomic and single-commit-sized: make one change, - suggest the commit message, then stop and wait for confirmation. - Exception: when the user invokes an **Agent Shortcut** (see that - section), the matching loop runs autonomously per its own - termination rule — neither a per-commit pause nor a per-tick - pause applies inside that loop. The default applies again as soon - as the shortcut terminates. -- When in doubt, ask. - -## Commits - -- Suggest a commit message after each change: code block, ≤72 chars, - imperative mood, no type prefix, no `Co-authored-by: Copilot`. - Examples: - - Add ChebyshevPolynomialBackground class - - Implement background_type setter on Experiment - - Standardize switchable-category naming convention -- Stage only the files modified for the step, using explicit paths where - practical. Do not include data, project, CIF, or other generated - artifacts produced by integration/script/notebook tests unless the - user explicitly asked to update them. -- Before each commit, inspect the worktree and avoid staging unrelated - user changes. If unrelated dirty files exist, leave them untouched and - mention them only when relevant. - -## Workflow - -Non-trivial changes use a two-phase workflow: - -- **Phase 1 — Implementation.** Code and docs updates only. Update ADRs - when the change affects architecture or documented decisions. Do not - create or run tests unless the user explicitly asks. When done, - present for review and iterate until approved. -- **Phase 2 — Verification.** Add/update tests, then run `pixi run fix`, - `pixi run check`, `pixi run unit-tests`, `pixi run integration-tests`, - `pixi run script-tests`. - -Notes: - -- `pixi run fix` regenerates `docs/dev/package-structure/full.md` and - `docs/dev/package-structure/short.md` automatically — never edit those - by hand. Don't review auto-fixes; accept and move on. Then - `pixi run check` until clean. -- When a check command needs saved output for analysis, capture the log - and preserve the command exit code with a zsh-safe variable name: - `pixi run check > /tmp/easydiffraction-check.log 2>&1; check_exit_code=$?; tail -n 200 /tmp/easydiffraction-check.log; exit $check_exit_code`. - Never assign to `status` in zsh; it is readonly. Use task-specific - names such as `check_exit_code`, `unit_tests_exit_code`, or - `script_tests_exit_code`. -- Open issues / design questions / planned improvements live in - `docs/dev/issues/open.md` (priority-ordered). On resolution, move to - `docs/dev/issues/closed.md` and update the relevant ADR or - `docs/dev/adrs/index.md` if affected. - -### Planning - -When asked to create a plan: - -- Start the plan by referencing this file: - `.github/copilot-instructions.md`. State any deliberate exception to - these instructions in the plan itself. -- First gather enough repository context to make the plan concrete. Ask - all ambiguous or unclear questions in one concise batch; record - unresolved questions in the plan if the user wants it saved before - answering them. -- Save plans as `docs/dev/plans/.md` (lowercase, - dash-separated, e.g. `docs/dev/plans/background-refactor.md`). When a - plan implements one ADR, use the same slug as the ADR file; for - example, `docs/dev/adrs/suggestions/foo.md` maps to - `docs/dev/plans/foo.md`. If a plan has no corresponding ADR or spans - multiple ADRs, choose a concise feature slug and list all related ADRs - in the plan. Use the same `` as a **flat-slug - implementation branch** off `develop` (no `feature/` prefix — - e.g. `emcee-minimizer`, not `feature/emcee-minimizer`). PRs target - `develop`, not `master`. Do not push the branch unless asked. -- Include a status checklist with `[ ]` items; mark `[x]` as completed - during implementation. -- Apply the two-phase workflow (Phase 1 implementation, Phase 2 - verification) to non-trivial plans. Stop after Phase 1 and ask the - user to review before starting Phase 2. -- The plan must explicitly state that, when an AI agent follows it, - every completed Phase 1 implementation step must be staged with - explicit paths and committed locally before moving to the next - implementation step or the Phase 1 review gate. Follow the rules in - **Commits**. Keep commits atomic, single-purpose, and aligned with - plan steps. -- If implementation uncovers a serious requirement, risk, design issue, - or scope change not covered by the plan, stop and ask the user for - clarification or approval before proceeding. Record the unresolved - issue in the plan when useful. -- The plan should be easy to maintain while working: include concrete - files likely to change, decisions already made, open questions, - verification commands for Phase 2, and a short suggested commit - message or branch name when useful. -- Verification commands in plans must include the zsh-safe log-capture - pattern from **Workflow** whenever saved output is needed for later - analysis. -- Before saving a plan, verify that referenced files, directories, - scripts, and task names exist locally when that is practical. If a - referenced tool is optional or missing, include an available fallback. -- End every plan with a "Suggested Pull Request" section containing a - short PR title and a brief end-user-oriented description. Keep this - section non-technical enough for scientists and other users to - understand the benefit. Update it during implementation if extra - approved changes become important enough to mention in the PR title or - description. -- When replying to a plan review, save the reply alongside the review. - Reviews live at `docs/dev/plans/_review-.md`; the - matching reply goes to `docs/dev/plans/_reply-.md` - (same slug, same number, swap `review` → `reply`). One reply file per - review file; do not bundle replies to multiple reviews into one - document. Structure the reply with one section per finding, each - containing a verdict (agree / disagree / partial), the action taken in - the plan, and a pointer to the affected plan section. After updating - the plan, also update the reply if a numbered step shifts so that - cross-references stay accurate. - -## Agent Shortcuts - -When the user's message starts with one of these literal keywords, -treat it as an operational command, not as ordinary prose. Execute the -matching task instead of asking for the full instructions every time. -The first non-whitespace token is the trigger; arguments follow on the -same line. This applies in future turns and after context compaction: -if the newest user message begins with `/draft-adr`, `/review-adr`, -`/draft-plan`, or `/review-plan`, enter that shortcut's stateful loop. - -Common preamble for every shortcut (run once at task start): - -- Ask the user, in one batch, for any permission grants needed to - run unattended for the full task. At minimum: `Bash` with - `run_in_background` for polling, `Edit`/`Write` on `docs/`, the - shell primitives `git`, `until`, `sleep`, `ls`, `grep`. Cite this - section so the user knows why. -- Stay on the current branch. Do not switch or create branches. -- Polling cadence is 60 s. Use Bash with `run_in_background` and an - `until [ -f ]; do sleep 60; done` body so the harness - notifies you when the awaited file appears. Do not poll inline. - If the current harness exposes a native recurring automation or - heartbeat mechanism instead of background Bash, use that mechanism - with the same cadence and file target. Do not downgrade the shortcut - into a one-shot review/reply because background Bash is unavailable. -- Filename suffixes follow the existing convention: - `_review-N.md` and `_reply-N.md` next to the parent - ADR or plan, where `N` is one greater than the highest existing - number for that stem (starting at 1). -- Each shortcut runs autonomously. Do not pause for confirmation - between rounds; auto-apply every finding. Only stop when the - termination condition for that shortcut is met, or the user sends - an explicit message asking you to stop or change direction. -- Before starting any poll, first check whether the file being awaited - already exists. If it does, process it immediately, then continue the - loop from the next expected suffix. -- Never stop after writing only the first review, first reply, or first - draft. After every loop action, immediately arrange the next poll - unless the shortcut's termination condition has been reached. -- **Final-review sentinel.** When a reviewer shortcut decides no - findings remain, it writes a final review file whose body begins - with the literal sentinel line - `**No findings. Ready to commit.**` on its own line (immediately - after the title and any boilerplate header). The matching author - shortcut, before writing a reply, runs - `grep -q "^\*\*No findings\. Ready to commit\.\*\*$" - docs/dev/{adrs/suggestions,plans}/_review-.md` on the - newest review. If the sentinel is found, the author shortcut - **stops the polling loop** without writing a reply and reports - that the review cycle is closed. The sentinel is the only - termination signal between author and reviewer — no other phrase - triggers it. -- **Existing shortcuts never delete files and never commit.** The - four shortcuts in this section - (`/draft-adr`, `/review-adr`, `/draft-plan`, `/review-plan`) - only read existing files and write new ones. They never call - `git rm`, `git add`, or `git commit`. Cleanup and commit happen - in a separate dedicated shortcut (`/implement-plan`). - -### `/draft-adr ` - -Act as the ADR author. Draft an ADR suggestion, then respond to -incoming reviews in a polling loop. - -**Setup (once):** - -1. Run the common preamble. -2. Pick a flat lowercase-dash slug from ``. Save the ADR at - `docs/dev/adrs/suggestions/.md` using the project's ADR - template (Status: Proposed; Context; Decision; Consequences; - Alternatives Considered; Deferred Work as needed). -3. Start polling for `_review-1.md` next to the ADR. - -**Loop (per tick):** - -- When `_review-N.md` appears, first check for the - final-review sentinel (see Common preamble). If - `**No findings. Ready to commit.**` is the first body line, - stop the polling loop: report that the review cycle is closed - and that `/implement-plan` (or a manual commit) is the next - step. Do not write a reply. -- Otherwise read the review, update the ADR to address every - finding, and write `_reply-N.md` with one section per - finding (verdict + action taken + pointer to the affected ADR - section). -- Start polling for `_review-(N+1).md`. -- **Do not commit and do not delete any file.** Leave every edit - in the worktree as modified or untracked. Commit and cleanup - happen later in `/implement-plan`. - -**Termination:** either the sentinel-driven stop above or an -explicit user message asking you to stop or change direction. - -### `/review-adr []` - -Act as the ADR reviewer. Review an ADR suggestion in a polling loop -until all findings are addressed. - -**Setup (once):** - -1. Run the common preamble. -2. Identify the target ADR. If `` is given, target - `docs/dev/adrs/suggestions/.md`. Otherwise monitor - `docs/dev/adrs/suggestions/` for a `.md` with no existing - `_review-*.md`. If none exists, report "no ADR has - appeared yet" and start polling for it. -3. When the ADR appears, run a static review per - [`.github/copilot-instructions.md`](.github/copilot-instructions.md) - → **Change Discipline** plan-review rule. **Do not run tests, - lint, build, formatters, or any `pixi` command — ADR reviews are - static reads only.** Save the review at `_review-1.md` - next to the ADR. - -**Loop (per tick):** - -- Poll for `_reply-N.md` matching the most recent review. -- When the reply appears, re-read the ADR against the new reply - and every prior review/reply. Pick exactly one branch: - - **Findings remain:** write `_review-(N+1).md` listing - only the open findings, then poll for the next reply. - - **All findings addressed:** write the final review file - `_review-(N+1).md` whose body begins with the - final-review sentinel from the Common preamble: - `**No findings. Ready to commit.**` on its own line, followed - by a short paragraph summarising the round (no findings list). - Then **stop the loop**. **Do not delete any review or reply - file. Do not commit anything.** Cleanup and commit happen in - `/implement-plan`. - -**Termination:** either the sentinel-written final review above -or an explicit user message asking you to stop or change -direction. - -### `/draft-plan []` - -Act as the implementation-plan author. Same loop as `/draft-adr`, -applied to an implementation plan instead of an ADR. - -- Source ADR: if `` is given, target the ADR at - `docs/dev/adrs/accepted/.md` (or `suggestions/` if the ADR - is not yet promoted). Otherwise pick the newest accepted ADR - matching the most recent `/review-adr` cycle. Use the same slug - for the plan. -- Save the plan at `docs/dev/plans/.md` using the project's - plan template per - [`.github/copilot-instructions.md`](.github/copilot-instructions.md) - → **Planning**: ADR cross-reference, branch + PR notes, - Decisions, Open questions, Concrete files, Phase 1 steps with - status checklist, Phase 2 verification commands using the - zsh-safe log-capture pattern, and a Suggested Pull Request - section. -- Loop behaviour identical to `/draft-adr`, including the - sentinel-driven stop: before writing a reply, check the newest - `docs/dev/plans/_review-N.md` for the final-review - sentinel and stop if present. Otherwise read the review, update - the plan, write `_reply-N.md`, poll for the next review. - **No commits and no file deletions in this loop.** - -**Termination:** either the sentinel-driven stop or an explicit -user message asking you to stop or change direction. - -### `/review-plan []` - -Act as the implementation-plan reviewer. Same loop as `/review-adr`, -applied to an implementation plan. - -- Target: if `` is given, target - `docs/dev/plans/.md`. Otherwise monitor `docs/dev/plans/` - for the newest plan matching the most recent `/draft-plan` - cycle. -- Static reads only — same "no tests / lint / build / formatters / - pixi" rule applies during plan reviews. -- Loop and termination identical to `/review-adr`, including the - final-review-sentinel rule: when all findings are addressed, - write the final review with `**No findings. Ready to commit.**` - as the first body line and stop. **Do not delete any review or - reply file and do not commit anything** — those steps belong to - `/implement-plan`. -- A bare `/review-plan` is still enough to start the full reviewer - loop. If a target plan already exists, write the first static review - and then immediately wait for `_reply-1.md`; do not return a - final answer that implies the task is complete after the first review. - -### `/implement-plan []` - -Clean up the review/reply deliberation history, commit the latest -ADR + plan, then execute the plan's Phase 1 implementation steps -autonomously. This is the only shortcut in this section that -deletes files and commits changes. - -**Setup (once):** - -1. Ask the user, **in one batch**, to grant every permission this - shortcut will need so the implementation phase runs without - per-step prompts. List explicitly: - - `Bash` (general, including `run_in_background`) for `git`, - `ls`, `grep`, `find`, `until`, `sleep`, and any plan-step - command (`pixi run notebook-prepare`, etc.). - - `Read` / `Edit` / `Write` on `src/`, `tests/`, - `docs/dev/adrs/`, `docs/dev/plans/`, `docs/docs/tutorials/`, - `pyproject.toml`, `pixi.toml`, `pixi.lock`, and any other - paths the plan's `Concrete files likely to change` section - enumerates. - - `git rm`, `git add`, `git commit` (subsumed under Bash). - - Cite this section so the user knows the scope. -2. **Do not** run tests, lint, `pixi run fix`, `pixi run check`, - or the integration/script-test suites during this shortcut — - those belong to the plan's Phase 2 verification gate and run - only after the Phase 1 review gate the plan defines. -3. Stay on the current branch. -4. Identify targets: - - **Plan**: if `` is given, target - `docs/dev/plans/.md`. Otherwise pick the most recently - modified plan in `docs/dev/plans/` whose stem matches the - current branch slug. - - **ADR**: derive from the plan's `## ADR` reference. The ADR - usually lives in `docs/dev/adrs/suggestions/.md` at - this point (a Phase 1 plan step typically promotes it to - `accepted/`); accept either location. - -**Phase A — cleanup + commits (two atomic commits):** - -1. **ADR commit.** - - `git rm` every `docs/dev/adrs//_review-*.md` - and `_reply-*.md` next to the ADR. - - `git add docs/dev/adrs//.md`. - - Commit with message `Add ADR suggestion` (or - `Promote ADR to accepted` if the ADR was already - under `accepted/`). -2. **Plan commit.** - - `git rm` every `docs/dev/plans/_review-*.md` and - `_reply-*.md`. - - `git add docs/dev/plans/.md`. - - Commit with message `Add implementation plan`. - -If the ADR or plan has no review/reply siblings (e.g. the user -drafted it directly without running the loop), skip the `git rm` -step but still stage + commit the clean artifact. - -**Phase B — implementation:** - -3. Parse the plan's `## Implementation steps (Phase 1)` section - in order. Each `- [ ] **P1.X — **` item is one - atomic commit. For each step in turn: - 1. Read the step body to identify the files and edits. - 2. Apply the edits exactly as the step prescribes. If the - step says to run a build-step command (e.g. - `pixi run notebook-prepare`, `pixi lock`), run it — those - are part of the plan, not Phase 2 verification. - 3. Stage the files the step enumerates with explicit paths - (per **Commits** rule "Stage only the files modified for - the step"). Do not stage unrelated dirty files. - 4. Commit with the suggested commit message from the step - (the `Commit:` line). Edit the `- [ ]` checkbox to `- [x]` - in the plan file before staging, so the checklist tracks - progress, and include the checklist update in the same - commit. -4. **Phase 1 review gate.** The plan's final P1 step is always - "Phase 1 review gate. No code change. Stop and request user - review." When you reach that step, mark it `[x]`, commit the - checklist update alone, and **stop**. Report: - - How many P1 commits landed. - - The current branch and tip commit hash. - - That Phase 2 verification (`pixi run fix`, `pixi run check`, - `pixi run unit-tests`, `pixi run integration-tests`, - `pixi run script-tests`) is still pending and is the user's - next action. - -**If a step fails:** stop immediately, do not skip ahead, report -the failure with the step ID and the error. Do not run Phase 2 -verification commands as a debugging tool — fix the step or wait -for user input. - -**Permissions / approval flow.** The setup-step permission batch -is the only prompt this shortcut emits during normal operation. -If the plan introduces a step that requires a permission the user -did not pre-approve (e.g. an `npm` invocation when the upfront -batch only covered `pixi`), pause at that step and ask for the -extra grant rather than blocking on a single tool call. Resume -the loop once granted. diff --git a/.gitignore b/.gitignore index ec2e0d270..cb7130005 100644 --- a/.gitignore +++ b/.gitignore @@ -45,6 +45,10 @@ CMakeLists.txt.user* *.log *.zip +# Agents +AGENTS.md +CLAUDE.md + # ED # Used to fetch tutorials data during their runtime. Need to have '/' at # the beginning to avoid ignoring 'data' module in the src/. diff --git a/AGENTS.md b/AGENTS.md deleted file mode 100644 index 60be90eb9..000000000 --- a/AGENTS.md +++ /dev/null @@ -1,5 +0,0 @@ -# Agent Instructions - -Follow -[`.github/copilot-instructions.md`](.github/copilot-instructions.md) for -this repository. diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 100644 index b21d16db8..000000000 --- a/CLAUDE.md +++ /dev/null @@ -1 +0,0 @@ -@.github/copilot-instructions.md diff --git a/docs/dev/adrs/accepted/minimizer-category-consolidation.md b/docs/dev/adrs/accepted/minimizer-category-consolidation.md index ed1e24f36..8d41ae6af 100644 --- a/docs/dev/adrs/accepted/minimizer-category-consolidation.md +++ b/docs/dev/adrs/accepted/minimizer-category-consolidation.md @@ -403,10 +403,9 @@ category's class-level `_engine_metadata` dict. - Hand-editing CIF to switch minimizer types requires touching both `_fitting.minimizer_type` and the relevant `_minimizer.*` tags. - Existing projects saved under the seven-category layout cannot load - unchanged. The project is in beta; per - `.github/copilot-instructions.md` "no legacy shims" applies. Saved - fixtures under `tmp/tutorials/projects/` are regenerated by the - implementation plan. + unchanged. The project is in beta; per `AGENTS.md` "no legacy shims" + applies. Saved fixtures under `tmp/tutorials/projects/` are + regenerated by the implementation plan. ### ADRs amended by this ADR diff --git a/docs/dev/adrs/accepted/switchable-category-owned-selectors.md b/docs/dev/adrs/accepted/switchable-category-owned-selectors.md index f74fdf849..8d39fbfcd 100644 --- a/docs/dev/adrs/accepted/switchable-category-owned-selectors.md +++ b/docs/dev/adrs/accepted/switchable-category-owned-selectors.md @@ -564,10 +564,9 @@ categories and their swap hooks at the start of Phase 1. ### 7. Beta posture: hard cutover, no shims -[`.github/copilot-instructions.md`](../../../../.github/copilot-instructions.md) -→ **Change Discipline**: "Project is in beta: no legacy shims, no -deprecation warnings — update tests and tutorials to the current API." -This ADR keeps that posture: +[`AGENTS.md`](../../../../AGENTS.md) → **Change Discipline**: "Project +is in beta: no legacy shims, no deprecation warnings — update tests and +tutorials to the current API." This ADR keeps that posture: - `._type` is **deleted**, not deprecated. - `show_supported__types()` / `show_current__type()` are @@ -872,10 +871,9 @@ full grep results.) Replace `self.__class__` with the new concrete class so the user's reference keeps pointing at "the same object". Rejected: -[`.github/copilot-instructions.md`](../../../../.github/copilot-instructions.md) -→ **Architecture** forbids it ("no monkey-patching or runtime class -mutation"). It would also confuse `isinstance` checks and break -descriptor introspection. +[`AGENTS.md`](../../../../AGENTS.md) → **Architecture** forbids it ("no +monkey-patching or runtime class mutation"). It would also confuse +`isinstance` checks and break descriptor introspection. ### B. Single category class with internal mode switching diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index 277d74b3c..0930cd93e 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -1,8 +1,7 @@ # Plan: Emcee Minimizer -> This plan follows -> [`.github/copilot-instructions.md`](../../../.github/copilot-instructions.md). -> No deliberate exceptions. +> This plan follows [`AGENTS.md`](../../../AGENTS.md). No deliberate +> exceptions. ## Prerequisite @@ -15,35 +14,33 @@ This plan depends on three accepted ADRs, all merged on `develop`: — `analysis.minimizer.type = 'X'` is the writable surface; no owner-level `._type` shims. - [`minimizer-input-output-split`](../adrs/accepted/minimizer-input-output-split.md) - — fit-filled outputs live on a paired `analysis.fit_result` - instance (`LeastSquaresFitResult` or `BayesianFitResult`), not on - the minimizer. + — fit-filled outputs live on a paired `analysis.fit_result` instance + (`LeastSquaresFitResult` or `BayesianFitResult`), not on the + minimizer. emcee inherits the entire paired surface for free: `BayesianMinimizerBase._fit_result_class = BayesianFitResult`, so -`Analysis._swap_minimizer` instantiates both -`EmceeMinimizer` (inputs) and `BayesianFitResult` (outputs) -atomically. +`Analysis._swap_minimizer` instantiates both `EmceeMinimizer` (inputs) +and `BayesianFitResult` (outputs) atomically. ## ADR -This plan implements the emcee follow-on described in §1, §5, §6 and -§9 of +This plan implements the emcee follow-on described in §1, §5, §6 and §9 +of [`docs/dev/adrs/accepted/minimizer-category-consolidation.md`](../adrs/accepted/minimizer-category-consolidation.md). No new ADR is required. -If implementation uncovers a design question not covered by the -ADRs (e.g. resume semantics on parameter-set mismatch, or -`InitializationMethodEnum` ↔ -`DreamPopulationInitializationEnum` reconciliation), stop and ask -before proceeding. +If implementation uncovers a design question not covered by the ADRs +(e.g. resume semantics on parameter-set mismatch, or +`InitializationMethodEnum` ↔ `DreamPopulationInitializationEnum` +reconciliation), stop and ask before proceeding. ## Branch and PR - Branch: `emcee-minimizer`. Do not push unless asked. - Each step in §"Implementation steps (Phase 1)" must be staged with explicit paths and committed locally **before** moving to the next - step. See `.github/copilot-instructions.md` → **Commits**. + step. See `AGENTS.md` → **Commits**. - After P1.8, stop and wait for the user review gate before starting Phase 2. @@ -53,68 +50,68 @@ before proceeding. `MinimizerTypeEnum.EMCEE = 'emcee'`. Selected via `project.analysis.minimizer.type = 'emcee'`. 2. emcee inherits two paired surfaces from `BayesianMinimizerBase`: - - **Settings** (writable inputs): `sampling_steps`, - `burn_in_steps`, `thinning_interval`, `population_size`, - `parallel_workers`, `initialization_method`, `random_seed` — all - declared by `BayesianMinimizerBase.__init__`. emcee may override - class-level defaults (e.g. `sampling_steps=5000`, - `population_size=32`) and adds one emcee-specific input - `proposal_moves`. - - **Outputs** (fit-filled, internal `_set_*`): `acceptance_rate_mean`, - `gelman_rubin_max`, `effective_sample_size_min`, - `best_log_posterior`, `point_estimate_name`, - `sampler_completed`, `credible_interval_inner/outer` — already - on `BayesianFitResult`, not on the minimizer. emcee's projection - writer calls `self.fit_result._set_*` (not `self.minimizer._set_*`). + - **Settings** (writable inputs): `sampling_steps`, `burn_in_steps`, + `thinning_interval`, `population_size`, `parallel_workers`, + `initialization_method`, `random_seed` — all declared by + `BayesianMinimizerBase.__init__`. emcee may override class-level + defaults (e.g. `sampling_steps=5000`, `population_size=32`) and + adds one emcee-specific input `proposal_moves`. + - **Outputs** (fit-filled, internal `_set_*`): + `acceptance_rate_mean`, `gelman_rubin_max`, + `effective_sample_size_min`, `best_log_posterior`, + `point_estimate_name`, `sampler_completed`, + `credible_interval_inner/outer` — already on `BayesianFitResult`, + not on the minimizer. emcee's projection writer calls + `self.fit_result._set_*` (not `self.minimizer._set_*`). 3. Verbose attribute names map to emcee native kwargs via - `EmceeMinimizer._native_key_map` (overrides the DREAM-style - defaults on `BayesianMinimizerBase._native_key_map`): - - | Verbose | emcee native engine attribute | - | --- | --- | - | `sampling_steps` | `nsteps` | - | `burn_in_steps` | `nburn` | - | `thinning_interval` | `thin` | - | `population_size` | `nwalkers` | - | `parallel_workers` | `parallel_workers` (engine integer; **not** mapped directly to emcee's `pool`) | - | `initialization_method` | (custom — see §6) | - | `random_seed` | `random_seed` | - - **`parallel_workers` semantics.** The category persists an - integer; the engine class holds it under the same name. emcee's + `EmceeMinimizer._native_key_map` (overrides the DREAM-style defaults + on `BayesianMinimizerBase._native_key_map`): + + | Verbose | emcee native engine attribute | + | ----------------------- | ------------------------------------------------------------------------------ | + | `sampling_steps` | `nsteps` | + | `burn_in_steps` | `nburn` | + | `thinning_interval` | `thin` | + | `population_size` | `nwalkers` | + | `parallel_workers` | `parallel_workers` (engine integer; **not** mapped directly to emcee's `pool`) | + | `initialization_method` | (custom — see §6) | + | `random_seed` | `random_seed` | + + **`parallel_workers` semantics.** The category persists an integer; + the engine class holds it under the same name. emcee's `EnsembleSampler` takes a `pool=` argument that is a pool object - (anything with a `.map` method) or `None` for serial execution — - not an integer. The engine builds the actual pool around - `run_mcmc`: + (anything with a `.map` method) or `None` for serial execution — not + an integer. The engine builds the actual pool around `run_mcmc`: - | `parallel_workers` value | Engine behaviour | - | --- | --- | - | `1` | `pool=None` (serial); single process | - | `0` | `multiprocessing.Pool(os.cpu_count())` | - | `N > 1` | `multiprocessing.Pool(N)` | + | `parallel_workers` value | Engine behaviour | + | ------------------------ | -------------------------------------- | + | `1` | `pool=None` (serial); single process | + | `0` | `multiprocessing.Pool(os.cpu_count())` | + | `N > 1` | `multiprocessing.Pool(N)` | The pool is closed in a `finally:` block after `run_mcmc` returns. - `Analysis._sync_engine_from_minimizer_category` must therefore - skip `parallel_workers` from the native-attribute sync (it - already skips `random_seed` for the same "engine handles it" - reason; add `parallel_workers` to the - `_engine_sync_skip_keys` frozenset introduced in P1.5). -4. Resume uses emcee's `HDFBackend` against the `/emcee_chain` group - of the same `analysis/results.h5` file used by the snapshot - writer. No separate sidecar file. A non-resume `fit()` follows - the prerequisite plan's lifecycle and **truncates** `results.h5` - (after the standard warning); resume opens it in append mode. + `Analysis._sync_engine_from_minimizer_category` must therefore skip + `parallel_workers` from the native-attribute sync (it already skips + `random_seed` for the same "engine handles it" reason; add + `parallel_workers` to the `_engine_sync_skip_keys` frozenset + introduced in P1.5). + +4. Resume uses emcee's `HDFBackend` against the `/emcee_chain` group of + the same `analysis/results.h5` file used by the snapshot writer. No + separate sidecar file. A non-resume `fit()` follows the prerequisite + plan's lifecycle and **truncates** `results.h5` (after the standard + warning); resume opens it in append mode. **`Project.save()` must not delete `/emcee_chain`.** The current `write_analysis_results_sidecar` ([`results_sidecar.py:282`](../../../src/easydiffraction/io/results_sidecar.py)) - opens `results.h5` with mode `'w'` (full truncate). Every save - after a fit will erase the emcee chain unless this writer is - modified. After this plan, the writer opens the file in append - mode (`'a'`) and replaces only the EasyDiffraction-canonical - groups (`/posterior`, `/distribution_cache`, `/pair_cache`, - `/predictive`) by deleting them first if present, leaving any - other top-level groups (currently only `/emcee_chain`) untouched. + opens `results.h5` with mode `'w'` (full truncate). Every save after + a fit will erase the emcee chain unless this writer is modified. + After this plan, the writer opens the file in append mode (`'a'`) and + replaces only the EasyDiffraction-canonical groups (`/posterior`, + `/distribution_cache`, `/pair_cache`, `/predictive`) by deleting them + first if present, leaving any other top-level groups (currently only + `/emcee_chain`) untouched. **Non-resume truncate must still happen — and explicitly.** The current `_warn_results_sidecar_overwrite` @@ -122,99 +119,94 @@ before proceeding. / [`warn_analysis_results_sidecar_overwrite`](../../../src/easydiffraction/io/results_sidecar.py) only **warns**; it does not delete. With the writer switched to - append mode, a fresh non-resume emcee fit after an older emcee - run would otherwise leave the stale `/emcee_chain` group in - `results.h5`. This plan therefore adds a real preparation step - that warns **and removes** the file before the engine starts. See - P1.5a; the resume path bypasses it. -5. `Analysis.fit()` gains an optional `resume=True, extra_steps=N` - call shape for minimizers that support incremental sampling. For - other minimizers, passing `resume=True` raises immediately. -6. emcee outputs translate to the existing `BayesianFitResults` - runtime shape (plural — note the singular `BayesianFitResult` is - the persistence category, not the runtime object) exactly as - DREAM does — same `PosteriorSamples`, `PosteriorParameterSummary`, - etc. Plotting and display code needs no specialization. -7. Two `EmceeMinimizer` classes coexist with the same DREAM - precedent: - - `src/easydiffraction/analysis/categories/minimizer/emcee.py` - — the persisted **category** class (`BayesianMinimizerBase` - subclass) used for CIF persistence and the user-facing setter - surface. + append mode, a fresh non-resume emcee fit after an older emcee run + would otherwise leave the stale `/emcee_chain` group in `results.h5`. + This plan therefore adds a real preparation step that warns **and + removes** the file before the engine starts. See P1.5a; the resume + path bypasses it. + +5. `Analysis.fit()` gains an optional `resume=True, extra_steps=N` call + shape for minimizers that support incremental sampling. For other + minimizers, passing `resume=True` raises immediately. +6. emcee outputs translate to the existing `BayesianFitResults` runtime + shape (plural — note the singular `BayesianFitResult` is the + persistence category, not the runtime object) exactly as DREAM does — + same `PosteriorSamples`, `PosteriorParameterSummary`, etc. Plotting + and display code needs no specialization. +7. Two `EmceeMinimizer` classes coexist with the same DREAM precedent: + - `src/easydiffraction/analysis/categories/minimizer/emcee.py` — the + persisted **category** class (`BayesianMinimizerBase` subclass) + used for CIF persistence and the user-facing setter surface. - `src/easydiffraction/analysis/minimizers/emcee.py` — the live - **engine** class registered with the engine - `MinimizerFactory`. Holds the `emcee.EnsembleSampler` instance - and runs the sampler. - This mirrors the existing `BumpsDreamMinimizer` split between - `categories/minimizer/bumps_dream.py` and - `minimizers/bumps_dream.py`. + **engine** class registered with the engine `MinimizerFactory`. + Holds the `emcee.EnsembleSampler` instance and runs the sampler. + This mirrors the existing `BumpsDreamMinimizer` split between + `categories/minimizer/bumps_dream.py` and + `minimizers/bumps_dream.py`. ## Open questions -- **Resume after parameter-set change.** If the user fits, then - edits which parameters are free, then calls - `fit(resume=True, ...)`, emcee's `HDFBackend.shape` mismatches the - current parameter count. Plan default: detect mismatch in P1.5 and - raise `ValueError` with a clear "start a fresh run" message. -- **Resume after a non-emcee fit.** If the user runs DREAM, switches - to emcee, and calls `fit(resume=True, ...)`, the `/emcee_chain` - group will be missing. Plan default: raise `ValueError` pointing - at the prerequisite ADR's lifecycle rule ("a new fit overwrites - the file"). +- **Resume after parameter-set change.** If the user fits, then edits + which parameters are free, then calls `fit(resume=True, ...)`, emcee's + `HDFBackend.shape` mismatches the current parameter count. Plan + default: detect mismatch in P1.5 and raise `ValueError` with a clear + "start a fresh run" message. +- **Resume after a non-emcee fit.** If the user runs DREAM, switches to + emcee, and calls `fit(resume=True, ...)`, the `/emcee_chain` group + will be missing. Plan default: raise `ValueError` pointing at the + prerequisite ADR's lifecycle rule ("a new fit overwrites the file"). - **`InitializationMethodEnum` ↔ `DreamPopulationInitializationEnum` reconciliation** (deferred from review-8 F6 of the consolidation - work). emcee uses different init methods (`ball`, `uniform`, - `prior`); DREAM exposes a broader engine-level enum with `EPS`, - `COV`, `LHS`, `RANDOM`. The persisted user-facing enum - (`InitializationMethodEnum`) is narrower - (`LATIN_HYPERCUBE`, `BALL`, `UNIFORM`, `PRIOR`). P1.3 must decide - whether to narrow DREAM's engine enum to match, or accept the - asymmetry. Recommend: keep DREAM's broader engine enum (legacy - DREAM users may pass `EPS`/`COV`/`RANDOM` directly to the engine) - but document that only `LATIN_HYPERCUBE` is persistable for DREAM; - emcee accepts `BALL`/`UNIFORM`/`PRIOR` only. -- **Move-mix semantics.** emcee supports proposal-move mixtures - (e.g. 70 % stretch + 30 % differential evolution). The - consolidation ADR §5 exposes `proposal_moves` as a single string - descriptor. Plan default: limit `proposal_moves` to single-move - strings for v1 (`stretch`, `de`, `de_snooker`, `walk`). Mixtures - deferred to a follow-on plan. Record this in the descriptor's - `description=` text. + work). emcee uses different init methods (`ball`, `uniform`, `prior`); + DREAM exposes a broader engine-level enum with `EPS`, `COV`, `LHS`, + `RANDOM`. The persisted user-facing enum (`InitializationMethodEnum`) + is narrower (`LATIN_HYPERCUBE`, `BALL`, `UNIFORM`, `PRIOR`). P1.3 must + decide whether to narrow DREAM's engine enum to match, or accept the + asymmetry. Recommend: keep DREAM's broader engine enum (legacy DREAM + users may pass `EPS`/`COV`/`RANDOM` directly to the engine) but + document that only `LATIN_HYPERCUBE` is persistable for DREAM; emcee + accepts `BALL`/`UNIFORM`/`PRIOR` only. +- **Move-mix semantics.** emcee supports proposal-move mixtures (e.g. 70 + % stretch + 30 % differential evolution). The consolidation ADR §5 + exposes `proposal_moves` as a single string descriptor. Plan default: + limit `proposal_moves` to single-move strings for v1 (`stretch`, `de`, + `de_snooker`, `walk`). Mixtures deferred to a follow-on plan. Record + this in the descriptor's `description=` text. ## Cleanup opportunities inherited from earlier work The input/output-split work left four cleanup items still open in -[`docs/dev/issues/open.md`](../issues/open.md) that touch code this -plan modifies. Fold them in opportunistically while the surrounding -code is already being edited; the plan does not block on them. +[`docs/dev/issues/open.md`](../issues/open.md) that touch code this plan +modifies. Fold them in opportunistically while the surrounding code is +already being edited; the plan does not block on them. - **#100 — Collapse duplicate predictive-cache-key helpers.** `Analysis._predictive_cache_key` ([analysis.py:528](../../../src/easydiffraction/analysis/analysis.py)) and `Plotter._posterior_predictive_key` ([plotting.py:3823](../../../src/easydiffraction/display/plotting.py)) - build the identical string; keep one canonical helper. P1.6 may - touch the predictive plotting path while validating emcee - posterior emission. -- **#101 — Remove dead branch in - `Analysis._fit_state_categories`.** Both branches return the same - list since the Bayesian categories were absorbed. One-line fix at + build the identical string; keep one canonical helper. P1.6 may touch + the predictive plotting path while validating emcee posterior + emission. +- **#101 — Remove dead branch in `Analysis._fit_state_categories`.** + Both branches return the same list since the Bayesian categories were + absorbed. One-line fix at [analysis.py:1184-1205](../../../src/easydiffraction/analysis/analysis.py). - **#102 — Drop compute-and-ignore `result_kind` validation.** `_restore_persisted_fit_state` ([serialize.py:590-606](../../../src/easydiffraction/io/cif/serialize.py)) - calls `FitResultKindEnum(result_kind_value)` for its side effect - only. Move the warning into the descriptor setter, or extract a - validator helper. + calls `FitResultKindEnum(result_kind_value)` for its side effect only. + Move the warning into the descriptor setter, or extract a validator + helper. - **#103 — Make `_sync_engine_from_minimizer_category` skip-keys declarative.** This plan adds `proposal_moves` as a second engine-level "ambient" key (alongside `random_seed`). The current magic-string skip at [analysis.py:1138](../../../src/easydiffraction/analysis/analysis.py) should become a class-level - `_engine_sync_skip_keys: ClassVar[frozenset[str]] = frozenset(...)` - on `MinimizerCategoryBase` before the second member lands. - Recommend addressing as part of P1.5. + `_engine_sync_skip_keys: ClassVar[frozenset[str]] = frozenset(...)` on + `MinimizerCategoryBase` before the second member lands. Recommend + addressing as part of P1.5. When the matching open-issue is fully resolved, move it to [`closed.md`](../issues/closed.md) and update @@ -228,46 +220,43 @@ When the matching open-issue is fully resolved, move it to persisted **category** class `EmceeMinimizer(BayesianMinimizerBase)` with class-level defaults, `_engine_metadata`, and the overridden `_native_key_map`. -- `src/easydiffraction/analysis/minimizers/emcee.py` — live - **engine** class `EmceeMinimizer(BayesianMinimizerEngineBase or - equivalent)` registered with `MinimizerFactory`. Holds - `emcee.EnsembleSampler` and the `HDFBackend`. +- `src/easydiffraction/analysis/minimizers/emcee.py` — live **engine** + class `EmceeMinimizer(BayesianMinimizerEngineBase or equivalent)` + registered with `MinimizerFactory`. Holds `emcee.EnsembleSampler` and + the `HDFBackend`. - `tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py`. - `tests/unit/easydiffraction/analysis/minimizers/test_emcee.py`. -- `tests/integration/fitting/test_emcee.py` (cross-check vs DREAM on - a shared toy fit; assert posterior medians agree to within - tolerance). -- `docs/docs/tutorials/ed-25.py` (emcee + resume tutorial). The - next free tutorial slot — `ed-23` is already the "Co2SiO4 - Sequential Fit" tutorial and `ed-24` is the "LBCO Bayesian - Display" tutorial. Verify `ed-25.py` is unused before creating - it at P1.7 start; bump if a newer slot is already occupied. +- `tests/integration/fitting/test_emcee.py` (cross-check vs DREAM on a + shared toy fit; assert posterior medians agree to within tolerance). +- `docs/docs/tutorials/ed-25.py` (emcee + resume tutorial). The next + free tutorial slot — `ed-23` is already the "Co2SiO4 Sequential Fit" + tutorial and `ed-24` is the "LBCO Bayesian Display" tutorial. Verify + `ed-25.py` is unused before creating it at P1.7 start; bump if a newer + slot is already occupied. ### Modified - `src/easydiffraction/analysis/minimizers/enums.py` — add `MinimizerTypeEnum.EMCEE = 'emcee'`. -- `src/easydiffraction/analysis/categories/minimizer/__init__.py` - — add explicit `EmceeMinimizer` (category) import so registration - fires. +- `src/easydiffraction/analysis/categories/minimizer/__init__.py` — add + explicit `EmceeMinimizer` (category) import so registration fires. - `src/easydiffraction/analysis/minimizers/__init__.py` (or the factory's package init) — add explicit `EmceeMinimizer` (engine) import so engine registration fires. -- `src/easydiffraction/analysis/categories/minimizer/bayesian_base.py` - — only if review-8 F6 reconciliation (open question above) calls - for narrowing the persisted enum surface; otherwise unchanged. -- `src/easydiffraction/analysis/analysis.py` — `fit()` signature - gains `resume: bool = False, extra_steps: int | None = None`; - validation + dispatch to engine. Wire `_engine_sync_skip_keys` - (#103) before adding `proposal_moves` to the ambient set. +- `src/easydiffraction/analysis/categories/minimizer/bayesian_base.py` — + only if review-8 F6 reconciliation (open question above) calls for + narrowing the persisted enum surface; otherwise unchanged. +- `src/easydiffraction/analysis/analysis.py` — `fit()` signature gains + `resume: bool = False, extra_steps: int | None = None`; validation + + dispatch to engine. Wire `_engine_sync_skip_keys` (#103) before adding + `proposal_moves` to the ambient set. - `src/easydiffraction/io/results_sidecar.py` — read path: when `/emcee_chain` is present, expose a helper to construct an - `emcee.backends.HDFBackend(path, name='emcee_chain', - read_only=True)` for inspection/visualisation. -- `pyproject.toml`, `pixi.toml`, and `pixi.lock` — add `emcee>=3.1` - as a direct runtime dependency and refresh the lockfile via - `pixi lock` (CI installs from the lockfile, not from the manifest - files alone). + `emcee.backends.HDFBackend(path, name='emcee_chain', read_only=True)` + for inspection/visualisation. +- `pyproject.toml`, `pixi.toml`, and `pixi.lock` — add `emcee>=3.1` as a + direct runtime dependency and refresh the lockfile via `pixi lock` (CI + installs from the lockfile, not from the manifest files alone). ### Deleted @@ -278,51 +267,47 @@ When the matching open-issue is fully resolved, move it to Mark `[x]` as each step lands. - [ ] **P1.1 — Add emcee dependency and refresh the lockfile.** - - Add `emcee>=3.1` to `pyproject.toml` (runtime dependencies, not - just the `doc` extra — the existing lockfile carries emcee only - as `extra == 'doc'` which CI does not install for runtime). + - Add `emcee>=3.1` to `pyproject.toml` (runtime dependencies, not just + the `doc` extra — the existing lockfile carries emcee only as + `extra == 'doc'` which CI does not install for runtime). - Add the same dependency to `pixi.toml` (runtime feature). - - Run `pixi lock` to regenerate `pixi.lock` with `emcee` as a - direct runtime dependency. The refreshed `pixi.lock` is the - artifact CI consumes; `pixi install` is a local sanity check - only. + - Run `pixi lock` to regenerate `pixi.lock` with `emcee` as a direct + runtime dependency. The refreshed `pixi.lock` is the artifact CI + consumes; `pixi install` is a local sanity check only. - Stage `pyproject.toml`, `pixi.toml`, and `pixi.lock` together. Files modified by this step: `pyproject.toml`, `pixi.toml`, `pixi.lock`. Commit: `Add emcee runtime dependency` -- [ ] **P1.2 — Register `MinimizerTypeEnum.EMCEE`.** Add the enum - member with value `'emcee'` to - `src/easydiffraction/analysis/minimizers/enums.py`. No other - code wiring yet. Commit: - `Register emcee minimizer enum value` +- [ ] **P1.2 — Register `MinimizerTypeEnum.EMCEE`.** Add the enum member + with value `'emcee'` to + `src/easydiffraction/analysis/minimizers/enums.py`. No other code + wiring yet. Commit: `Register emcee minimizer enum value` - [ ] **P1.3 — Add `EmceeMinimizer` category class.** New file `src/easydiffraction/analysis/categories/minimizer/emcee.py`. `EmceeMinimizer(BayesianMinimizerBase)` declares: - `type_info` with `tag=MinimizerTypeEnum.EMCEE` and a description. - - `_engine_metadata: ClassVar[dict[str, str]] = {'optimizer_name': - 'emcee', 'method_name': 'stretch'}` (matching the - `BumpsDreamMinimizer` precedent for the + - `_engine_metadata: ClassVar[dict[str, str]] = {'optimizer_name': 'emcee', 'method_name': 'stretch'}` + (matching the `BumpsDreamMinimizer` precedent for the `_restore_fit_results_from_projection` lookup). - `_native_key_map` override mapping the verbose names to emcee's native kwargs (see §"Decisions already made" point 3). - Class-level defaults for emcee-specific values: - `sampling_steps=5000`, `burn_in_steps=1000`, - `thinning_interval=5`, `population_size=32`, - `parallel_workers=0`, `proposal_moves='stretch'`. + `sampling_steps=5000`, `burn_in_steps=1000`, `thinning_interval=5`, + `population_size=32`, `parallel_workers=0`, + `proposal_moves='stretch'`. - `__init__` constructs descriptors via the inherited helpers (`_sampling_steps_descriptor(default)`, etc. from - `BayesianMinimizerBase`) and adds a new `proposal_moves` - descriptor with a `MembershipValidator` over the single-move - set (`stretch`, `de`, `de_snooker`, `walk`). + `BayesianMinimizerBase`) and adds a new `proposal_moves` descriptor + with a `MembershipValidator` over the single-move set (`stretch`, + `de`, `de_snooker`, `walk`). - **Decide the `InitializationMethodEnum` reconciliation** (open - question above) before wiring. `EmceeMinimizer._supported_initialization_methods` - should list `(BALL, UNIFORM, PRIOR)` regardless of the DREAM - decision. + question above) before wiring. + `EmceeMinimizer._supported_initialization_methods` should list + `(BALL, UNIFORM, PRIOR)` regardless of the DREAM decision. - Update - `src/easydiffraction/analysis/categories/minimizer/__init__.py` + Update `src/easydiffraction/analysis/categories/minimizer/__init__.py` to import `EmceeMinimizer` (registration trigger via `@MinimizerCategoryFactory.register`). @@ -335,21 +320,20 @@ Mark `[x]` as each step lands. - [ ] **P1.4 — Add `EmceeMinimizer` engine class.** New file `src/easydiffraction/analysis/minimizers/emcee.py`. The engine class is registered with `MinimizerFactory` and holds the - `emcee.EnsembleSampler` plus an `HDFBackend` attribute. Mirror - the shape of + `emcee.EnsembleSampler` plus an `HDFBackend` attribute. Mirror the + shape of [`bumps_dream.py BumpsDreamMinimizer`](../../../src/easydiffraction/analysis/minimizers/bumps_dream.py) - — descriptor attributes (`burn`, `thin`, `pop`, `init`, …) - that `Analysis._sync_engine_from_minimizer_category` writes to - from the category's `_native_kwargs()`. The descriptor names - on the engine must match the keys returned by - `EmceeMinimizer._native_kwargs()`, which are: `nsteps`, - `nburn`, `thin`, `nwalkers`, `parallel_workers`, - `random_seed`, plus `initialization_method` and - `proposal_moves` handled by custom hooks (see §3 for the - mapping table and the `parallel_workers` semantics — the - engine attribute is an integer; the actual emcee `pool` - object is built and torn down inside `EmceeMinimizer.fit`, - not by the native-key sync). + — descriptor attributes (`burn`, `thin`, `pop`, `init`, …) that + `Analysis._sync_engine_from_minimizer_category` writes to from the + category's `_native_kwargs()`. The descriptor names on the engine + must match the keys returned by `EmceeMinimizer._native_kwargs()`, + which are: `nsteps`, `nburn`, `thin`, `nwalkers`, + `parallel_workers`, `random_seed`, plus `initialization_method` + and `proposal_moves` handled by custom hooks (see §3 for the + mapping table and the `parallel_workers` semantics — the engine + attribute is an integer; the actual emcee `pool` object is built + and torn down inside `EmceeMinimizer.fit`, not by the native-key + sync). **Engine-facing contract.** `EmceeMinimizer.fit` matches the existing @@ -507,152 +491,143 @@ Mark `[x]` as each step lands. Commit: `Add EmceeMinimizer engine class` -- [ ] **P1.5 — Wire `fit(resume=True, extra_steps=N)` end-to-end.** - The current fit stack does not accept `resume` / `extra_steps` +- [ ] **P1.5 — Wire `fit(resume=True, extra_steps=N)` end-to-end.** The + current fit stack does not accept `resume` / `extra_steps` anywhere. Every signature and call site listed below must be - updated in this step. Each item is one short edit; the step - lands as a single commit because the signatures must change in - lockstep. + updated in this step. Each item is one short edit; the step lands + as a single commit because the signatures must change in lockstep. **`Fitter` stays the layer that owns `structures`, `experiments`, - `weights`, parameter collection, and objective construction.** - Engines receive only `parameters` and `objective_function` - (already-built) via the existing `MinimizerBase.fit` shape. This - plan adds `resume` and `extra_steps` to the same shape. - - **Signatures (add `resume: bool = False, extra_steps: int | None = None`):** + `weights`, parameter collection, and objective construction.** Engines + receive only `parameters` and `objective_function` (already-built) via + the existing `MinimizerBase.fit` shape. This plan adds `resume` and + `extra_steps` to the same shape. + **Signatures (add + `resume: bool = False, extra_steps: int | None = None`):** - `Analysis.fit` ([analysis.py:929](../../../src/easydiffraction/analysis/analysis.py)). User-facing entry point. - `Analysis._run_single`, `Analysis._run_joint`, `Analysis._prepare_fit_run`, `Analysis._fit_single`, - `Analysis._fit_joint` (every internal helper that takes the - fit through to `Fitter.fit`). + `Analysis._fit_joint` (every internal helper that takes the fit + through to `Fitter.fit`). - `Fitter.fit` ([fitting.py:140-150](../../../src/easydiffraction/analysis/fitting.py)) - — adds the keyword pair to its existing signature - (`structures`, `experiments`, `weights`, `analysis`, - `verbosity`, `use_physical_limits`, `random_seed`, **+ - `resume`, `extra_steps`**). Forwards the pair to - `self.minimizer.fit(...)`. + — adds the keyword pair to its existing signature (`structures`, + `experiments`, `weights`, `analysis`, `verbosity`, + `use_physical_limits`, `random_seed`, **+ `resume`, + `extra_steps`**). Forwards the pair to `self.minimizer.fit(...)`. - `MinimizerBase.fit` ([base.py:351-360](../../../src/easydiffraction/analysis/minimizers/base.py)) — adds the keyword pair to its existing engine-facing shape - (`parameters`, `objective_function`, `verbosity`, *keyword*: - `finalize_tracking`, `use_physical_limits`, `random_seed`, - **+ `resume`, `extra_steps`**). The base implementation - handles non-resume calls unchanged and raises - `NotImplementedError(f"Minimizer '{self.name}' does not - support resume.")` when `resume=True`. `EmceeMinimizer.fit` - overrides with the same signature and honours both args. - - **Sidecar-path plumbing.** `Fitter.fit` resolves the sidecar - path from `analysis.project.info.path` (when both are non-None) - and sets `self.minimizer._sidecar_path` on the engine before - calling `self.minimizer.fit(...)`. Engines that do not need it - ignore the attribute; `EmceeMinimizer.fit` reads it (and raises - `RuntimeError` if `None` as defence in depth — but normal users - never hit that path because of the upfront save-required guard - in the next bullet). + (`parameters`, `objective_function`, `verbosity`, _keyword_: + `finalize_tracking`, `use_physical_limits`, `random_seed`, **+ + `resume`, `extra_steps`**). The base implementation handles + non-resume calls unchanged and raises + `NotImplementedError(f"Minimizer '{self.name}' does not support resume.")` + when `resume=True`. `EmceeMinimizer.fit` overrides with the same + signature and honours both args. + + **Sidecar-path plumbing.** `Fitter.fit` resolves the sidecar path from + `analysis.project.info.path` (when both are non-None) and sets + `self.minimizer._sidecar_path` on the engine before calling + `self.minimizer.fit(...)`. Engines that do not need it ignore the + attribute; `EmceeMinimizer.fit` reads it (and raises `RuntimeError` if + `None` as defence in depth — but normal users never hit that path + because of the upfront save-required guard in the next bullet). **Behaviour rules:** - - **Single mode only for v1.** `Analysis._run_joint` raises - `ValueError('Resume is supported in single fit mode only')` - when `resume=True`. Joint-mode resume is deferred; recorded - explicitly in §"Open questions". + `ValueError('Resume is supported in single fit mode only')` when + `resume=True`. Joint-mode resume is deferred; recorded explicitly in + §"Open questions". - **Validate the active minimizer.** `Analysis.fit` raises `ValueError` when `resume=True` and - `self.minimizer.type != MinimizerTypeEnum.EMCEE.value`. Match - the clear-error pattern used elsewhere. - - **Require a saved project for emcee.** Unlike DREAM (which - keeps the chain in memory), emcee's `HDFBackend` is the - sampler's live chain store — it needs a real file path. - `Analysis.fit` raises `ValueError` when + `self.minimizer.type != MinimizerTypeEnum.EMCEE.value`. Match the + clear-error pattern used elsewhere. + - **Require a saved project for emcee.** Unlike DREAM (which keeps the + chain in memory), emcee's `HDFBackend` is the sampler's live chain + store — it needs a real file path. `Analysis.fit` raises + `ValueError` when `self.minimizer.type == MinimizerTypeEnum.EMCEE.value` and `self.project.info.path is None`, with a clear scientist-facing message that points at the fix: - `"emcee requires a saved project; call project.save_as() - before analysis.fit()."` The check fires for both the initial - run and `resume=True`. The engine's `_sidecar_path is None` - `RuntimeError` (per P1.4) remains as defence-in-depth for - direct engine calls outside the `Analysis` flow but is not - reachable from the user-facing path. + `"emcee requires a saved project; call project.save_as() before analysis.fit()."` + The check fires for both the initial run and `resume=True`. The + engine's `_sidecar_path is None` `RuntimeError` (per P1.4) remains + as defence-in-depth for direct engine calls outside the `Analysis` + flow but is not reachable from the user-facing path. - **Validate `extra_steps`.** Require positive integer when `resume=True`. Raise on `None`, `0`, or negative. - **Bypass reset.** Skip the new - `prepare_analysis_results_sidecar_for_new_fit` helper (see - P1.5a) and `_clear_persisted_fit_state` when `resume=True` — - both would clobber the chain and the persisted fit state. - - **Defence in depth.** `MinimizerBase.fit`'s - `NotImplementedError` on `resume=True` only matters if an - engine is called directly outside the `Analysis` flow — the - `Analysis.fit` guard above fires first in normal use. + `prepare_analysis_results_sidecar_for_new_fit` helper (see P1.5a) + and `_clear_persisted_fit_state` when `resume=True` — both would + clobber the chain and the persisted fit state. + - **Defence in depth.** `MinimizerBase.fit`'s `NotImplementedError` on + `resume=True` only matters if an engine is called directly outside + the `Analysis` flow — the `Analysis.fit` guard above fires first in + normal use. **Open issue #103 cleanup.** Introduce `_engine_sync_skip_keys: ClassVar[frozenset[str]] = frozenset({'random_seed', 'parallel_workers'})` on `MinimizerCategoryBase`, and update `_sync_engine_from_minimizer_category` ([analysis.py:1134-1146](../../../src/easydiffraction/analysis/analysis.py)) - to use it. `EmceeMinimizer` (category) overrides the frozenset to - add `'proposal_moves'` if the engine consumes that key - differently from the category attribute (sketch in P1.4). + to use it. `EmceeMinimizer` (category) overrides the frozenset to add + `'proposal_moves'` if the engine consumes that key differently from + the category attribute (sketch in P1.4). Commit: `Wire emcee resume through fit stack` -- [ ] **P1.5a — Make `results.h5` append-on-save and add an - explicit truncate-on-new-fit prep step.** Two coordinated - changes that land in a single commit because they jointly - preserve the ADR lifecycle (resume keeps `/emcee_chain`; - new fit removes it). +- [ ] **P1.5a — Make `results.h5` append-on-save and add an explicit + truncate-on-new-fit prep step.** Two coordinated changes that land + in a single commit because they jointly preserve the ADR lifecycle + (resume keeps `/emcee_chain`; new fit removes it). In [`src/easydiffraction/io/results_sidecar.py`](../../../src/easydiffraction/io/results_sidecar.py): - - Replace `h5py.File(sidecar_path, 'w')` (line 282) with `h5py.File(sidecar_path, 'a')`. Before writing each EasyDiffraction-canonical group (`/posterior`, - `/distribution_cache`, `/pair_cache`, `/predictive`), delete - that group first if present so the writer's behaviour for those - groups is unchanged. + `/distribution_cache`, `/pair_cache`, `/predictive`), delete that + group first if present so the writer's behaviour for those groups is + unchanged. - Do **not** touch any other top-level group from the writer. `/emcee_chain` survives every save. - - **Add a new helper** `prepare_analysis_results_sidecar_for_new_fit(*, analysis_dir: Path) -> None` + - **Add a new helper** + `prepare_analysis_results_sidecar_for_new_fit(*, analysis_dir: Path) -> None` that takes the same `analysis_dir` shape as - `warn_analysis_results_sidecar_overwrite`, **warns** when the - file exists (matching the current warning text), and then - **removes** the file entirely so a fresh fit starts from a - clean slate. The old `warn_analysis_results_sidecar_overwrite` - becomes a thin wrapper that delegates to the new helper, or is - replaced outright by the new helper at every call site. + `warn_analysis_results_sidecar_overwrite`, **warns** when the file + exists (matching the current warning text), and then **removes** the + file entirely so a fresh fit starts from a clean slate. The old + `warn_analysis_results_sidecar_overwrite` becomes a thin wrapper + that delegates to the new helper, or is replaced outright by the new + helper at every call site. In [`src/easydiffraction/analysis/analysis.py`](../../../src/easydiffraction/analysis/analysis.py): - - - Replace `_warn_results_sidecar_overwrite` (lines 943-953) with - a call to the new `prepare_analysis_results_sidecar_for_new_fit` + - Replace `_warn_results_sidecar_overwrite` (lines 943-953) with a + call to the new `prepare_analysis_results_sidecar_for_new_fit` helper. Same call sites in `_run_single` and `_run_joint`. - **Bypass on resume.** The `resume=True` branch in - `Analysis._prepare_fit_run` (added by P1.5) must **not** call - this helper — that is the whole point of resume keeping the - chain alive. The bypass rule listed in P1.5 "Behaviour rules - → Bypass reset" therefore now also covers the + `Analysis._prepare_fit_run` (added by P1.5) must **not** call this + helper — that is the whole point of resume keeping the chain alive. + The bypass rule listed in P1.5 "Behaviour rules → Bypass reset" + therefore now also covers the `prepare_analysis_results_sidecar_for_new_fit` call. Add focused unit tests in Phase 2 (P2.1) covering: - - - **Append preserves `/emcee_chain`.** Write a sidecar payload, - create a stub `/emcee_chain` group on the same file, re-write - the sidecar — the `/emcee_chain` group must survive. - - **New-fit prep removes the file.** Create a sidecar with a - stub `/emcee_chain` group; call - `prepare_analysis_results_sidecar_for_new_fit`; assert the - file is gone (or empty) and the stale group is unreachable. - - **Resume bypass.** Set up an `Analysis` with a saved project, - invoke the resume code path (mocked engine), and assert - `prepare_analysis_results_sidecar_for_new_fit` was **not** - called. + - **Append preserves `/emcee_chain`.** Write a sidecar payload, create + a stub `/emcee_chain` group on the same file, re-write the sidecar — + the `/emcee_chain` group must survive. + - **New-fit prep removes the file.** Create a sidecar with a stub + `/emcee_chain` group; call + `prepare_analysis_results_sidecar_for_new_fit`; assert the file is + gone (or empty) and the stale group is unreachable. + - **Resume bypass.** Set up an `Analysis` with a saved project, invoke + the resume code path (mocked engine), and assert + `prepare_analysis_results_sidecar_for_new_fit` was **not** called. Commit: `Append-on-save plus explicit truncate-on-new-fit prep` @@ -660,16 +635,14 @@ Mark `[x]` as each step lands. sidecar pipeline.** Verify the existing `_store_posterior_fit_projection` ([analysis.py](../../../src/easydiffraction/analysis/analysis.py)) - writes to `self.fit_result._set_*` (the - `BayesianFitResult` instance auto-paired with the - `EmceeMinimizer`) and that the `/posterior`, - `/distribution_cache`, `/pair_cache`, `/predictive` groups in - `results.h5` receive emcee output without modification. - Adjust only where emcee surfaces data differently from - DREAM (e.g. `EnsembleSampler.get_chain(flat=False, - discard=burn, thin=thin)` vs the DREAM extraction helper). - Cache derivations (KDE, pair grids) reuse the existing - pipeline. + writes to `self.fit_result._set_*` (the `BayesianFitResult` + instance auto-paired with the `EmceeMinimizer`) and that the + `/posterior`, `/distribution_cache`, `/pair_cache`, `/predictive` + groups in `results.h5` receive emcee output without modification. + Adjust only where emcee surfaces data differently from DREAM (e.g. + `EnsembleSampler.get_chain(flat=False, discard=burn, thin=thin)` + vs the DREAM extraction helper). Cache derivations (KDE, pair + grids) reuse the existing pipeline. Opportunistic cleanup: address open issue #100 if the predictive plotting path is being touched anyway. Collapse @@ -680,11 +653,11 @@ Mark `[x]` as each step lands. - [ ] **P1.7 — Add `ed-25.py` tutorial.** Verify first that `docs/docs/tutorials/ed-25.py` is unused. `ed-23.py` is the - "Co2SiO4 Sequential Fit" tutorial and `ed-24.py` is the - "LBCO Bayesian Display" tutorial — do **not** overwrite - either. If `ed-25.py` already exists by the time this step - runs, pick the next free integer slot and adjust the file - name + references below to match. + "Co2SiO4 Sequential Fit" tutorial and `ed-24.py` is the "LBCO + Bayesian Display" tutorial — do **not** overwrite either. If + `ed-25.py` already exists by the time this step runs, pick the + next free integer slot and adjust the file name + references below + to match. New notebook source at `docs/docs/tutorials/ed-25.py` covering: @@ -695,14 +668,12 @@ Mark `[x]` as each step lands. tutorial speed). - `project.analysis.fit()` and a posterior plot. - `project.save()`. - - `project.analysis.fit(resume=True, extra_steps=500)` continues - the chain. + - `project.analysis.fit(resume=True, extra_steps=500)` continues the + chain. - Final posterior plot after resume. Update the docs navigation in the same step: - - - Add an entry under "MCMC / Bayesian" (or the appropriate - section) in + - Add an entry under "MCMC / Bayesian" (or the appropriate section) in [`docs/docs/tutorials/index.md`](../../docs/tutorials/index.md) pointing at `ed-25.ipynb`. - Add a navigation entry under the matching section in @@ -723,13 +694,13 @@ Mark `[x]` as each step lands. Commit: `Add ed-25 emcee tutorial` -- [ ] **P1.8 — Phase 1 review gate.** No code change. Stop and - request user review. After approval, proceed to Phase 2. +- [ ] **P1.8 — Phase 1 review gate.** No code change. Stop and request + user review. After approval, proceed to Phase 2. ## Verification (Phase 2) Each command captures its log with a zsh-safe exit-code variable as -required by `.github/copilot-instructions.md` → **Workflow**. +required by `AGENTS.md` → **Workflow**. - [ ] **P2.1 — Add unit + integration tests.** - `tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py`: @@ -739,19 +710,18 @@ required by `.github/copilot-instructions.md` → **Workflow**. - `tests/unit/easydiffraction/analysis/minimizers/test_emcee.py`: engine-class registration, descriptor defaults, native kwargs plumbing. - - `tests/unit/easydiffraction/analysis/test_analysis.py` (or - matching coverage file): assert `Analysis.fit()` raises - `ValueError` with a save-prompt message when emcee is the - active minimizer and `project.info.path is None`, for both - initial fits and `resume=True`. + - `tests/unit/easydiffraction/analysis/test_analysis.py` (or matching + coverage file): assert `Analysis.fit()` raises `ValueError` with a + save-prompt message when emcee is the active minimizer and + `project.info.path is None`, for both initial fits and + `resume=True`. - `tests/unit/easydiffraction/io/test_results_sidecar.py`: the - save-after-resume invariant from P1.5a — write a sidecar - payload, then write a stub `/emcee_chain` group on the same - file, then re-write the sidecar; the `/emcee_chain` group - must survive. - - `tests/integration/fitting/test_emcee.py`: end-to-end fit on a - small synthetic problem; resume; assert posterior medians - agree with a DREAM run within tolerance. + save-after-resume invariant from P1.5a — write a sidecar payload, + then write a stub `/emcee_chain` group on the same file, then + re-write the sidecar; the `/emcee_chain` group must survive. + - `tests/integration/fitting/test_emcee.py`: end-to-end fit on a small + synthetic problem; resume; assert posterior medians agree with a + DREAM run within tolerance. Layout check: @@ -813,9 +783,9 @@ required by `.github/copilot-instructions.md` → **Workflow**. **Description (user-facing):** -EasyDiffraction adds emcee — a widely-used affine-invariant MCMC -sampler — as a second Bayesian fitter. It is selected exactly like -the existing samplers, via the uniform switchable-category surface: +EasyDiffraction adds emcee — a widely-used affine-invariant MCMC sampler +— as a second Bayesian fitter. It is selected exactly like the existing +samplers, via the uniform switchable-category surface: ```python project.analysis.minimizer.type = 'emcee' @@ -829,11 +799,11 @@ Long runs can be **resumed** without starting over: project.analysis.fit(resume=True, extra_steps=2000) ``` -emcee's chain state lives inside the same `analysis/results.h5` file -as the other posterior data, so saving and reopening a project is a +emcee's chain state lives inside the same `analysis/results.h5` file as +the other posterior data, so saving and reopening a project is a single-file affair. Plots, parameter posteriors, and tables work the -same as for DREAM, so switching between samplers to cross-check -results is straightforward. +same as for DREAM, so switching between samplers to cross-check results +is straightforward. -A new tutorial (`ed-25`) walks through a short run, saving the -project, and resuming for additional steps. +A new tutorial (`ed-25`) walks through a short run, saving the project, +and resuming for additional steps. diff --git a/docs/dev/plans/minimizer-input-output-split.md b/docs/dev/plans/minimizer-input-output-split.md index 0d9361a8a..e93345ad7 100644 --- a/docs/dev/plans/minimizer-input-output-split.md +++ b/docs/dev/plans/minimizer-input-output-split.md @@ -1,8 +1,7 @@ # Plan: Minimizer Input/Output Split -> This plan follows -> [`.github/copilot-instructions.md`](../../../.github/copilot-instructions.md). -> No deliberate exceptions. +> This plan follows [`AGENTS.md`](../../../AGENTS.md). No deliberate +> exceptions. ## ADR @@ -33,7 +32,7 @@ Affected ADRs that this plan amends (per the ADR's §"ADRs amended"): ADR was drafted on). Do not push unless asked. - Each step in §"Implementation steps (Phase 1)" must be staged with explicit paths and committed locally **before** moving to the next - step. See `.github/copilot-instructions.md` → **Commits**. + step. See `AGENTS.md` → **Commits**. - After P1.17, stop and wait for the user review gate before starting Phase 2. @@ -530,7 +529,7 @@ Mark `[x]` as each step lands. ## Verification (Phase 2) Each command captures its log with a zsh-safe exit-code variable as -required by `.github/copilot-instructions.md` → **Workflow**. +required by `AGENTS.md` → **Workflow**. - [x] **P2.1 — Migrate existing tests off the removed minimizer output fields.** `git grep` `tests/` for the same patterns as P1.15. From 4ff0098eef47602ad7c639a4384998324bf10a4a Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sun, 24 May 2026 21:56:14 +0200 Subject: [PATCH 08/65] Add emcee runtime dependency --- docs/dev/plans/emcee-minimizer.md | 2 +- pixi.lock | 484 +++++++++++++++--------------- pixi.toml | 3 + pyproject.toml | 1 + 4 files changed, 242 insertions(+), 248 deletions(-) diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index 0930cd93e..29c5c5472 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -266,7 +266,7 @@ When the matching open-issue is fully resolved, move it to Mark `[x]` as each step lands. -- [ ] **P1.1 — Add emcee dependency and refresh the lockfile.** +- [x] **P1.1 — Add emcee dependency and refresh the lockfile.** - Add `emcee>=3.1` to `pyproject.toml` (runtime dependencies, not just the `doc` extra — the existing lockfile carries emcee only as `extra == 'doc'` which CI does not install for runtime). diff --git a/pixi.lock b/pixi.lock index e17dcaa8c..a4c4ec86f 100644 --- a/pixi.lock +++ b/pixi.lock @@ -155,7 +155,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/setuptools-82.0.1-pyh332efcf_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/six-1.17.0-pyhe01879c_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/sniffio-1.3.1-pyhd8ed1ab_2.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.3-pyhd8ed1ab_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.4-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/stack_data-0.6.3-pyhd8ed1ab_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/terminado-0.18.1-pyhc90fa1f_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/tinycss2-1.4.0-pyhd8ed1ab_0.conda @@ -201,7 +201,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/2b/76/7f1bfd6afff4c5e38e36a3c6d68eb5f4b7311ea80baf693db78d95b603c4/propcache-0.5.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/2c/01/bc663630c510822c95c47a66af9fa7a443c295b47d5f041e5e6ae62ef659/mkdocs_material-9.7.6-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/2e/84/efc7c0bf3a1c5eef81d397f6fddac855becdbb11cb38ff957888603014a7/sqlalchemy-2.0.49-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/2f/40/dc34d1a8d5f1e51fc64640b62b191684da52ca469da9cd74e84936ffa4a6/msgpack-1.1.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/2f/c8/005d1de3af80f54411703d1263a0b9d31276411ec9f273d9432c59b17299/arviz_plots-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl @@ -223,6 +222,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/46/b4/0887c88ddfaba1d7140ea335144eb904af97550786ee58bdb295ff10d255/crysfml-0.6.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl - pypi: https://files.pythonhosted.org/packages/52/59/0782e51887ac6b07ffd1570e0364cf901ebc36345fea669969d2084baebb/simple_websocket-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/57/da/6fbf010c8ebb347679d0d100b22fe9ba5e13fd04046c5df7280d2f0bf706/sqlalchemy-2.0.50.tar.gz - pypi: https://files.pythonhosted.org/packages/58/3b/1cdec6772bdbaf7b25dab360c59f03cadf05492dd724c6540af905389b07/pandas-3.0.3-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/58/e0/f1871f520c359e4e3a2eb7437c9e7e792bb6c356414e8617937561167caf/pycifrw-5.0.1.tar.gz - pypi: https://files.pythonhosted.org/packages/5b/29/74eeb4d3f3ae61ca096b018ad486b3b3c74b17bec09ab4edab721cbefec3/typeguard-4.5.2-py3-none-any.whl @@ -318,6 +318,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/f4/a4/61adb19f3c74b0dc0e411de4f06ebef564b1f179928f9dffcbd4b378f2ef/jupyter_notebook_parser-0.1.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f6/f0/10642828a8dfb741e5f3fbaac830550a518a775c7fff6f04a007259b0548/py-1.11.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f7/ec/67fbef5d497f86283db54c22eec6f6140243aae73265799baaaa19cd17fb/ghp_import-2.1.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/f9/ef/2196b9bf88ffa1bde45853c72df021fbd07a8fa91a0f59a22d14a050dc04/emcee-3.1.6-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fa/bc/8b8ec5a4bfc5b9cf3ce27a118339e994f88410be5677c96493e0ea28e76d/dunamai-1.26.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fb/76/641ae371508676492379f16e2fa48f4e2c11741bd63c48be4b12a6b09cba/aiosignal-1.4.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fd/7b/122376b1fd3c62c1ed9dc80c931ace4844b3c55407b6fb2d199377c9736f/pydantic-2.13.4-py3-none-any.whl @@ -414,7 +415,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/setuptools-82.0.1-pyh332efcf_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/six-1.17.0-pyhe01879c_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/sniffio-1.3.1-pyhd8ed1ab_2.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.3-pyhd8ed1ab_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.4-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/stack_data-0.6.3-pyhd8ed1ab_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/terminado-0.18.1-pyhc90fa1f_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/tinycss2-1.4.0-pyhd8ed1ab_0.conda @@ -530,7 +531,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/51/ac/b9d68ebddfe1b02c77af5bf81120e12b036b4432dc6af7a303d90e2bc38b/chardet-7.4.3-cp314-cp314-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/52/59/0782e51887ac6b07ffd1570e0364cf901ebc36345fea669969d2084baebb/simple_websocket-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/55/33/bf28f618c0a9597d14e0b9ee7d1e0622faff738d44fe986ee287cdf1b8d0/sqlalchemy-2.0.49-cp314-cp314-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/56/9e/d13e40f83b8d0a94430e6778ce1d94a43b38cf2efe63278bdd2b4c65abbf/ruff-0.15.14-py3-none-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/58/e0/f1871f520c359e4e3a2eb7437c9e7e792bb6c356414e8617937561167caf/pycifrw-5.0.1.tar.gz - pypi: https://files.pythonhosted.org/packages/5b/29/74eeb4d3f3ae61ca096b018ad486b3b3c74b17bec09ab4edab721cbefec3/typeguard-4.5.2-py3-none-any.whl @@ -615,6 +615,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/dc/83/6d810a8a9ebc9c307989b418840c20e46907c74d707beb67ab566773e6fc/xarray-2026.4.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/df/32/10ac51b4be7cdecd7e93d069251c86dfbf70b7adbd7c67b48ccea6c49e1c/sqlalchemy-2.0.50-cp314-cp314-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/eb/be/b257e12f9710819fde40adc972578bee6b72c5992da1bc8369bef2597756/nbmake-1.5.5-py3-none-any.whl @@ -627,6 +628,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/f4/a4/61adb19f3c74b0dc0e411de4f06ebef564b1f179928f9dffcbd4b378f2ef/jupyter_notebook_parser-0.1.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f6/f0/10642828a8dfb741e5f3fbaac830550a518a775c7fff6f04a007259b0548/py-1.11.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f7/ec/67fbef5d497f86283db54c22eec6f6140243aae73265799baaaa19cd17fb/ghp_import-2.1.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/f9/ef/2196b9bf88ffa1bde45853c72df021fbd07a8fa91a0f59a22d14a050dc04/emcee-3.1.6-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fa/bc/8b8ec5a4bfc5b9cf3ce27a118339e994f88410be5677c96493e0ea28e76d/dunamai-1.26.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fb/76/641ae371508676492379f16e2fa48f4e2c11741bd63c48be4b12a6b09cba/aiosignal-1.4.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fd/7b/122376b1fd3c62c1ed9dc80c931ace4844b3c55407b6fb2d199377c9736f/pydantic-2.13.4-py3-none-any.whl @@ -721,7 +723,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/setuptools-82.0.1-pyh332efcf_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/six-1.17.0-pyhe01879c_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/sniffio-1.3.1-pyhd8ed1ab_2.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.3-pyhd8ed1ab_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.4-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/stack_data-0.6.3-pyhd8ed1ab_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/terminado-0.18.1-pyh6dadd2b_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/tinycss2-1.4.0-pyhd8ed1ab_0.conda @@ -851,6 +853,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/7b/67/b1944235474aac3f0b0e1b232ce49547f9f9461ca4b943df1b88da5d3f1d/bumps-1.0.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/7d/c2/57f54b03d0f22d4044b8afb9ca0e184f8b1afd57b4f735c2fa70883dc601/contourpy-1.3.3-cp314-cp314-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/7e/85/545a951eecc270fcd688288c600017e2050a1aacb56c711d208586d3e470/pymdown_extensions-10.21.3-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/7f/a0/fe92bb9817863bc13ba093bda931979a26cc2ca69f8e8f26d07add3d7c6f/sqlalchemy-2.0.50-cp314-cp314-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/80/6e/4b28b62ecb6aae56769c34a8ff1d661473ec1e9519e2d5f8b2c150086b26/pre_commit-4.6.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/81/47/dd9a212ef6e343a6857485ffe25bba537304f1913bdbed446a23f7f592e1/filelock-3.29.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/82/3b/64d4899d73f91ba49a8c18a8ff3f0ea8f1c1d75481760df8c68ef5235bf5/rich-15.0.0-py3-none-any.whl @@ -904,7 +907,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/ca/31/d4e37e9e550c2b92a9cbc2e4d0b7420a27224968580b5a447f420847c975/pytest_xdist-3.8.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/cd/8b/601e6566b957ca50e28725cb6c355c59c2c8609751efbecd980db44e0349/pillow-12.2.0-cp314-cp314-win_amd64.whl - - pypi: https://files.pythonhosted.org/packages/cf/4f/8297e4ed88e80baa1f5aa3c484a0ee29ef3c69c7582f206c916973b75057/sqlalchemy-2.0.49-cp314-cp314-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/d1/b7/4ea2c170f10ad760fff2a5250beb18897719dc8b52b53a24cddbb9dd3f19/ruff-0.15.14-py3-none-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/d2/f0/834e479e47e499b6478e807fb57b31cc2db696c4db30557bb6f5aea4a90b/mando-0.7.1-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/d4/24/a372aaf5c9b7208e7112038812994107bc65a84cd00e0354a88c2c77a617/pytest-9.0.3-py3-none-any.whl @@ -928,6 +930,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/f6/f0/10642828a8dfb741e5f3fbaac830550a518a775c7fff6f04a007259b0548/py-1.11.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f7/ec/67fbef5d497f86283db54c22eec6f6140243aae73265799baaaa19cd17fb/ghp_import-2.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f9/58/6e1b8f52fdc3184b47dc5037f5070d83a3d11042db1594b02d2a44d786c8/coverage-7.14.0-cp314-cp314-win_amd64.whl + - pypi: https://files.pythonhosted.org/packages/f9/ef/2196b9bf88ffa1bde45853c72df021fbd07a8fa91a0f59a22d14a050dc04/emcee-3.1.6-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fa/bc/8b8ec5a4bfc5b9cf3ce27a118339e994f88410be5677c96493e0ea28e76d/dunamai-1.26.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fa/d9/5582d57e2b2db9b85eb6663a22efdd78e08805f3f5389566e9fcad254d1b/yarl-1.24.2-cp314-cp314-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/fb/76/641ae371508676492379f16e2fa48f4e2c11741bd63c48be4b12a6b09cba/aiosignal-1.4.0-py3-none-any.whl @@ -1086,7 +1089,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/setuptools-82.0.1-pyh332efcf_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/six-1.17.0-pyhe01879c_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/sniffio-1.3.1-pyhd8ed1ab_2.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.3-pyhd8ed1ab_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.4-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/stack_data-0.6.3-pyhd8ed1ab_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/terminado-0.18.1-pyhc90fa1f_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/tinycss2-1.4.0-pyhd8ed1ab_0.conda @@ -1130,7 +1133,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/2a/2d/d4bf65e47cea8ff2c794a600c4fd1273a7902f268757c531e0ee9f18aa58/pooch-1.9.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/01/bc663630c510822c95c47a66af9fa7a443c295b47d5f041e5e6ae62ef659/mkdocs_material-9.7.6-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/2c/fa/65fcae2ed62f84ab72cf89536c7c3217a156e71a2c111b1305ab6f0690e2/sqlalchemy-2.0.49-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/2f/c8/005d1de3af80f54411703d1263a0b9d31276411ec9f273d9432c59b17299/arviz_plots-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/91/d024616abdba99e83120e07a20658976f6a343646710760c4a51df126029/matplotlib-3.10.9-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl @@ -1152,6 +1154,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/52/59/0782e51887ac6b07ffd1570e0364cf901ebc36345fea669969d2084baebb/simple_websocket-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/57/d8/8d44036d7eb7b6a8ec4c5494ea0c8c8b94fbc0ed3991c1a7adf230df03bf/aiohttp-3.13.5-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl + - pypi: https://files.pythonhosted.org/packages/57/da/6fbf010c8ebb347679d0d100b22fe9ba5e13fd04046c5df7280d2f0bf706/sqlalchemy-2.0.50.tar.gz - pypi: https://files.pythonhosted.org/packages/5b/29/74eeb4d3f3ae61ca096b018ad486b3b3c74b17bec09ab4edab721cbefec3/typeguard-4.5.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/5b/54/662a4743aa81d9582ee9339d4ffa3c8fd40a4965e033d77b9da9774d3960/mkdocs_material_extensions-1.3.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/5b/69/93b34728cc386efdde0c342f8c680b9187dea7beb7adaf6b58a0713be101/mpld3-0.5.12-py3-none-any.whl @@ -1250,6 +1253,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/f4/a4/61adb19f3c74b0dc0e411de4f06ebef564b1f179928f9dffcbd4b378f2ef/jupyter_notebook_parser-0.1.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f6/f0/10642828a8dfb741e5f3fbaac830550a518a775c7fff6f04a007259b0548/py-1.11.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f7/ec/67fbef5d497f86283db54c22eec6f6140243aae73265799baaaa19cd17fb/ghp_import-2.1.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/f9/ef/2196b9bf88ffa1bde45853c72df021fbd07a8fa91a0f59a22d14a050dc04/emcee-3.1.6-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fa/bc/8b8ec5a4bfc5b9cf3ce27a118339e994f88410be5677c96493e0ea28e76d/dunamai-1.26.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fb/76/641ae371508676492379f16e2fa48f4e2c11741bd63c48be4b12a6b09cba/aiosignal-1.4.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fd/7b/122376b1fd3c62c1ed9dc80c931ace4844b3c55407b6fb2d199377c9736f/pydantic-2.13.4-py3-none-any.whl @@ -1344,7 +1348,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/setuptools-82.0.1-pyh332efcf_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/six-1.17.0-pyhe01879c_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/sniffio-1.3.1-pyhd8ed1ab_2.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.3-pyhd8ed1ab_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.4-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/stack_data-0.6.3-pyhd8ed1ab_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/terminado-0.18.1-pyhc90fa1f_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/tinycss2-1.4.0-pyhd8ed1ab_0.conda @@ -1461,7 +1465,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/3f/d0/7b958df957e4827837b590944008f0b28078f552b451f7407b4b3d54f574/asciichartpy-1.5.25-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/3f/f9/2b3ff4e56e5fa7debfaf9eb135d0da96f3e9a1d5b27222223c7296336e5f/typer-0.25.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/49/b3/2de412451330756aaaa72d27131db6dde23995efe62c941184e15242a5fa/sqlalchemy-2.0.49-cp312-cp312-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/52/59/0782e51887ac6b07ffd1570e0364cf901ebc36345fea669969d2084baebb/simple_websocket-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/53/3e/405b59cfa13021a56bba395a6b3aca8cec012b45bf177b0eaf7a202cde2c/contourpy-1.3.3-cp312-cp312-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl @@ -1528,6 +1531,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/b7/6f/a05a317a66fee0aad270011461f1a63a453ed12471249f172f7d2e2bc7b4/python_discovery-1.3.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/b7/ee/e9c95cda829131f71a8dff5ce0406059fd16e591c074414e31ada19ba7c3/validate_pyproject-0.25-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/bd/24/12818598c362d7f300f18e74db45963dbcb85150324092410c8b49405e42/pyproject_hooks-1.2.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/be/b0/a9d19b43f38f878b1278bca5b00b909f7540d41494396dd2561f9ad0956d/sqlalchemy-2.0.50-cp312-cp312-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/bf/50/98b146aea0f1cd7531d25f12bea69fa9ce8d1662124f93fb30dc4511b65e/docstring_parser_fork-0.0.14-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/c1/d4/59e74daffcb57a07668852eeeb6035af9f32cbfd7a1d2511f17d2fe6a738/smmap-5.0.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/c6/3d/020a6b6248c3d4a37797db068256f0b3f15b01bc481327ba888c50309aa8/mkdocs_plugin_inline_svg-0.1.0-py3-none-any.whl @@ -1558,6 +1562,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/f4/a4/61adb19f3c74b0dc0e411de4f06ebef564b1f179928f9dffcbd4b378f2ef/jupyter_notebook_parser-0.1.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f6/f0/10642828a8dfb741e5f3fbaac830550a518a775c7fff6f04a007259b0548/py-1.11.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f7/ec/67fbef5d497f86283db54c22eec6f6140243aae73265799baaaa19cd17fb/ghp_import-2.1.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/f9/ef/2196b9bf88ffa1bde45853c72df021fbd07a8fa91a0f59a22d14a050dc04/emcee-3.1.6-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fa/bc/8b8ec5a4bfc5b9cf3ce27a118339e994f88410be5677c96493e0ea28e76d/dunamai-1.26.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fb/76/641ae371508676492379f16e2fa48f4e2c11741bd63c48be4b12a6b09cba/aiosignal-1.4.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fd/7b/122376b1fd3c62c1ed9dc80c931ace4844b3c55407b6fb2d199377c9736f/pydantic-2.13.4-py3-none-any.whl @@ -1650,7 +1655,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/setuptools-82.0.1-pyh332efcf_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/six-1.17.0-pyhe01879c_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/sniffio-1.3.1-pyhd8ed1ab_2.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.3-pyhd8ed1ab_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.4-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/stack_data-0.6.3-pyhd8ed1ab_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/terminado-0.18.1-pyh6dadd2b_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/tinycss2-1.4.0-pyhd8ed1ab_0.conda @@ -1760,7 +1765,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/3f/f9/2b3ff4e56e5fa7debfaf9eb135d0da96f3e9a1d5b27222223c7296336e5f/typer-0.25.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/40/8c/985c1d41ea1107c2534abd9870e4ed5c8e7669b5c308297835c001e7a1c4/pydantic_core-2.46.4-cp312-cp312-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/47/9e/fd90114059175cac64e4fafa9bf3ac20584384d66de40793ae2e2f26f3bb/sqlalchemy-2.0.49-cp312-cp312-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/52/59/0782e51887ac6b07ffd1570e0364cf901ebc36345fea669969d2084baebb/simple_websocket-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/5b/29/74eeb4d3f3ae61ca096b018ad486b3b3c74b17bec09ab4edab721cbefec3/typeguard-4.5.2-py3-none-any.whl @@ -1825,6 +1829,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/b3/52/bc858b1665d0dec3a2511f4e6f5c18ea85c0977563d624d597c95d6d0fd7/jupyterquiz-2.9.6.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/b3/81/4da04ced5a082363ecfa159c010d200ecbd959ae410c10c0264a38cac0f5/markdown_it_py-4.2.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/b4/5e/bf11645aebb9af7d8d35927c40d3855816a0855c799e8156eeca8d632c90/diffpy_structure-3.4.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/b6/2d/314a6690dda4b9cfc571eab1a63cf6fe6e1470aa3759ccda6aa016ee0f5a/sqlalchemy-2.0.50-cp312-cp312-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/b7/28/180bfc5c95e83d40cb2abce512684ccad44e4819ec899fc36cb404a19061/python_engineio-4.13.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/b7/6f/a05a317a66fee0aad270011461f1a63a453ed12471249f172f7d2e2bc7b4/python_discovery-1.3.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/b7/ee/e9c95cda829131f71a8dff5ce0406059fd16e591c074414e31ada19ba7c3/validate_pyproject-0.25-py3-none-any.whl @@ -1860,6 +1865,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/f4/a4/61adb19f3c74b0dc0e411de4f06ebef564b1f179928f9dffcbd4b378f2ef/jupyter_notebook_parser-0.1.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f6/f0/10642828a8dfb741e5f3fbaac830550a518a775c7fff6f04a007259b0548/py-1.11.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f7/ec/67fbef5d497f86283db54c22eec6f6140243aae73265799baaaa19cd17fb/ghp_import-2.1.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/f9/ef/2196b9bf88ffa1bde45853c72df021fbd07a8fa91a0f59a22d14a050dc04/emcee-3.1.6-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fa/bc/8b8ec5a4bfc5b9cf3ce27a118339e994f88410be5677c96493e0ea28e76d/dunamai-1.26.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fb/76/641ae371508676492379f16e2fa48f4e2c11741bd63c48be4b12a6b09cba/aiosignal-1.4.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fd/7b/122376b1fd3c62c1ed9dc80c931ace4844b3c55407b6fb2d199377c9736f/pydantic-2.13.4-py3-none-any.whl @@ -2014,7 +2020,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/setuptools-82.0.1-pyh332efcf_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/six-1.17.0-pyhe01879c_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/sniffio-1.3.1-pyhd8ed1ab_2.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.3-pyhd8ed1ab_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.4-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/stack_data-0.6.3-pyhd8ed1ab_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/terminado-0.18.1-pyhc90fa1f_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/tinycss2-1.4.0-pyhd8ed1ab_0.conda @@ -2060,7 +2066,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/2b/76/7f1bfd6afff4c5e38e36a3c6d68eb5f4b7311ea80baf693db78d95b603c4/propcache-0.5.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/2c/01/bc663630c510822c95c47a66af9fa7a443c295b47d5f041e5e6ae62ef659/mkdocs_material-9.7.6-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/2e/84/efc7c0bf3a1c5eef81d397f6fddac855becdbb11cb38ff957888603014a7/sqlalchemy-2.0.49-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/2f/40/dc34d1a8d5f1e51fc64640b62b191684da52ca469da9cd74e84936ffa4a6/msgpack-1.1.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/2f/c8/005d1de3af80f54411703d1263a0b9d31276411ec9f273d9432c59b17299/arviz_plots-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl @@ -2082,6 +2087,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/46/b4/0887c88ddfaba1d7140ea335144eb904af97550786ee58bdb295ff10d255/crysfml-0.6.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl - pypi: https://files.pythonhosted.org/packages/52/59/0782e51887ac6b07ffd1570e0364cf901ebc36345fea669969d2084baebb/simple_websocket-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/57/da/6fbf010c8ebb347679d0d100b22fe9ba5e13fd04046c5df7280d2f0bf706/sqlalchemy-2.0.50.tar.gz - pypi: https://files.pythonhosted.org/packages/58/3b/1cdec6772bdbaf7b25dab360c59f03cadf05492dd724c6540af905389b07/pandas-3.0.3-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/58/e0/f1871f520c359e4e3a2eb7437c9e7e792bb6c356414e8617937561167caf/pycifrw-5.0.1.tar.gz - pypi: https://files.pythonhosted.org/packages/5b/29/74eeb4d3f3ae61ca096b018ad486b3b3c74b17bec09ab4edab721cbefec3/typeguard-4.5.2-py3-none-any.whl @@ -2177,6 +2183,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/f4/a4/61adb19f3c74b0dc0e411de4f06ebef564b1f179928f9dffcbd4b378f2ef/jupyter_notebook_parser-0.1.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f6/f0/10642828a8dfb741e5f3fbaac830550a518a775c7fff6f04a007259b0548/py-1.11.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f7/ec/67fbef5d497f86283db54c22eec6f6140243aae73265799baaaa19cd17fb/ghp_import-2.1.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/f9/ef/2196b9bf88ffa1bde45853c72df021fbd07a8fa91a0f59a22d14a050dc04/emcee-3.1.6-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fa/bc/8b8ec5a4bfc5b9cf3ce27a118339e994f88410be5677c96493e0ea28e76d/dunamai-1.26.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fb/76/641ae371508676492379f16e2fa48f4e2c11741bd63c48be4b12a6b09cba/aiosignal-1.4.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fd/7b/122376b1fd3c62c1ed9dc80c931ace4844b3c55407b6fb2d199377c9736f/pydantic-2.13.4-py3-none-any.whl @@ -2273,7 +2280,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/setuptools-82.0.1-pyh332efcf_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/six-1.17.0-pyhe01879c_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/sniffio-1.3.1-pyhd8ed1ab_2.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.3-pyhd8ed1ab_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.4-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/stack_data-0.6.3-pyhd8ed1ab_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/terminado-0.18.1-pyhc90fa1f_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/tinycss2-1.4.0-pyhd8ed1ab_0.conda @@ -2389,7 +2396,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/51/ac/b9d68ebddfe1b02c77af5bf81120e12b036b4432dc6af7a303d90e2bc38b/chardet-7.4.3-cp314-cp314-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/52/59/0782e51887ac6b07ffd1570e0364cf901ebc36345fea669969d2084baebb/simple_websocket-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/55/33/bf28f618c0a9597d14e0b9ee7d1e0622faff738d44fe986ee287cdf1b8d0/sqlalchemy-2.0.49-cp314-cp314-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/56/9e/d13e40f83b8d0a94430e6778ce1d94a43b38cf2efe63278bdd2b4c65abbf/ruff-0.15.14-py3-none-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/58/e0/f1871f520c359e4e3a2eb7437c9e7e792bb6c356414e8617937561167caf/pycifrw-5.0.1.tar.gz - pypi: https://files.pythonhosted.org/packages/5b/29/74eeb4d3f3ae61ca096b018ad486b3b3c74b17bec09ab4edab721cbefec3/typeguard-4.5.2-py3-none-any.whl @@ -2474,6 +2480,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/dc/83/6d810a8a9ebc9c307989b418840c20e46907c74d707beb67ab566773e6fc/xarray-2026.4.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/df/32/10ac51b4be7cdecd7e93d069251c86dfbf70b7adbd7c67b48ccea6c49e1c/sqlalchemy-2.0.50-cp314-cp314-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/eb/be/b257e12f9710819fde40adc972578bee6b72c5992da1bc8369bef2597756/nbmake-1.5.5-py3-none-any.whl @@ -2486,6 +2493,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/f4/a4/61adb19f3c74b0dc0e411de4f06ebef564b1f179928f9dffcbd4b378f2ef/jupyter_notebook_parser-0.1.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f6/f0/10642828a8dfb741e5f3fbaac830550a518a775c7fff6f04a007259b0548/py-1.11.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f7/ec/67fbef5d497f86283db54c22eec6f6140243aae73265799baaaa19cd17fb/ghp_import-2.1.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/f9/ef/2196b9bf88ffa1bde45853c72df021fbd07a8fa91a0f59a22d14a050dc04/emcee-3.1.6-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fa/bc/8b8ec5a4bfc5b9cf3ce27a118339e994f88410be5677c96493e0ea28e76d/dunamai-1.26.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fb/76/641ae371508676492379f16e2fa48f4e2c11741bd63c48be4b12a6b09cba/aiosignal-1.4.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fd/7b/122376b1fd3c62c1ed9dc80c931ace4844b3c55407b6fb2d199377c9736f/pydantic-2.13.4-py3-none-any.whl @@ -2580,7 +2588,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/setuptools-82.0.1-pyh332efcf_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/six-1.17.0-pyhe01879c_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/sniffio-1.3.1-pyhd8ed1ab_2.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.3-pyhd8ed1ab_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.4-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/stack_data-0.6.3-pyhd8ed1ab_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/terminado-0.18.1-pyh6dadd2b_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/tinycss2-1.4.0-pyhd8ed1ab_0.conda @@ -2710,6 +2718,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/7b/67/b1944235474aac3f0b0e1b232ce49547f9f9461ca4b943df1b88da5d3f1d/bumps-1.0.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/7d/c2/57f54b03d0f22d4044b8afb9ca0e184f8b1afd57b4f735c2fa70883dc601/contourpy-1.3.3-cp314-cp314-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/7e/85/545a951eecc270fcd688288c600017e2050a1aacb56c711d208586d3e470/pymdown_extensions-10.21.3-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/7f/a0/fe92bb9817863bc13ba093bda931979a26cc2ca69f8e8f26d07add3d7c6f/sqlalchemy-2.0.50-cp314-cp314-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/80/6e/4b28b62ecb6aae56769c34a8ff1d661473ec1e9519e2d5f8b2c150086b26/pre_commit-4.6.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/81/47/dd9a212ef6e343a6857485ffe25bba537304f1913bdbed446a23f7f592e1/filelock-3.29.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/82/3b/64d4899d73f91ba49a8c18a8ff3f0ea8f1c1d75481760df8c68ef5235bf5/rich-15.0.0-py3-none-any.whl @@ -2763,7 +2772,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/ca/31/d4e37e9e550c2b92a9cbc2e4d0b7420a27224968580b5a447f420847c975/pytest_xdist-3.8.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/cd/8b/601e6566b957ca50e28725cb6c355c59c2c8609751efbecd980db44e0349/pillow-12.2.0-cp314-cp314-win_amd64.whl - - pypi: https://files.pythonhosted.org/packages/cf/4f/8297e4ed88e80baa1f5aa3c484a0ee29ef3c69c7582f206c916973b75057/sqlalchemy-2.0.49-cp314-cp314-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/d1/b7/4ea2c170f10ad760fff2a5250beb18897719dc8b52b53a24cddbb9dd3f19/ruff-0.15.14-py3-none-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/d2/f0/834e479e47e499b6478e807fb57b31cc2db696c4db30557bb6f5aea4a90b/mando-0.7.1-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/d4/24/a372aaf5c9b7208e7112038812994107bc65a84cd00e0354a88c2c77a617/pytest-9.0.3-py3-none-any.whl @@ -2787,6 +2795,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/f6/f0/10642828a8dfb741e5f3fbaac830550a518a775c7fff6f04a007259b0548/py-1.11.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f7/ec/67fbef5d497f86283db54c22eec6f6140243aae73265799baaaa19cd17fb/ghp_import-2.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f9/58/6e1b8f52fdc3184b47dc5037f5070d83a3d11042db1594b02d2a44d786c8/coverage-7.14.0-cp314-cp314-win_amd64.whl + - pypi: https://files.pythonhosted.org/packages/f9/ef/2196b9bf88ffa1bde45853c72df021fbd07a8fa91a0f59a22d14a050dc04/emcee-3.1.6-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fa/bc/8b8ec5a4bfc5b9cf3ce27a118339e994f88410be5677c96493e0ea28e76d/dunamai-1.26.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fa/d9/5582d57e2b2db9b85eb6663a22efdd78e08805f3f5389566e9fcad254d1b/yarl-1.24.2-cp314-cp314-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/fb/76/641ae371508676492379f16e2fa48f4e2c11741bd63c48be4b12a6b09cba/aiosignal-1.4.0-py3-none-any.whl @@ -2927,7 +2936,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/setuptools-82.0.1-pyh332efcf_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/six-1.17.0-pyhe01879c_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/sniffio-1.3.1-pyhd8ed1ab_2.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.3-pyhd8ed1ab_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.4-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/stack_data-0.6.3-pyhd8ed1ab_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/terminado-0.18.1-pyhc90fa1f_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/tinycss2-1.4.0-pyhd8ed1ab_0.conda @@ -2956,7 +2965,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/1e/d3/26bf1008eb3d2daa8ef4cacc7f3bfdc11818d111f7e2d0201bc6e3b49d45/annotated_doc-0.0.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2a/2d/d4bf65e47cea8ff2c794a600c4fd1273a7902f268757c531e0ee9f18aa58/pooch-1.9.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2b/76/7f1bfd6afff4c5e38e36a3c6d68eb5f4b7311ea80baf693db78d95b603c4/propcache-0.5.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - - pypi: https://files.pythonhosted.org/packages/2e/84/efc7c0bf3a1c5eef81d397f6fddac855becdbb11cb38ff957888603014a7/sqlalchemy-2.0.49-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/2f/40/dc34d1a8d5f1e51fc64640b62b191684da52ca469da9cd74e84936ffa4a6/msgpack-1.1.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/2f/c8/005d1de3af80f54411703d1263a0b9d31276411ec9f273d9432c59b17299/arviz_plots-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/5d/f7e914f7d9325abff4057cee62c0fa70263683189f774473cbfb534cd13b/matplotlib-3.10.9-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl @@ -2970,6 +2978,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/46/b4/0887c88ddfaba1d7140ea335144eb904af97550786ee58bdb295ff10d255/crysfml-0.6.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl - pypi: https://files.pythonhosted.org/packages/52/59/0782e51887ac6b07ffd1570e0364cf901ebc36345fea669969d2084baebb/simple_websocket-1.1.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/57/da/6fbf010c8ebb347679d0d100b22fe9ba5e13fd04046c5df7280d2f0bf706/sqlalchemy-2.0.50.tar.gz - pypi: https://files.pythonhosted.org/packages/58/3b/1cdec6772bdbaf7b25dab360c59f03cadf05492dd724c6540af905389b07/pandas-3.0.3-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/58/e0/f1871f520c359e4e3a2eb7437c9e7e792bb6c356414e8617937561167caf/pycifrw-5.0.1.tar.gz - pypi: https://files.pythonhosted.org/packages/5b/29/74eeb4d3f3ae61ca096b018ad486b3b3c74b17bec09ab4edab721cbefec3/typeguard-4.5.2-py3-none-any.whl @@ -3108,7 +3117,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/setuptools-82.0.1-pyh332efcf_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/six-1.17.0-pyhe01879c_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/sniffio-1.3.1-pyhd8ed1ab_2.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.3-pyhd8ed1ab_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.4-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/stack_data-0.6.3-pyhd8ed1ab_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/terminado-0.18.1-pyhc90fa1f_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/tinycss2-1.4.0-pyhd8ed1ab_0.conda @@ -3180,7 +3189,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/49/b2/97980f3ad4fae37dd7fe31626e2bf75fbf8bdf5d303950ec1fab39a12da8/kiwisolver-1.5.0-cp314-cp314-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/52/59/0782e51887ac6b07ffd1570e0364cf901ebc36345fea669969d2084baebb/simple_websocket-1.1.0-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/55/33/bf28f618c0a9597d14e0b9ee7d1e0622faff738d44fe986ee287cdf1b8d0/sqlalchemy-2.0.49-cp314-cp314-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/58/e0/f1871f520c359e4e3a2eb7437c9e7e792bb6c356414e8617937561167caf/pycifrw-5.0.1.tar.gz - pypi: https://files.pythonhosted.org/packages/5b/29/74eeb4d3f3ae61ca096b018ad486b3b3c74b17bec09ab4edab721cbefec3/typeguard-4.5.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/5b/69/93b34728cc386efdde0c342f8c680b9187dea7beb7adaf6b58a0713be101/mpld3-0.5.12-py3-none-any.whl @@ -3221,6 +3229,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/c7/a0/5ff05d1919ca249508012cad89f08fdc6cfbdaa15b41651c5fe6dffaf1d3/dfo_ls-1.6.5-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/db/7b/8624a203326675d7746a254083a187398090a179335b2e4a20e2ddc46e83/scipy-1.17.1-cp314-cp314-macosx_14_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/dc/83/6d810a8a9ebc9c307989b418840c20e46907c74d707beb67ab566773e6fc/xarray-2026.4.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/df/32/10ac51b4be7cdecd7e93d069251c86dfbf70b7adbd7c67b48ccea6c49e1c/sqlalchemy-2.0.50-cp314-cp314-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/ee/8c/83087ebc47ab0396ce092363001fa37c17153119ee282700c0713a195853/prettytable-3.17.0-py3-none-any.whl @@ -3318,7 +3327,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/setuptools-82.0.1-pyh332efcf_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/six-1.17.0-pyhe01879c_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/sniffio-1.3.1-pyhd8ed1ab_2.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.3-pyhd8ed1ab_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.4-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/stack_data-0.6.3-pyhd8ed1ab_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/terminado-0.18.1-pyh6dadd2b_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/tinycss2-1.4.0-pyhd8ed1ab_0.conda @@ -3404,6 +3413,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/78/3c/2a612b95ddbb9a6bdcb47b7a93c4884f74c6ff22356b2f7b213b16e65c35/pycifstar-0.3.0.tar.gz - pypi: https://files.pythonhosted.org/packages/7b/67/b1944235474aac3f0b0e1b232ce49547f9f9461ca4b943df1b88da5d3f1d/bumps-1.0.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/7d/c2/57f54b03d0f22d4044b8afb9ca0e184f8b1afd57b4f735c2fa70883dc601/contourpy-1.3.3-cp314-cp314-win_amd64.whl + - pypi: https://files.pythonhosted.org/packages/7f/a0/fe92bb9817863bc13ba093bda931979a26cc2ca69f8e8f26d07add3d7c6f/sqlalchemy-2.0.50-cp314-cp314-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/82/3b/64d4899d73f91ba49a8c18a8ff3f0ea8f1c1d75481760df8c68ef5235bf5/rich-15.0.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/82/d0/26c81ffbe588f936d05f395da34046c66322e8067c9fd331c788c4f682f2/diffpy_pdffit2-1.6.0-cp314-cp314-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/88/39/799be3f2f0f38cc727ee3b4f1445fe6d5e4133064ec2e4115069418a5bb6/cloudpickle-3.1.2-py3-none-any.whl @@ -3430,7 +3440,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/c7/0d/67e5b4109ea4a837e80daa87c2c696711955e40449a97e8926672534def2/click-8.4.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/c7/a0/5ff05d1919ca249508012cad89f08fdc6cfbdaa15b41651c5fe6dffaf1d3/dfo_ls-1.6.5-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/cd/8b/601e6566b957ca50e28725cb6c355c59c2c8609751efbecd980db44e0349/pillow-12.2.0-cp314-cp314-win_amd64.whl - - pypi: https://files.pythonhosted.org/packages/cf/4f/8297e4ed88e80baa1f5aa3c484a0ee29ef3c69c7582f206c916973b75057/sqlalchemy-2.0.49-cp314-cp314-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/dc/83/6d810a8a9ebc9c307989b418840c20e46907c74d707beb67ab566773e6fc/xarray-2026.4.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/df/ac/46de6dda46478f7942f839e094970be2d4a861e005c4b3bf07c92e291a09/numpy-2.4.6-cp314-cp314-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/e0/bf/52f25716bbe93745595800f36fb17b73711f14da59ed0bb2eba141bc9f0f/multidict-6.7.1-cp314-cp314-win_amd64.whl @@ -3989,6 +3998,7 @@ packages: - libgcc >=14 - __glibc >=2.17,<3.0.a0 license: MIT + license_family: MIT purls: [] size: 419935 timestamp: 1779396012261 @@ -4104,6 +4114,7 @@ packages: - libnghttp2 >=1.68.1,<2.0a0 - openssl >=3.5.6,<4.0a0 license: MIT + license_family: MIT purls: [] size: 19707853 timestamp: 1779471099457 @@ -4245,6 +4256,7 @@ packages: - cpython >=3.12 - zeromq >=4.3.5,<4.4.0a0 license: BSD-3-Clause + license_family: BSD purls: - pkg:pypi/pyzmq?source=hash-mapping size: 210896 @@ -5793,17 +5805,16 @@ packages: - pkg:pypi/sniffio?source=hash-mapping size: 15698 timestamp: 1762941572482 -- conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.3-pyhd8ed1ab_0.conda - sha256: 23b71ecf089967d2900126920e7f9ff18cdcef82dbff3e2f54ffa360243a17ac - md5: 18de09b20462742fe093ba39185d9bac +- conda: https://conda.anaconda.org/conda-forge/noarch/soupsieve-2.8.4-pyhd8ed1ab_0.conda + sha256: 2afa5fe9331c09b4c4689ddf6ace8fc16c837eae547c57dab325b844072fdd77 + md5: 9e21f087f087f805debe877d88e00a14 depends: - python >=3.10 license: MIT - license_family: MIT purls: - - pkg:pypi/soupsieve?source=hash-mapping - size: 38187 - timestamp: 1769034509657 + - pkg:pypi/soupsieve?source=compressed-mapping + size: 38802 + timestamp: 1779635534390 - conda: https://conda.anaconda.org/conda-forge/noarch/stack_data-0.6.3-pyhd8ed1ab_1.conda sha256: 570da295d421661af487f1595045760526964f41471021056e993e73089e9c41 md5: b1b505328da7a6b246787df4b5a49fbc @@ -6468,6 +6479,7 @@ packages: depends: - __osx >=11.0 license: MIT + license_family: MIT purls: [] size: 122732 timestamp: 1779396113397 @@ -6585,6 +6597,7 @@ packages: - icu >=78.3,<79.0a0 - libsqlite >=3.53.1,<4.0a0 license: MIT + license_family: MIT purls: [] size: 17981016 timestamp: 1779471179908 @@ -6780,6 +6793,7 @@ packages: - _python_abi3_support 1.* - cpython >=3.12 license: BSD-3-Clause + license_family: BSD purls: - pkg:pypi/pyzmq?source=compressed-mapping size: 191432 @@ -7372,6 +7386,7 @@ packages: sha256: 0fad158aaffdb78d3a386e9e078e9cf17f27614750ab5e148d47867bf7c3ee91 md5: d9b8ee334a3a6285cfc991c80edb3e13 license: MIT + license_family: MIT purls: [] size: 32513682 timestamp: 1779471184734 @@ -7580,6 +7595,7 @@ packages: - cpython >=3.12 - zeromq >=4.3.5,<4.3.6.0a0 license: BSD-3-Clause + license_family: BSD purls: - pkg:pypi/pyzmq?source=compressed-mapping size: 182831 @@ -7778,6 +7794,7 @@ packages: - dfo-ls - diffpy-pdffit2 - diffpy-utils + - emcee - gemmi - h5py - lmfit @@ -8576,82 +8593,6 @@ packages: version: 0.5.2 sha256: e5cbfac9f61484f7e9f3597775500cd3ebe8274e9b050c38f9525c77c97520bf requires_python: '>=3.10' -- pypi: https://files.pythonhosted.org/packages/2c/fa/65fcae2ed62f84ab72cf89536c7c3217a156e71a2c111b1305ab6f0690e2/sqlalchemy-2.0.49-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - name: sqlalchemy - version: 2.0.49 - sha256: 3bb9ec6436a820a4c006aad1ac351f12de2f2dbdaad171692ee457a02429b672 - requires_dist: - - importlib-metadata ; python_full_version < '3.8' - - greenlet>=1 ; platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64' - - typing-extensions>=4.6.0 - - greenlet>=1 ; extra == 'asyncio' - - mypy>=0.910 ; extra == 'mypy' - - pyodbc ; extra == 'mssql' - - pymssql ; extra == 'mssql-pymssql' - - pyodbc ; extra == 'mssql-pyodbc' - - mysqlclient>=1.4.0 ; extra == 'mysql' - - mysql-connector-python ; extra == 'mysql-connector' - - mariadb>=1.0.1,!=1.1.2,!=1.1.5,!=1.1.10 ; extra == 'mariadb-connector' - - cx-oracle>=8 ; extra == 'oracle' - - oracledb>=1.0.1 ; extra == 'oracle-oracledb' - - psycopg2>=2.7 ; extra == 'postgresql' - - pg8000>=1.29.1 ; extra == 'postgresql-pg8000' - - greenlet>=1 ; extra == 'postgresql-asyncpg' - - asyncpg ; extra == 'postgresql-asyncpg' - - psycopg2-binary ; extra == 'postgresql-psycopg2binary' - - psycopg2cffi ; extra == 'postgresql-psycopg2cffi' - - psycopg>=3.0.7 ; extra == 'postgresql-psycopg' - - psycopg[binary]>=3.0.7 ; extra == 'postgresql-psycopgbinary' - - pymysql ; extra == 'pymysql' - - greenlet>=1 ; extra == 'aiomysql' - - aiomysql>=0.2.0 ; extra == 'aiomysql' - - greenlet>=1 ; extra == 'aioodbc' - - aioodbc ; extra == 'aioodbc' - - greenlet>=1 ; extra == 'asyncmy' - - asyncmy>=0.2.3,!=0.2.4,!=0.2.6 ; extra == 'asyncmy' - - greenlet>=1 ; extra == 'aiosqlite' - - aiosqlite ; extra == 'aiosqlite' - - typing-extensions!=3.10.0.1 ; extra == 'aiosqlite' - - sqlcipher3-binary ; extra == 'sqlcipher' - requires_python: '>=3.7' -- pypi: https://files.pythonhosted.org/packages/2e/84/efc7c0bf3a1c5eef81d397f6fddac855becdbb11cb38ff957888603014a7/sqlalchemy-2.0.49-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - name: sqlalchemy - version: 2.0.49 - sha256: 685e93e9c8f399b0c96a624799820176312f5ceef958c0f88215af4013d29066 - requires_dist: - - importlib-metadata ; python_full_version < '3.8' - - greenlet>=1 ; platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64' - - typing-extensions>=4.6.0 - - greenlet>=1 ; extra == 'asyncio' - - mypy>=0.910 ; extra == 'mypy' - - pyodbc ; extra == 'mssql' - - pymssql ; extra == 'mssql-pymssql' - - pyodbc ; extra == 'mssql-pyodbc' - - mysqlclient>=1.4.0 ; extra == 'mysql' - - mysql-connector-python ; extra == 'mysql-connector' - - mariadb>=1.0.1,!=1.1.2,!=1.1.5,!=1.1.10 ; extra == 'mariadb-connector' - - cx-oracle>=8 ; extra == 'oracle' - - oracledb>=1.0.1 ; extra == 'oracle-oracledb' - - psycopg2>=2.7 ; extra == 'postgresql' - - pg8000>=1.29.1 ; extra == 'postgresql-pg8000' - - greenlet>=1 ; extra == 'postgresql-asyncpg' - - asyncpg ; extra == 'postgresql-asyncpg' - - psycopg2-binary ; extra == 'postgresql-psycopg2binary' - - psycopg2cffi ; extra == 'postgresql-psycopg2cffi' - - psycopg>=3.0.7 ; extra == 'postgresql-psycopg' - - psycopg[binary]>=3.0.7 ; extra == 'postgresql-psycopgbinary' - - pymysql ; extra == 'pymysql' - - greenlet>=1 ; extra == 'aiomysql' - - aiomysql>=0.2.0 ; extra == 'aiomysql' - - greenlet>=1 ; extra == 'aioodbc' - - aioodbc ; extra == 'aioodbc' - - greenlet>=1 ; extra == 'asyncmy' - - asyncmy>=0.2.3,!=0.2.4,!=0.2.6 ; extra == 'asyncmy' - - greenlet>=1 ; extra == 'aiosqlite' - - aiosqlite ; extra == 'aiosqlite' - - typing-extensions!=3.10.0.1 ; extra == 'aiosqlite' - - sqlcipher3-binary ; extra == 'sqlcipher' - requires_python: '>=3.7' - pypi: https://files.pythonhosted.org/packages/2f/40/dc34d1a8d5f1e51fc64640b62b191684da52ca469da9cd74e84936ffa4a6/msgpack-1.1.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl name: msgpack version: 1.1.2 @@ -9130,87 +9071,11 @@ packages: requires_dist: - numpy requires_python: '>=3.11,<3.15' -- pypi: https://files.pythonhosted.org/packages/47/9e/fd90114059175cac64e4fafa9bf3ac20584384d66de40793ae2e2f26f3bb/sqlalchemy-2.0.49-cp312-cp312-win_amd64.whl - name: sqlalchemy - version: 2.0.49 - sha256: 618a308215b6cececb6240b9abde545e3acdabac7ae3e1d4e666896bf5ba44b4 - requires_dist: - - importlib-metadata ; python_full_version < '3.8' - - greenlet>=1 ; platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64' - - typing-extensions>=4.6.0 - - greenlet>=1 ; extra == 'asyncio' - - mypy>=0.910 ; extra == 'mypy' - - pyodbc ; extra == 'mssql' - - pymssql ; extra == 'mssql-pymssql' - - pyodbc ; extra == 'mssql-pyodbc' - - mysqlclient>=1.4.0 ; extra == 'mysql' - - mysql-connector-python ; extra == 'mysql-connector' - - mariadb>=1.0.1,!=1.1.2,!=1.1.5,!=1.1.10 ; extra == 'mariadb-connector' - - cx-oracle>=8 ; extra == 'oracle' - - oracledb>=1.0.1 ; extra == 'oracle-oracledb' - - psycopg2>=2.7 ; extra == 'postgresql' - - pg8000>=1.29.1 ; extra == 'postgresql-pg8000' - - greenlet>=1 ; extra == 'postgresql-asyncpg' - - asyncpg ; extra == 'postgresql-asyncpg' - - psycopg2-binary ; extra == 'postgresql-psycopg2binary' - - psycopg2cffi ; extra == 'postgresql-psycopg2cffi' - - psycopg>=3.0.7 ; extra == 'postgresql-psycopg' - - psycopg[binary]>=3.0.7 ; extra == 'postgresql-psycopgbinary' - - pymysql ; extra == 'pymysql' - - greenlet>=1 ; extra == 'aiomysql' - - aiomysql>=0.2.0 ; extra == 'aiomysql' - - greenlet>=1 ; extra == 'aioodbc' - - aioodbc ; extra == 'aioodbc' - - greenlet>=1 ; extra == 'asyncmy' - - asyncmy>=0.2.3,!=0.2.4,!=0.2.6 ; extra == 'asyncmy' - - greenlet>=1 ; extra == 'aiosqlite' - - aiosqlite ; extra == 'aiosqlite' - - typing-extensions!=3.10.0.1 ; extra == 'aiosqlite' - - sqlcipher3-binary ; extra == 'sqlcipher' - requires_python: '>=3.7' - pypi: https://files.pythonhosted.org/packages/49/b2/97980f3ad4fae37dd7fe31626e2bf75fbf8bdf5d303950ec1fab39a12da8/kiwisolver-1.5.0-cp314-cp314-macosx_11_0_arm64.whl name: kiwisolver version: 1.5.0 sha256: 0cbe94b69b819209a62cb27bdfa5dc2a8977d8de2f89dfd97ba4f53ed3af754e requires_python: '>=3.10' -- pypi: https://files.pythonhosted.org/packages/49/b3/2de412451330756aaaa72d27131db6dde23995efe62c941184e15242a5fa/sqlalchemy-2.0.49-cp312-cp312-macosx_11_0_arm64.whl - name: sqlalchemy - version: 2.0.49 - sha256: 4bbccb45260e4ff1b7db0be80a9025bb1e6698bdb808b83fff0000f7a90b2c0b - requires_dist: - - importlib-metadata ; python_full_version < '3.8' - - greenlet>=1 ; platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64' - - typing-extensions>=4.6.0 - - greenlet>=1 ; extra == 'asyncio' - - mypy>=0.910 ; extra == 'mypy' - - pyodbc ; extra == 'mssql' - - pymssql ; extra == 'mssql-pymssql' - - pyodbc ; extra == 'mssql-pyodbc' - - mysqlclient>=1.4.0 ; extra == 'mysql' - - mysql-connector-python ; extra == 'mysql-connector' - - mariadb>=1.0.1,!=1.1.2,!=1.1.5,!=1.1.10 ; extra == 'mariadb-connector' - - cx-oracle>=8 ; extra == 'oracle' - - oracledb>=1.0.1 ; extra == 'oracle-oracledb' - - psycopg2>=2.7 ; extra == 'postgresql' - - pg8000>=1.29.1 ; extra == 'postgresql-pg8000' - - greenlet>=1 ; extra == 'postgresql-asyncpg' - - asyncpg ; extra == 'postgresql-asyncpg' - - psycopg2-binary ; extra == 'postgresql-psycopg2binary' - - psycopg2cffi ; extra == 'postgresql-psycopg2cffi' - - psycopg>=3.0.7 ; extra == 'postgresql-psycopg' - - psycopg[binary]>=3.0.7 ; extra == 'postgresql-psycopgbinary' - - pymysql ; extra == 'pymysql' - - greenlet>=1 ; extra == 'aiomysql' - - aiomysql>=0.2.0 ; extra == 'aiomysql' - - greenlet>=1 ; extra == 'aioodbc' - - aioodbc ; extra == 'aioodbc' - - greenlet>=1 ; extra == 'asyncmy' - - asyncmy>=0.2.3,!=0.2.4,!=0.2.6 ; extra == 'asyncmy' - - greenlet>=1 ; extra == 'aiosqlite' - - aiosqlite ; extra == 'aiosqlite' - - typing-extensions!=3.10.0.1 ; extra == 'aiosqlite' - - sqlcipher3-binary ; extra == 'sqlcipher' - requires_python: '>=3.7' - pypi: https://files.pythonhosted.org/packages/4b/39/f0e8ea762a764a9dc52aa7dabcfad51a354819de1f0d4652b6a1122424d6/scipy-1.17.1-cp314-cp314-win_amd64.whl name: scipy version: 1.17.1 @@ -9308,10 +9173,33 @@ packages: - pytest-benchmark ; extra == 'testing' - coverage ; extra == 'testing' requires_python: '>=3.9' -- pypi: https://files.pythonhosted.org/packages/55/33/bf28f618c0a9597d14e0b9ee7d1e0622faff738d44fe986ee287cdf1b8d0/sqlalchemy-2.0.49-cp314-cp314-macosx_11_0_arm64.whl +- pypi: https://files.pythonhosted.org/packages/56/9e/d13e40f83b8d0a94430e6778ce1d94a43b38cf2efe63278bdd2b4c65abbf/ruff-0.15.14-py3-none-macosx_11_0_arm64.whl + name: ruff + version: 0.15.14 + sha256: 48d5909d7d06276ce7dde6d32bfa4b0d4cb2651145cd8ee4b440722cbc77832f + requires_python: '>=3.7' +- pypi: https://files.pythonhosted.org/packages/57/d8/8d44036d7eb7b6a8ec4c5494ea0c8c8b94fbc0ed3991c1a7adf230df03bf/aiohttp-3.13.5-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl + name: aiohttp + version: 3.13.5 + sha256: b18f31b80d5a33661e08c89e202edabf1986e9b49c42b4504371daeaa11b47c1 + requires_dist: + - aiohappyeyeballs>=2.5.0 + - aiosignal>=1.4.0 + - async-timeout>=4.0,<6.0 ; python_full_version < '3.11' + - attrs>=17.3.0 + - frozenlist>=1.1.1 + - multidict>=4.5,<7.0 + - propcache>=0.2.0 + - yarl>=1.17.0,<2.0 + - aiodns>=3.3.0 ; extra == 'speedups' + - brotli>=1.2 ; platform_python_implementation == 'CPython' and extra == 'speedups' + - brotlicffi>=1.2 ; platform_python_implementation != 'CPython' and extra == 'speedups' + - backports-zstd ; python_full_version < '3.14' and platform_python_implementation == 'CPython' and extra == 'speedups' + requires_python: '>=3.9' +- pypi: https://files.pythonhosted.org/packages/57/da/6fbf010c8ebb347679d0d100b22fe9ba5e13fd04046c5df7280d2f0bf706/sqlalchemy-2.0.50.tar.gz name: sqlalchemy - version: 2.0.49 - sha256: 233088b4b99ebcbc5258c755a097aa52fbf90727a03a5a80781c4b9c54347a2e + version: 2.0.50 + sha256: af5607d11ef90fd6a5c0549fe0045dce1663d427426bcfb506dcb5346a85a3b9 requires_dist: - importlib-metadata ; python_full_version < '3.8' - greenlet>=1 ; platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64' @@ -9346,29 +9234,6 @@ packages: - typing-extensions!=3.10.0.1 ; extra == 'aiosqlite' - sqlcipher3-binary ; extra == 'sqlcipher' requires_python: '>=3.7' -- pypi: https://files.pythonhosted.org/packages/56/9e/d13e40f83b8d0a94430e6778ce1d94a43b38cf2efe63278bdd2b4c65abbf/ruff-0.15.14-py3-none-macosx_11_0_arm64.whl - name: ruff - version: 0.15.14 - sha256: 48d5909d7d06276ce7dde6d32bfa4b0d4cb2651145cd8ee4b440722cbc77832f - requires_python: '>=3.7' -- pypi: https://files.pythonhosted.org/packages/57/d8/8d44036d7eb7b6a8ec4c5494ea0c8c8b94fbc0ed3991c1a7adf230df03bf/aiohttp-3.13.5-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - name: aiohttp - version: 3.13.5 - sha256: b18f31b80d5a33661e08c89e202edabf1986e9b49c42b4504371daeaa11b47c1 - requires_dist: - - aiohappyeyeballs>=2.5.0 - - aiosignal>=1.4.0 - - async-timeout>=4.0,<6.0 ; python_full_version < '3.11' - - attrs>=17.3.0 - - frozenlist>=1.1.1 - - multidict>=4.5,<7.0 - - propcache>=0.2.0 - - yarl>=1.17.0,<2.0 - - aiodns>=3.3.0 ; extra == 'speedups' - - brotli>=1.2 ; platform_python_implementation == 'CPython' and extra == 'speedups' - - brotlicffi>=1.2 ; platform_python_implementation != 'CPython' and extra == 'speedups' - - backports-zstd ; python_full_version < '3.14' and platform_python_implementation == 'CPython' and extra == 'speedups' - requires_python: '>=3.9' - pypi: https://files.pythonhosted.org/packages/58/3b/1cdec6772bdbaf7b25dab360c59f03cadf05492dd724c6540af905389b07/pandas-3.0.3-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl name: pandas version: 3.0.3 @@ -10046,6 +9911,44 @@ packages: - pyyaml - pygments>=2.19.1 ; extra == 'extra' requires_python: '>=3.9' +- pypi: https://files.pythonhosted.org/packages/7f/a0/fe92bb9817863bc13ba093bda931979a26cc2ca69f8e8f26d07add3d7c6f/sqlalchemy-2.0.50-cp314-cp314-win_amd64.whl + name: sqlalchemy + version: 2.0.50 + sha256: 15708c613cd5005b7dffe1f66ee6a63ee8f5e46799f71c70ebad74178c676a39 + requires_dist: + - importlib-metadata ; python_full_version < '3.8' + - greenlet>=1 ; platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64' + - typing-extensions>=4.6.0 + - greenlet>=1 ; extra == 'asyncio' + - mypy>=0.910 ; extra == 'mypy' + - pyodbc ; extra == 'mssql' + - pymssql ; extra == 'mssql-pymssql' + - pyodbc ; extra == 'mssql-pyodbc' + - mysqlclient>=1.4.0 ; extra == 'mysql' + - mysql-connector-python ; extra == 'mysql-connector' + - mariadb>=1.0.1,!=1.1.2,!=1.1.5,!=1.1.10 ; extra == 'mariadb-connector' + - cx-oracle>=8 ; extra == 'oracle' + - oracledb>=1.0.1 ; extra == 'oracle-oracledb' + - psycopg2>=2.7 ; extra == 'postgresql' + - pg8000>=1.29.1 ; extra == 'postgresql-pg8000' + - greenlet>=1 ; extra == 'postgresql-asyncpg' + - asyncpg ; extra == 'postgresql-asyncpg' + - psycopg2-binary ; extra == 'postgresql-psycopg2binary' + - psycopg2cffi ; extra == 'postgresql-psycopg2cffi' + - psycopg>=3.0.7 ; extra == 'postgresql-psycopg' + - psycopg[binary]>=3.0.7 ; extra == 'postgresql-psycopgbinary' + - pymysql ; extra == 'pymysql' + - greenlet>=1 ; extra == 'aiomysql' + - aiomysql>=0.2.0 ; extra == 'aiomysql' + - greenlet>=1 ; extra == 'aioodbc' + - aioodbc ; extra == 'aioodbc' + - greenlet>=1 ; extra == 'asyncmy' + - asyncmy>=0.2.3,!=0.2.4,!=0.2.6 ; extra == 'asyncmy' + - greenlet>=1 ; extra == 'aiosqlite' + - aiosqlite ; extra == 'aiosqlite' + - typing-extensions!=3.10.0.1 ; extra == 'aiosqlite' + - sqlcipher3-binary ; extra == 'sqlcipher' + requires_python: '>=3.7' - pypi: https://files.pythonhosted.org/packages/80/31/0b2517913687895f5904325c2069d6a3b78f66cc641a86a2baf75a05dcbb/multidict-6.7.1-cp312-cp312-win_amd64.whl name: multidict version: 6.7.1 @@ -10846,6 +10749,44 @@ packages: requires_dist: - pyyaml>=3.10 ; extra == 'watchmedo' requires_python: '>=3.9' +- pypi: https://files.pythonhosted.org/packages/b6/2d/314a6690dda4b9cfc571eab1a63cf6fe6e1470aa3759ccda6aa016ee0f5a/sqlalchemy-2.0.50-cp312-cp312-win_amd64.whl + name: sqlalchemy + version: 2.0.50 + sha256: 9d1af51558029a156a70986b7df88f042b3d158d7c8d8fb5072912d4b32d89c7 + requires_dist: + - importlib-metadata ; python_full_version < '3.8' + - greenlet>=1 ; platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64' + - typing-extensions>=4.6.0 + - greenlet>=1 ; extra == 'asyncio' + - mypy>=0.910 ; extra == 'mypy' + - pyodbc ; extra == 'mssql' + - pymssql ; extra == 'mssql-pymssql' + - pyodbc ; extra == 'mssql-pyodbc' + - mysqlclient>=1.4.0 ; extra == 'mysql' + - mysql-connector-python ; extra == 'mysql-connector' + - mariadb>=1.0.1,!=1.1.2,!=1.1.5,!=1.1.10 ; extra == 'mariadb-connector' + - cx-oracle>=8 ; extra == 'oracle' + - oracledb>=1.0.1 ; extra == 'oracle-oracledb' + - psycopg2>=2.7 ; extra == 'postgresql' + - pg8000>=1.29.1 ; extra == 'postgresql-pg8000' + - greenlet>=1 ; extra == 'postgresql-asyncpg' + - asyncpg ; extra == 'postgresql-asyncpg' + - psycopg2-binary ; extra == 'postgresql-psycopg2binary' + - psycopg2cffi ; extra == 'postgresql-psycopg2cffi' + - psycopg>=3.0.7 ; extra == 'postgresql-psycopg' + - psycopg[binary]>=3.0.7 ; extra == 'postgresql-psycopgbinary' + - pymysql ; extra == 'pymysql' + - greenlet>=1 ; extra == 'aiomysql' + - aiomysql>=0.2.0 ; extra == 'aiomysql' + - greenlet>=1 ; extra == 'aioodbc' + - aioodbc ; extra == 'aioodbc' + - greenlet>=1 ; extra == 'asyncmy' + - asyncmy>=0.2.3,!=0.2.4,!=0.2.6 ; extra == 'asyncmy' + - greenlet>=1 ; extra == 'aiosqlite' + - aiosqlite ; extra == 'aiosqlite' + - typing-extensions!=3.10.0.1 ; extra == 'aiosqlite' + - sqlcipher3-binary ; extra == 'sqlcipher' + requires_python: '>=3.7' - pypi: https://files.pythonhosted.org/packages/b7/18/4880dd762e40cd360c1bf06e890c5a97b997e91cb324602b1a19950ad5ce/matplotlib-3.10.9-cp312-cp312-macosx_11_0_arm64.whl name: matplotlib version: 3.10.9 @@ -11067,6 +11008,44 @@ packages: - brotlicffi>=1.2 ; platform_python_implementation != 'CPython' and extra == 'speedups' - backports-zstd ; python_full_version < '3.14' and platform_python_implementation == 'CPython' and extra == 'speedups' requires_python: '>=3.9' +- pypi: https://files.pythonhosted.org/packages/be/b0/a9d19b43f38f878b1278bca5b00b909f7540d41494396dd2561f9ad0956d/sqlalchemy-2.0.50-cp312-cp312-macosx_11_0_arm64.whl + name: sqlalchemy + version: 2.0.50 + sha256: 23ae23d8b9d344d30d0a92f06d45825024a5790f1c1dd4cf452636a50d3e58cb + requires_dist: + - importlib-metadata ; python_full_version < '3.8' + - greenlet>=1 ; platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64' + - typing-extensions>=4.6.0 + - greenlet>=1 ; extra == 'asyncio' + - mypy>=0.910 ; extra == 'mypy' + - pyodbc ; extra == 'mssql' + - pymssql ; extra == 'mssql-pymssql' + - pyodbc ; extra == 'mssql-pyodbc' + - mysqlclient>=1.4.0 ; extra == 'mysql' + - mysql-connector-python ; extra == 'mysql-connector' + - mariadb>=1.0.1,!=1.1.2,!=1.1.5,!=1.1.10 ; extra == 'mariadb-connector' + - cx-oracle>=8 ; extra == 'oracle' + - oracledb>=1.0.1 ; extra == 'oracle-oracledb' + - psycopg2>=2.7 ; extra == 'postgresql' + - pg8000>=1.29.1 ; extra == 'postgresql-pg8000' + - greenlet>=1 ; extra == 'postgresql-asyncpg' + - asyncpg ; extra == 'postgresql-asyncpg' + - psycopg2-binary ; extra == 'postgresql-psycopg2binary' + - psycopg2cffi ; extra == 'postgresql-psycopg2cffi' + - psycopg>=3.0.7 ; extra == 'postgresql-psycopg' + - psycopg[binary]>=3.0.7 ; extra == 'postgresql-psycopgbinary' + - pymysql ; extra == 'pymysql' + - greenlet>=1 ; extra == 'aiomysql' + - aiomysql>=0.2.0 ; extra == 'aiomysql' + - greenlet>=1 ; extra == 'aioodbc' + - aioodbc ; extra == 'aioodbc' + - greenlet>=1 ; extra == 'asyncmy' + - asyncmy>=0.2.3,!=0.2.4,!=0.2.6 ; extra == 'asyncmy' + - greenlet>=1 ; extra == 'aiosqlite' + - aiosqlite ; extra == 'aiosqlite' + - typing-extensions!=3.10.0.1 ; extra == 'aiosqlite' + - sqlcipher3-binary ; extra == 'sqlcipher' + requires_python: '>=3.7' - pypi: https://files.pythonhosted.org/packages/bf/50/98b146aea0f1cd7531d25f12bea69fa9ce8d1662124f93fb30dc4511b65e/docstring_parser_fork-0.0.14-py3-none-any.whl name: docstring-parser-fork version: 0.0.14 @@ -11331,44 +11310,6 @@ packages: - xlsxwriter>=3.2.0 ; extra == 'all' - zstandard>=0.23.0 ; extra == 'all' requires_python: '>=3.11' -- pypi: https://files.pythonhosted.org/packages/cf/4f/8297e4ed88e80baa1f5aa3c484a0ee29ef3c69c7582f206c916973b75057/sqlalchemy-2.0.49-cp314-cp314-win_amd64.whl - name: sqlalchemy - version: 2.0.49 - sha256: 77641d299179c37b89cf2343ca9972c88bb6eef0d5fc504a2f86afd15cd5adf5 - requires_dist: - - importlib-metadata ; python_full_version < '3.8' - - greenlet>=1 ; platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64' - - typing-extensions>=4.6.0 - - greenlet>=1 ; extra == 'asyncio' - - mypy>=0.910 ; extra == 'mypy' - - pyodbc ; extra == 'mssql' - - pymssql ; extra == 'mssql-pymssql' - - pyodbc ; extra == 'mssql-pyodbc' - - mysqlclient>=1.4.0 ; extra == 'mysql' - - mysql-connector-python ; extra == 'mysql-connector' - - mariadb>=1.0.1,!=1.1.2,!=1.1.5,!=1.1.10 ; extra == 'mariadb-connector' - - cx-oracle>=8 ; extra == 'oracle' - - oracledb>=1.0.1 ; extra == 'oracle-oracledb' - - psycopg2>=2.7 ; extra == 'postgresql' - - pg8000>=1.29.1 ; extra == 'postgresql-pg8000' - - greenlet>=1 ; extra == 'postgresql-asyncpg' - - asyncpg ; extra == 'postgresql-asyncpg' - - psycopg2-binary ; extra == 'postgresql-psycopg2binary' - - psycopg2cffi ; extra == 'postgresql-psycopg2cffi' - - psycopg>=3.0.7 ; extra == 'postgresql-psycopg' - - psycopg[binary]>=3.0.7 ; extra == 'postgresql-psycopgbinary' - - pymysql ; extra == 'pymysql' - - greenlet>=1 ; extra == 'aiomysql' - - aiomysql>=0.2.0 ; extra == 'aiomysql' - - greenlet>=1 ; extra == 'aioodbc' - - aioodbc ; extra == 'aioodbc' - - greenlet>=1 ; extra == 'asyncmy' - - asyncmy>=0.2.3,!=0.2.4,!=0.2.6 ; extra == 'asyncmy' - - greenlet>=1 ; extra == 'aiosqlite' - - aiosqlite ; extra == 'aiosqlite' - - typing-extensions!=3.10.0.1 ; extra == 'aiosqlite' - - sqlcipher3-binary ; extra == 'sqlcipher' - requires_python: '>=3.7' - pypi: https://files.pythonhosted.org/packages/cf/a9/599c28631bad314d219cf9ffd40e985b24d603fc8a2f4ccc5ae8419a535b/scipy-1.17.1-cp312-cp312-macosx_14_0_arm64.whl name: scipy version: 1.17.1 @@ -11633,6 +11574,44 @@ packages: - mkdocs-section-index ; extra == 'docs' - mkdocs-literate-nav ; extra == 'docs' requires_python: '>=3.10' +- pypi: https://files.pythonhosted.org/packages/df/32/10ac51b4be7cdecd7e93d069251c86dfbf70b7adbd7c67b48ccea6c49e1c/sqlalchemy-2.0.50-cp314-cp314-macosx_11_0_arm64.whl + name: sqlalchemy + version: 2.0.50 + sha256: c966932507a4d7d0a37314927dbfcd89720e3f37d2a1e3352e7ae7939fa8e8a0 + requires_dist: + - importlib-metadata ; python_full_version < '3.8' + - greenlet>=1 ; platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64' + - typing-extensions>=4.6.0 + - greenlet>=1 ; extra == 'asyncio' + - mypy>=0.910 ; extra == 'mypy' + - pyodbc ; extra == 'mssql' + - pymssql ; extra == 'mssql-pymssql' + - pyodbc ; extra == 'mssql-pyodbc' + - mysqlclient>=1.4.0 ; extra == 'mysql' + - mysql-connector-python ; extra == 'mysql-connector' + - mariadb>=1.0.1,!=1.1.2,!=1.1.5,!=1.1.10 ; extra == 'mariadb-connector' + - cx-oracle>=8 ; extra == 'oracle' + - oracledb>=1.0.1 ; extra == 'oracle-oracledb' + - psycopg2>=2.7 ; extra == 'postgresql' + - pg8000>=1.29.1 ; extra == 'postgresql-pg8000' + - greenlet>=1 ; extra == 'postgresql-asyncpg' + - asyncpg ; extra == 'postgresql-asyncpg' + - psycopg2-binary ; extra == 'postgresql-psycopg2binary' + - psycopg2cffi ; extra == 'postgresql-psycopg2cffi' + - psycopg>=3.0.7 ; extra == 'postgresql-psycopg' + - psycopg[binary]>=3.0.7 ; extra == 'postgresql-psycopgbinary' + - pymysql ; extra == 'pymysql' + - greenlet>=1 ; extra == 'aiomysql' + - aiomysql>=0.2.0 ; extra == 'aiomysql' + - greenlet>=1 ; extra == 'aioodbc' + - aioodbc ; extra == 'aioodbc' + - greenlet>=1 ; extra == 'asyncmy' + - asyncmy>=0.2.3,!=0.2.4,!=0.2.6 ; extra == 'asyncmy' + - greenlet>=1 ; extra == 'aiosqlite' + - aiosqlite ; extra == 'aiosqlite' + - typing-extensions!=3.10.0.1 ; extra == 'aiosqlite' + - sqlcipher3-binary ; extra == 'sqlcipher' + requires_python: '>=3.7' - pypi: https://files.pythonhosted.org/packages/df/ac/46de6dda46478f7942f839e094970be2d4a861e005c4b3bf07c92e291a09/numpy-2.4.6-cp314-cp314-win_amd64.whl name: numpy version: 2.4.6 @@ -11798,6 +11777,17 @@ packages: requires_dist: - tomli ; python_full_version <= '3.11' and extra == 'toml' requires_python: '>=3.10' +- pypi: https://files.pythonhosted.org/packages/f9/ef/2196b9bf88ffa1bde45853c72df021fbd07a8fa91a0f59a22d14a050dc04/emcee-3.1.6-py2.py3-none-any.whl + name: emcee + version: 3.1.6 + sha256: f2d63752023bdccf744461450e512a5b417ae7d28f18e12acd76a33de87580cb + requires_dist: + - numpy + - h5py ; extra == 'extras' + - scipy ; extra == 'extras' + - pytest ; extra == 'tests' + - pytest-cov ; extra == 'tests' + - coverage[toml] ; extra == 'tests' - pypi: https://files.pythonhosted.org/packages/fa/bc/8b8ec5a4bfc5b9cf3ce27a118339e994f88410be5677c96493e0ea28e76d/dunamai-1.26.1-py3-none-any.whl name: dunamai version: 1.26.1 diff --git a/pixi.toml b/pixi.toml index 70a8a310e..c6605061f 100644 --- a/pixi.toml +++ b/pixi.toml @@ -31,6 +31,9 @@ macos = '14.0' #libc = { family = 'glibc', version = '2.35' } libc = '2.35' +[pypi-dependencies] +emcee = '>=3.1' + # Non-default features: # Set specific Python versions to be used in CI testing. diff --git a/pyproject.toml b/pyproject.toml index d953cfa19..92ba22ef2 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -34,6 +34,7 @@ dependencies = [ 'sympy', # Symbolic mathematics library 'lmfit', # Non-linear optimization and curve fitting 'bumps', # Non-linear optimization and curve fitting + 'emcee>=3.1', # Affine-invariant MCMC sampler 'dfo-ls', # Non-linear optimization and curve fitting 'gemmi', # Crystallography library 'cryspy', # Calculations of diffraction patterns From cf08a5599d01b17e0a5c93f3499e179f85407e61 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sun, 24 May 2026 21:57:55 +0200 Subject: [PATCH 09/65] Register emcee minimizer enum value --- docs/dev/plans/emcee-minimizer.md | 2 +- src/easydiffraction/analysis/minimizers/enums.py | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index 29c5c5472..b586d0f67 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -279,7 +279,7 @@ Mark `[x]` as each step lands. Files modified by this step: `pyproject.toml`, `pixi.toml`, `pixi.lock`. Commit: `Add emcee runtime dependency` -- [ ] **P1.2 — Register `MinimizerTypeEnum.EMCEE`.** Add the enum member +- [x] **P1.2 — Register `MinimizerTypeEnum.EMCEE`.** Add the enum member with value `'emcee'` to `src/easydiffraction/analysis/minimizers/enums.py`. No other code wiring yet. Commit: `Register emcee minimizer enum value` diff --git a/src/easydiffraction/analysis/minimizers/enums.py b/src/easydiffraction/analysis/minimizers/enums.py index 6931960e0..272861835 100644 --- a/src/easydiffraction/analysis/minimizers/enums.py +++ b/src/easydiffraction/analysis/minimizers/enums.py @@ -19,6 +19,7 @@ class MinimizerTypeEnum(StrEnum): BUMPS_DREAM = 'bumps (dream)' BUMPS_AMOEBA = 'bumps (amoeba)' BUMPS_DE = 'bumps (de)' + EMCEE = 'emcee' @classmethod def default(cls) -> MinimizerTypeEnum: @@ -49,6 +50,7 @@ def description(self) -> str: MinimizerTypeEnum.BUMPS_DREAM: ('BUMPS library with DREAM Bayesian sampling'), MinimizerTypeEnum.BUMPS_AMOEBA: ('BUMPS library with Nelder-Mead simplex method'), MinimizerTypeEnum.BUMPS_DE: ('BUMPS library with differential evolution method'), + MinimizerTypeEnum.EMCEE: ('emcee affine-invariant ensemble Bayesian sampling'), } return descriptions.get(self, '') From f7b57794d47ec80d165515f0db72f616bb78ae8d Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sun, 24 May 2026 21:59:27 +0200 Subject: [PATCH 10/65] Remove emcee version pin --- docs/dev/plans/emcee-minimizer.md | 4 ++-- pixi.lock | 3 +++ pyproject.toml | 2 +- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index b586d0f67..f1920132e 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -254,7 +254,7 @@ When the matching open-issue is fully resolved, move it to `/emcee_chain` is present, expose a helper to construct an `emcee.backends.HDFBackend(path, name='emcee_chain', read_only=True)` for inspection/visualisation. -- `pyproject.toml`, `pixi.toml`, and `pixi.lock` — add `emcee>=3.1` as a +- `pyproject.toml` — add `emcee` as a direct runtime dependency and refresh the lockfile via `pixi lock` (CI installs from the lockfile, not from the manifest files alone). @@ -267,7 +267,7 @@ When the matching open-issue is fully resolved, move it to Mark `[x]` as each step lands. - [x] **P1.1 — Add emcee dependency and refresh the lockfile.** - - Add `emcee>=3.1` to `pyproject.toml` (runtime dependencies, not just + - Add `emcee` to `pyproject.toml` (runtime dependencies, not just the `doc` extra — the existing lockfile carries emcee only as `extra == 'doc'` which CI does not install for runtime). - Add the same dependency to `pixi.toml` (runtime feature). diff --git a/pixi.lock b/pixi.lock index a4c4ec86f..5a9141de8 100644 --- a/pixi.lock +++ b/pixi.lock @@ -3023,6 +3023,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/f1/16/d905e7f53e661ce2c24686c38048d8e2b750ffc4350009d41c4e6c6c9826/h5py-3.16.0-cp314-cp314-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/f2/f2/728f041460f1b9739b85ee23b45fa5a505962ea11fd85bdbe2a02b021373/darkdetect-0.8.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f3/eb/ebffaa97dc55502df69584a8f0dcf07f69a3e0b3e2323670a2722db9aa39/numpy-2.4.6-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl + - pypi: https://files.pythonhosted.org/packages/f9/ef/2196b9bf88ffa1bde45853c72df021fbd07a8fa91a0f59a22d14a050dc04/emcee-3.1.6-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fb/76/641ae371508676492379f16e2fa48f4e2c11741bd63c48be4b12a6b09cba/aiosignal-1.4.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fe/3b/8ec5074bcfc450fe84273713b4b0a0dd47c0249358f5d82eb8104ffe2520/multidict-6.7.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl osx-arm64: @@ -3235,6 +3236,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/ee/8c/83087ebc47ab0396ce092363001fa37c17153119ee282700c0713a195853/prettytable-3.17.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f1/4f/733c48f270565d78b4544f2baddc2fb2a245e5a8640254b12c36ac7ac68e/multidict-6.7.1-cp314-cp314-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/f2/f2/728f041460f1b9739b85ee23b45fa5a505962ea11fd85bdbe2a02b021373/darkdetect-0.8.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/f9/ef/2196b9bf88ffa1bde45853c72df021fbd07a8fa91a0f59a22d14a050dc04/emcee-3.1.6-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fb/76/641ae371508676492379f16e2fa48f4e2c11741bd63c48be4b12a6b09cba/aiosignal-1.4.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fd/e1/3542a9cb596cadd76fcef413f19c79216e002623158befe6daa03dbfa88c/contourpy-1.3.3-cp314-cp314-macosx_11_0_arm64.whl win-64: @@ -3447,6 +3449,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/ee/8c/83087ebc47ab0396ce092363001fa37c17153119ee282700c0713a195853/prettytable-3.17.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f2/f2/728f041460f1b9739b85ee23b45fa5a505962ea11fd85bdbe2a02b021373/darkdetect-0.8.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/f9/ef/2196b9bf88ffa1bde45853c72df021fbd07a8fa91a0f59a22d14a050dc04/emcee-3.1.6-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/fa/d9/5582d57e2b2db9b85eb6663a22efdd78e08805f3f5389566e9fcad254d1b/yarl-1.24.2-cp314-cp314-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/fb/76/641ae371508676492379f16e2fa48f4e2c11741bd63c48be4b12a6b09cba/aiosignal-1.4.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/ff/1c/a28b27effb13a381fe077ea3e3e78f6debd6315f2b3edff67bbb93d0ef51/gemmi-0.7.5-cp314-cp314-win_amd64.whl diff --git a/pyproject.toml b/pyproject.toml index 92ba22ef2..790d72195 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -34,7 +34,7 @@ dependencies = [ 'sympy', # Symbolic mathematics library 'lmfit', # Non-linear optimization and curve fitting 'bumps', # Non-linear optimization and curve fitting - 'emcee>=3.1', # Affine-invariant MCMC sampler + 'emcee', # Affine-invariant MCMC sampler 'dfo-ls', # Non-linear optimization and curve fitting 'gemmi', # Crystallography library 'cryspy', # Calculations of diffraction patterns From 85af677e9bf37e38f35453bcaf34c75b04412876 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sun, 24 May 2026 21:59:59 +0200 Subject: [PATCH 11/65] Add EmceeMinimizer category class --- docs/dev/plans/emcee-minimizer.md | 2 +- .../analysis/categories/minimizer/__init__.py | 1 + .../analysis/categories/minimizer/emcee.py | 116 ++++++++++++++++++ 3 files changed, 118 insertions(+), 1 deletion(-) create mode 100644 src/easydiffraction/analysis/categories/minimizer/emcee.py diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index f1920132e..8ab321dc9 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -284,7 +284,7 @@ Mark `[x]` as each step lands. `src/easydiffraction/analysis/minimizers/enums.py`. No other code wiring yet. Commit: `Register emcee minimizer enum value` -- [ ] **P1.3 — Add `EmceeMinimizer` category class.** New file +- [x] **P1.3 — Add `EmceeMinimizer` category class.** New file `src/easydiffraction/analysis/categories/minimizer/emcee.py`. `EmceeMinimizer(BayesianMinimizerBase)` declares: - `type_info` with `tag=MinimizerTypeEnum.EMCEE` and a description. diff --git a/src/easydiffraction/analysis/categories/minimizer/__init__.py b/src/easydiffraction/analysis/categories/minimizer/__init__.py index 17ab65c81..933b35a69 100644 --- a/src/easydiffraction/analysis/categories/minimizer/__init__.py +++ b/src/easydiffraction/analysis/categories/minimizer/__init__.py @@ -9,6 +9,7 @@ from easydiffraction.analysis.categories.minimizer.bumps_dream import BumpsDreamMinimizer from easydiffraction.analysis.categories.minimizer.bumps_lm import BumpsLmMinimizer from easydiffraction.analysis.categories.minimizer.dfols import DfolsMinimizer +from easydiffraction.analysis.categories.minimizer.emcee import EmceeMinimizer from easydiffraction.analysis.categories.minimizer.factory import MinimizerCategoryFactory from easydiffraction.analysis.categories.minimizer.lmfit import LmfitMinimizer from easydiffraction.analysis.categories.minimizer.lmfit_least_squares import ( diff --git a/src/easydiffraction/analysis/categories/minimizer/emcee.py b/src/easydiffraction/analysis/categories/minimizer/emcee.py new file mode 100644 index 000000000..e71d8b1c7 --- /dev/null +++ b/src/easydiffraction/analysis/categories/minimizer/emcee.py @@ -0,0 +1,116 @@ +# SPDX-FileCopyrightText: 2026 EasyScience contributors +# SPDX-License-Identifier: BSD-3-Clause +"""Persisted category for the emcee minimizer.""" + +from __future__ import annotations + +from typing import ClassVar + +from easydiffraction.analysis.categories.minimizer.bayesian_base import BayesianMinimizerBase +from easydiffraction.analysis.categories.minimizer.factory import MinimizerCategoryFactory +from easydiffraction.analysis.minimizers.enums import InitializationMethodEnum +from easydiffraction.analysis.minimizers.enums import MinimizerTypeEnum +from easydiffraction.core.metadata import TypeInfo +from easydiffraction.core.validation import AttributeSpec +from easydiffraction.core.validation import MembershipValidator +from easydiffraction.core.variable import StringDescriptor +from easydiffraction.io.cif.handler import CifHandler + +DEFAULT_SAMPLING_STEPS = 5000 +DEFAULT_BURN_IN_STEPS = 1000 +DEFAULT_THINNING_INTERVAL = 5 +DEFAULT_POPULATION_SIZE = 32 +DEFAULT_PARALLEL_WORKERS = 0 +DEFAULT_INITIALIZATION_METHOD = InitializationMethodEnum.BALL +DEFAULT_PROPOSAL_MOVES = 'stretch' +SUPPORTED_PROPOSAL_MOVES = ('stretch', 'de', 'de_snooker', 'walk') + + +@MinimizerCategoryFactory.register +class EmceeMinimizer(BayesianMinimizerBase): + """Persisted settings for the emcee minimizer.""" + + _engine_metadata: ClassVar[dict[str, str]] = { + 'optimizer_name': 'emcee', + 'method_name': 'stretch', + } + _expected_descriptor_names: ClassVar[tuple[str, ...]] = ( + *BayesianMinimizerBase._expected_descriptor_names, + 'proposal_moves', + ) + _native_key_map: ClassVar[dict[str, str]] = { + 'sampling_steps': 'nsteps', + 'burn_in_steps': 'nburn', + 'thinning_interval': 'thin', + 'population_size': 'nwalkers', + 'parallel_workers': 'parallel_workers', + 'initialization_method': 'initialization_method', + 'random_seed': 'random_seed', + 'proposal_moves': 'proposal_moves', + } + _setting_descriptor_names: ClassVar[tuple[str, ...]] = ( + *BayesianMinimizerBase._setting_descriptor_names, + 'proposal_moves', + ) + _supported_initialization_methods: ClassVar[tuple[InitializationMethodEnum, ...]] = ( + InitializationMethodEnum.BALL, + InitializationMethodEnum.UNIFORM, + InitializationMethodEnum.PRIOR, + ) + _native_initialization_methods: ClassVar[dict[InitializationMethodEnum, str]] = { + InitializationMethodEnum.BALL: InitializationMethodEnum.BALL.value, + InitializationMethodEnum.UNIFORM: InitializationMethodEnum.UNIFORM.value, + InitializationMethodEnum.PRIOR: InitializationMethodEnum.PRIOR.value, + } + + type_info = TypeInfo( + tag=MinimizerTypeEnum.EMCEE, + description='emcee affine-invariant ensemble Bayesian sampling', + ) + + def __init__(self) -> None: + super().__init__() + self._sampling_steps = self._sampling_steps_descriptor(DEFAULT_SAMPLING_STEPS) + self._burn_in_steps = self._burn_in_steps_descriptor(DEFAULT_BURN_IN_STEPS) + self._thinning_interval = self._thinning_interval_descriptor(DEFAULT_THINNING_INTERVAL) + self._population_size = self._population_size_descriptor(DEFAULT_POPULATION_SIZE) + self._parallel_workers = self._parallel_workers_descriptor(DEFAULT_PARALLEL_WORKERS) + self._initialization_method = self._initialization_method_descriptor() + self._random_seed = self._random_seed_descriptor() + self._proposal_moves = self._proposal_moves_descriptor() + + @classmethod + def _initialization_method_descriptor(cls) -> StringDescriptor: + """Create an emcee initialization-method descriptor.""" + allowed = [member.value for member in cls._supported_initialization_methods] + return StringDescriptor( + name='initialization_method', + description='emcee walker initialization method.', + value_spec=AttributeSpec( + default=DEFAULT_INITIALIZATION_METHOD.value, + validator=MembershipValidator(allowed=allowed), + ), + cif_handler=CifHandler(names=['_minimizer.initialization_method']), + ) + + @staticmethod + def _proposal_moves_descriptor() -> StringDescriptor: + """Create an emcee proposal-moves descriptor.""" + return StringDescriptor( + name='proposal_moves', + description='Single emcee proposal move; move mixtures are not persisted in v1.', + value_spec=AttributeSpec( + default=DEFAULT_PROPOSAL_MOVES, + validator=MembershipValidator(allowed=SUPPORTED_PROPOSAL_MOVES), + ), + cif_handler=CifHandler(names=['_minimizer.proposal_moves']), + ) + + @property + def proposal_moves(self) -> StringDescriptor: + """Single emcee proposal move.""" + return self._proposal_moves + + @proposal_moves.setter + def proposal_moves(self, value: str) -> None: + self._proposal_moves.value = value From 4f41db9855407adac89f521677f7301988a01178 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sun, 24 May 2026 22:10:03 +0200 Subject: [PATCH 12/65] Add EmceeMinimizer engine class --- docs/dev/plans/emcee-minimizer.md | 2 +- .../analysis/minimizers/__init__.py | 1 + .../analysis/minimizers/emcee.py | 902 ++++++++++++++++++ 3 files changed, 904 insertions(+), 1 deletion(-) create mode 100644 src/easydiffraction/analysis/minimizers/emcee.py diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index 8ab321dc9..7de1b22ab 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -317,7 +317,7 @@ Mark `[x]` as each step lands. Commit: `Add EmceeMinimizer category class` -- [ ] **P1.4 — Add `EmceeMinimizer` engine class.** New file +- [x] **P1.4 — Add `EmceeMinimizer` engine class.** New file `src/easydiffraction/analysis/minimizers/emcee.py`. The engine class is registered with `MinimizerFactory` and holds the `emcee.EnsembleSampler` plus an `HDFBackend` attribute. Mirror the diff --git a/src/easydiffraction/analysis/minimizers/__init__.py b/src/easydiffraction/analysis/minimizers/__init__.py index 1006eefb7..cd3a1c144 100644 --- a/src/easydiffraction/analysis/minimizers/__init__.py +++ b/src/easydiffraction/analysis/minimizers/__init__.py @@ -7,6 +7,7 @@ from easydiffraction.analysis.minimizers.bumps_dream import BumpsDreamMinimizer from easydiffraction.analysis.minimizers.bumps_lm import BumpsLmMinimizer from easydiffraction.analysis.minimizers.dfols import DfolsMinimizer +from easydiffraction.analysis.minimizers.emcee import EmceeMinimizer from easydiffraction.analysis.minimizers.enums import DreamPopulationInitializationEnum from easydiffraction.analysis.minimizers.lmfit import LmfitMinimizer from easydiffraction.analysis.minimizers.lmfit_least_squares import LmfitLeastSquaresMinimizer diff --git a/src/easydiffraction/analysis/minimizers/emcee.py b/src/easydiffraction/analysis/minimizers/emcee.py new file mode 100644 index 000000000..ade6868c4 --- /dev/null +++ b/src/easydiffraction/analysis/minimizers/emcee.py @@ -0,0 +1,902 @@ +# SPDX-FileCopyrightText: 2026 EasyScience contributors +# SPDX-License-Identifier: BSD-3-Clause +"""Minimizer using the emcee ensemble sampler.""" + +from __future__ import annotations + +import multiprocessing +import os +import pickle # noqa: S403 - used only to test whether multiprocessing can serialize a callable. +from collections.abc import Callable +from pathlib import Path + +import emcee +import numpy as np +from scipy.optimize import OptimizeResult + +from easydiffraction.analysis.fit_helpers.bayesian import BayesianFitResults +from easydiffraction.analysis.fit_helpers.bayesian import PosteriorSamples +from easydiffraction.analysis.fit_helpers.bayesian import compute_convergence_diagnostics +from easydiffraction.analysis.fit_helpers.bayesian import standard_deviations_from_summaries +from easydiffraction.analysis.fit_helpers.bayesian import summarize_posterior_parameters +from easydiffraction.analysis.fit_helpers.tracking import SamplerProgressUpdate +from easydiffraction.analysis.minimizers.base import MinimizerBase +from easydiffraction.analysis.minimizers.enums import InitializationMethodEnum +from easydiffraction.analysis.minimizers.enums import MinimizerTypeEnum +from easydiffraction.analysis.minimizers.factory import MinimizerFactory +from easydiffraction.core.metadata import TypeInfo +from easydiffraction.utils.enums import VerbosityEnum + +DEFAULT_METHOD = 'stretch' +DEFAULT_NSTEPS = 5000 +DEFAULT_NBURN = 1000 +DEFAULT_THIN = 5 +DEFAULT_NWALKERS = 32 +DEFAULT_PARALLEL_WORKERS = 0 +DEFAULT_INITIALIZATION_METHOD = InitializationMethodEnum.BALL +DEFAULT_PROPOSAL_MOVES = 'stretch' +MAX_RANDOM_SEED = int(np.iinfo(np.uint32).max) +EMCEE_CHAIN_GROUP = 'emcee_chain' +EMCEE_FAILURES = (ArithmeticError, RuntimeError, TypeError, ValueError) + + +@MinimizerFactory.register +class EmceeMinimizer(MinimizerBase): + """emcee affine-invariant ensemble Bayesian sampler.""" + + type_info = TypeInfo( + tag=MinimizerTypeEnum.EMCEE, + description='emcee affine-invariant ensemble Bayesian sampling', + ) + + _sidecar_path: Path | None = None + + def __init__( + self, + name: str = MinimizerTypeEnum.EMCEE, + method: str = DEFAULT_METHOD, + max_iterations: int = DEFAULT_NSTEPS, + ) -> None: + super().__init__( + name=name, + method=method, + max_iterations=max_iterations, + ) + self._nburn: int = DEFAULT_NBURN + self._thin: int = DEFAULT_THIN + self._nwalkers: int = DEFAULT_NWALKERS + self._parallel_workers: int = DEFAULT_PARALLEL_WORKERS + self._initialization_method: InitializationMethodEnum = DEFAULT_INITIALIZATION_METHOD + self._proposal_moves: str = DEFAULT_PROPOSAL_MOVES + self._sampler: emcee.EnsembleSampler | None = None + self._backend: emcee.backends.HDFBackend | None = None + + @property + def nsteps(self) -> int: + """Number of emcee steps to run per walker.""" + return self._validated_positive_integer('nsteps', self._max_iterations) + + @nsteps.setter + def nsteps(self, value: int) -> None: + self._max_iterations = self._validated_positive_integer('nsteps', value) + + @property + def nburn(self) -> int: + """Number of initial emcee steps discarded as burn-in.""" + return self._nburn + + @nburn.setter + def nburn(self, value: int) -> None: + self._nburn = self._validated_non_negative_integer('nburn', value) + + @property + def thin(self) -> int: + """emcee thinning interval.""" + return self._thin + + @thin.setter + def thin(self, value: int) -> None: + self._thin = self._validated_positive_integer('thin', value) + + @property + def nwalkers(self) -> int: + """Number of emcee walkers.""" + return self._nwalkers + + @nwalkers.setter + def nwalkers(self, value: int) -> None: + self._nwalkers = self._validated_positive_integer('nwalkers', value) + + @property + def parallel_workers(self) -> int: + """Worker count; ``0`` asks for all CPUs and ``1`` runs serially.""" + return self._parallel_workers + + @parallel_workers.setter + def parallel_workers(self, value: int) -> None: + self._parallel_workers = self._validated_non_negative_integer('parallel_workers', value) + + @property + def initialization_method(self) -> InitializationMethodEnum: + """emcee walker initialization method.""" + return self._initialization_method + + @initialization_method.setter + def initialization_method(self, value: InitializationMethodEnum | str) -> None: + self._initialization_method = self._validated_initialization_method(value) + + @property + def proposal_moves(self) -> str: + """emcee proposal move name.""" + return self._proposal_moves + + @proposal_moves.setter + def proposal_moves(self, value: str) -> None: + self._proposal_moves = self._validated_proposal_moves(value) + + def fit( + self, + parameters: list[object], + objective_function: Callable[..., object], + verbosity: VerbosityEnum = VerbosityEnum.FULL, + *, + finalize_tracking: bool = True, + use_physical_limits: bool = False, + random_seed: int | None = None, + resume: bool = False, + extra_steps: int | None = None, + ) -> BayesianFitResults: + """ + Run emcee sampling and return Bayesian fit results. + """ + if use_physical_limits: + self._apply_physical_limits(parameters) + + resolved_random_seed = self._resolve_random_seed(random_seed) + minimizer_name = self.name or 'emcee' + self._start_tracking(minimizer_name, verbosity=verbosity) + + try: + solver_args = self._prepare_solver_args(parameters) + solver_args['random_seed'] = resolved_random_seed + solver_args['resume'] = resume + solver_args['extra_steps'] = extra_steps + raw_result = self._run_solver(objective_function, **solver_args) + return self._finalize_fit(parameters, raw_result) + finally: + if finalize_tracking: + self._stop_tracking() + + @staticmethod + def _tracking_mode() -> str: + """Use sampler-style progress reporting for emcee runs.""" + return 'sampling' + + def _resolve_random_seed(self, random_seed: int | None) -> int: + """ + Return a user-provided or generated random seed. + """ + if random_seed is None: + generator = np.random.default_rng() + random_seed = int(generator.integers(0, np.iinfo(np.int32).max)) + + integer_seed = self._validated_random_seed_value(random_seed) + self._resolved_random_seed = integer_seed + return self._resolved_random_seed + + @staticmethod + def _validated_random_seed_value(random_seed: object) -> int: + """Validate and normalize an emcee random seed.""" + if isinstance(random_seed, bool): + msg = f'emcee random_seed must be an integer between 0 and {MAX_RANDOM_SEED}.' + raise TypeError(msg) + + integer_seed = int(random_seed) + if integer_seed != random_seed or integer_seed < 0 or integer_seed > MAX_RANDOM_SEED: + msg = f'emcee random_seed must be an integer between 0 and {MAX_RANDOM_SEED}.' + raise ValueError(msg) + return integer_seed + + def _prepare_solver_args( + self, + parameters: list[object], + ) -> dict[str, object]: + """ + Prepare emcee solver arguments in EasyDiffraction order. + """ + self._validate_sampled_parameter_bounds(parameters) + return { + 'parameters': parameters, + 'parameter_names': [parameter.unique_name for parameter in parameters], + 'parameter_display_names': [ + getattr(parameter, 'name', parameter.unique_name) for parameter in parameters + ], + 'starting_values': np.array( + [parameter.value for parameter in parameters], + dtype=float, + ), + 'starting_uncertainties': [parameter.uncertainty for parameter in parameters], + } + + @classmethod + def _validate_sampled_parameter_bounds( + cls, + parameters: list[object], + ) -> None: + """Validate finite ordered bounds for sampled emcee parameters.""" + issues: list[str] = [] + for parameter in parameters: + parameter_name = cls._parameter_name_for_bound_validation(parameter) + parameter_issues = cls._parameter_bound_issues(parameter) + if parameter_issues: + issues.append(f'- {parameter_name}: {"; ".join(parameter_issues)}') + + if not issues: + return + + message = 'emcee requires finite valid bounds for every sampled parameter:\n' + '\n'.join( + issues + ) + raise ValueError(message) + + @staticmethod + def _parameter_name_for_bound_validation(parameter: object) -> str: + """Return the user-facing name for emcee bound validation.""" + unique_name = getattr(parameter, 'unique_name', None) + if unique_name: + return str(unique_name) + + parameter_name = getattr(parameter, 'name', None) + if parameter_name: + return str(parameter_name) + return '' + + @classmethod + def _parameter_bound_issues( + cls, + parameter: object, + ) -> list[str]: + """Return bound-validation issues for one sampled parameter.""" + lower_bound = getattr(parameter, 'fit_min', None) + upper_bound = getattr(parameter, 'fit_max', None) + value = getattr(parameter, 'value', None) + issues: list[str] = [] + + lower_is_finite = cls._is_finite_bound_value(lower_bound) + upper_is_finite = cls._is_finite_bound_value(upper_bound) + value_is_finite = cls._is_finite_bound_value(value) + + if not lower_is_finite: + issues.append(f'fit_min must be finite (got {lower_bound!r})') + if not upper_is_finite: + issues.append(f'fit_max must be finite (got {upper_bound!r})') + + bounds_are_ordered = lower_is_finite and upper_is_finite and lower_bound < upper_bound + if lower_is_finite and upper_is_finite and not bounds_are_ordered: + issues.append(f'fit_min ({lower_bound}) must be smaller than fit_max ({upper_bound})') + + if not value_is_finite: + issues.append(f'starting value must be finite (got {value!r})') + elif bounds_are_ordered and not lower_bound <= value <= upper_bound: + issues.append(f'starting value {value} is outside [{lower_bound}, {upper_bound}]') + + return issues + + @staticmethod + def _is_finite_bound_value(value: object) -> bool: + """Return whether a bound-validation value is finite.""" + try: + return bool(np.isfinite(value)) + except TypeError: + return False + + @staticmethod + def _validated_positive_integer(name: str, value: float) -> int: + """Validate an emcee setting that must be a positive integer.""" + if isinstance(value, bool): + msg = f"emcee setting '{name}' must be a positive integer." + raise TypeError(msg) + + try: + integer_value = int(value) + except (TypeError, ValueError): + msg = f"emcee setting '{name}' must be a positive integer." + raise TypeError(msg) from None + if integer_value != value or integer_value < 1: + msg = f"emcee setting '{name}' must be a positive integer." + raise ValueError(msg) + return integer_value + + @staticmethod + def _validated_non_negative_integer(name: str, value: float) -> int: + """Validate an emcee setting that must be a non-negative integer.""" + if isinstance(value, bool): + msg = f"emcee setting '{name}' must be a non-negative integer." + raise TypeError(msg) + + try: + integer_value = int(value) + except (TypeError, ValueError): + msg = f"emcee setting '{name}' must be a non-negative integer." + raise TypeError(msg) from None + if integer_value != value or integer_value < 0: + msg = f"emcee setting '{name}' must be a non-negative integer." + raise ValueError(msg) + return integer_value + + @staticmethod + def _validated_initialization_method( + value: InitializationMethodEnum | str, + ) -> InitializationMethodEnum: + """Validate an emcee initialization method.""" + try: + method = InitializationMethodEnum(value) + except ValueError: + valid_values = ', '.join( + initialization.value + for initialization in ( + InitializationMethodEnum.BALL, + InitializationMethodEnum.UNIFORM, + InitializationMethodEnum.PRIOR, + ) + ) + msg = f"emcee setting 'initialization_method' must be one of: {valid_values}." + raise ValueError(msg) from None + + if method not in ( + InitializationMethodEnum.BALL, + InitializationMethodEnum.UNIFORM, + InitializationMethodEnum.PRIOR, + ): + valid_values = ', '.join( + initialization.value + for initialization in ( + InitializationMethodEnum.BALL, + InitializationMethodEnum.UNIFORM, + InitializationMethodEnum.PRIOR, + ) + ) + msg = f"emcee setting 'initialization_method' must be one of: {valid_values}." + raise ValueError(msg) + return method + + @staticmethod + def _validated_proposal_moves(value: str) -> str: + """Validate an emcee proposal move name.""" + valid_values = ('stretch', 'de', 'de_snooker', 'walk') + if value not in valid_values: + choices = ', '.join(valid_values) + msg = f"emcee setting 'proposal_moves' must be one of: {choices}." + raise ValueError(msg) + return value + + def _run_solver( + self, + objective_function: Callable[[dict[str, object]], object], + **kwargs: object, + ) -> object: + """ + Run emcee and normalize its posterior outputs. + """ + parameters = list(kwargs['parameters']) + parameter_names = list(kwargs['parameter_names']) + random_seed = int(kwargs['random_seed']) + resume = bool(kwargs.get('resume', False)) + extra_steps = kwargs.get('extra_steps') + + self._validate_walker_count(n_parameters=len(parameter_names)) + sidecar_path = self._resolved_sidecar_path() + sidecar_path.parent.mkdir(parents=True, exist_ok=True) + + total_iterations = self._resolved_total_iterations( + resume=resume, + extra_steps=extra_steps, + ) + self.tracker.start_sampler_pre_processing(total_iterations=total_iterations) + + backend = emcee.backends.HDFBackend( + str(sidecar_path), + name=EMCEE_CHAIN_GROUP, + read_only=False, + ) + self._backend = backend + + log_prob = self._build_log_probability( + parameters=parameters, + parameter_names=parameter_names, + objective_function=objective_function, + ) + pool = self._build_pool(log_prob) + try: + if resume: + self._validate_resume( + backend=backend, + n_parameters=len(parameter_names), + extra_steps=extra_steps, + ) + else: + backend.reset(self.nwalkers, len(parameter_names)) + + sampler = emcee.EnsembleSampler( + nwalkers=self.nwalkers, + ndim=len(parameter_names), + log_prob_fn=log_prob, + pool=pool, + moves=self._resolve_moves(self.proposal_moves), + backend=backend, + ) + self._sampler = sampler + + if resume: + sampler.run_mcmc( + None, + nsteps=int(extra_steps), + skip_initial_state_check=True, + progress=False, + ) + else: + initial_state = self._initial_state(parameters, random_seed=random_seed) + sampler.run_mcmc(initial_state, nsteps=self.nsteps, progress=False) + except EMCEE_FAILURES as error: + return self._failure_result( + message=f'emcee sampling failed: {error}', + starting_values=kwargs['starting_values'], + starting_uncertainties=kwargs['starting_uncertainties'], + sampler_settings=self._sampler_settings( + random_seed=random_seed, + total_steps=self._backend_iteration(backend), + n_parameters=len(parameter_names), + ), + raw_state=getattr(self._sampler, 'random_state', None), + sampler_completed=False, + ) + finally: + if pool is not None: + pool.close() + pool.join() + + self.tracker.start_sampler_post_processing() + return self._build_success_result( + sampler=sampler, + backend=backend, + parameter_names=parameter_names, + parameter_display_names=list(kwargs['parameter_display_names']), + random_seed=random_seed, + starting_values=kwargs['starting_values'], + starting_uncertainties=kwargs['starting_uncertainties'], + ) + + @staticmethod + def _backend_iteration(backend: object) -> int: + """Return backend iteration count, or zero when unavailable.""" + try: + return int(getattr(backend, 'iteration', 0)) + except (AttributeError, TypeError, ValueError): + return 0 + + @staticmethod + def _build_log_probability( + *, + parameters: list[object], + parameter_names: list[str], + objective_function: Callable[[dict[str, object]], object], + ) -> Callable[[np.ndarray], float]: + """Return an emcee log-probability adapter.""" + bounds = { + name: (float(parameter.fit_min), float(parameter.fit_max)) + for name, parameter in zip(parameter_names, parameters, strict=True) + } + + def log_prob(theta: np.ndarray) -> float: + for name, value in zip(parameter_names, theta, strict=True): + lower_bound, upper_bound = bounds[name] + if not lower_bound <= float(value) <= upper_bound: + return -np.inf + + engine_params = { + name: float(value) for name, value in zip(parameter_names, theta, strict=True) + } + try: + residuals = np.asarray(objective_function(engine_params), dtype=float) + except Exception: # noqa: BLE001 - calculator failures make this proposal invalid. + return -np.inf + if residuals.size == 0 or not np.all(np.isfinite(residuals)): + return -np.inf + return -0.5 * float(np.sum(residuals**2)) + + return log_prob + + def _resolved_sidecar_path(self) -> Path: + """Return the HDF sidecar path required by the emcee backend.""" + if self._sidecar_path is None: + msg = 'emcee engine requires Fitter.fit to set _sidecar_path; was Analysis configured?' + raise RuntimeError(msg) + return Path(self._sidecar_path) + + def _resolved_total_iterations( + self, + *, + resume: bool, + extra_steps: object, + ) -> int: + """Return the total iterations expected for progress display.""" + if not resume: + return self.nsteps + return self._validated_positive_integer('extra_steps', extra_steps) + + def _validate_walker_count(self, *, n_parameters: int) -> None: + """Validate emcee's minimum walker count for red-blue moves.""" + if n_parameters < 1: + msg = 'emcee requires at least one sampled parameter.' + raise ValueError(msg) + minimum_walkers = 2 * n_parameters + if self.nwalkers < minimum_walkers: + msg = ( + f"emcee setting 'nwalkers' must be at least twice the sampled " + f'parameter count ({minimum_walkers}).' + ) + raise ValueError(msg) + + def _validate_resume( + self, + *, + backend: object, + n_parameters: int, + extra_steps: object, + ) -> None: + """Validate that an existing emcee backend can be resumed.""" + self._validated_positive_integer('extra_steps', extra_steps) + iteration = self._backend_iteration(backend) + if iteration < 1: + msg = 'No existing emcee chain was found; start a fresh run instead.' + raise ValueError(msg) + + backend_shape = getattr(backend, 'shape', None) + if backend_shape != (self.nwalkers, n_parameters): + msg = ( + 'Existing emcee chain shape does not match current parameters; ' + 'start a fresh run.' + ) + raise ValueError(msg) + + def _build_pool(self, log_prob: Callable[[np.ndarray], float]) -> object | None: + """Build an emcee map pool for picklable objectives.""" + workers = self.parallel_workers + if workers == 1: + return None + + try: + pickle.dumps(log_prob) # noqa: S301 - no untrusted data is deserialized. + except (AttributeError, TypeError, pickle.PickleError): + self._warn_after_tracking( + 'emcee parallel evaluation requires a picklable objective; ' + 'falling back to serial execution.' + ) + return None + + worker_count = os.cpu_count() if workers == 0 else workers + if worker_count is None or worker_count <= 1: + return None + return multiprocessing.Pool(worker_count) + + @staticmethod + def _resolve_moves(proposal_moves: str) -> object: + """Return the emcee move object for a persisted move name.""" + if proposal_moves == 'stretch': + return emcee.moves.StretchMove() + if proposal_moves == 'de': + return emcee.moves.DEMove() + if proposal_moves == 'de_snooker': + return emcee.moves.DESnookerMove() + if proposal_moves == 'walk': + return emcee.moves.WalkMove() + msg = f"Unsupported emcee proposal move '{proposal_moves}'." + raise ValueError(msg) + + def _initial_state( + self, + parameters: list[object], + *, + random_seed: int, + ) -> np.ndarray: + """Build an initial walker state for emcee.""" + rng = np.random.default_rng(random_seed) + lower = np.array([float(parameter.fit_min) for parameter in parameters], dtype=float) + upper = np.array([float(parameter.fit_max) for parameter in parameters], dtype=float) + center = np.array([float(parameter.value) for parameter in parameters], dtype=float) + + if self.initialization_method == InitializationMethodEnum.BALL: + scale = np.maximum((upper - lower) * 1.0e-4, np.finfo(float).eps) + initial = center + rng.normal(scale=scale, size=(self.nwalkers, len(parameters))) + return np.clip(initial, lower, upper) + + return rng.uniform(lower, upper, size=(self.nwalkers, len(parameters))) + + def _sampler_settings( + self, + *, + random_seed: int, + total_steps: int, + n_parameters: int, + ) -> dict[str, object]: + """Build sampler settings recorded in results.""" + samples = total_steps * self.nwalkers * n_parameters + return { + 'random_seed': int(random_seed), + 'steps': int(total_steps), + 'burn': int(self.nburn), + 'thin': int(self.thin), + 'pop': int(self.nwalkers), + 'parallel': int(self.parallel_workers), + 'init': self.initialization_method.value, + 'proposal_moves': self.proposal_moves, + 'samples': int(samples), + 'nsteps': int(total_steps), + 'nburn': int(self.nburn), + 'nwalkers': int(self.nwalkers), + 'parallel_workers': int(self.parallel_workers), + 'initialization_method': self.initialization_method.value, + } + + @staticmethod + def _failure_result( + *, + message: str, + starting_values: object, + starting_uncertainties: object, + sampler_settings: dict[str, object], + raw_state: object, + sampler_completed: bool, + ) -> OptimizeResult: + """Build a normalized failure result for an incomplete emcee run.""" + return OptimizeResult( + x=np.asarray(starting_values, dtype=float), + dx=None, + fun=None, + success=False, + status=-1, + message=message, + var_names=[], + posterior_samples=None, + posterior_parameter_summaries=[], + convergence_diagnostics={}, + sampler_settings=sampler_settings, + sampler_completed=sampler_completed, + raw_state=raw_state, + best_log_posterior=None, + starting_values=np.asarray(starting_values, dtype=float), + starting_uncertainties=list(starting_uncertainties), + ) + + def _build_success_result( + self, + *, + sampler: emcee.EnsembleSampler, + backend: emcee.backends.HDFBackend, + parameter_names: list[str], + parameter_display_names: list[str], + random_seed: int, + starting_values: object, + starting_uncertainties: object, + ) -> OptimizeResult: + """Normalize a completed emcee run into an OptimizeResult.""" + total_steps = self._backend_iteration(backend) + discard = min(self.nburn, max(total_steps - 1, 0)) + chain = np.asarray(sampler.get_chain(discard=discard, thin=self.thin), dtype=float) + log_posterior = np.asarray( + sampler.get_log_prob(discard=discard, thin=self.thin), + dtype=float, + ) + sampler_settings = self._sampler_settings( + random_seed=random_seed, + total_steps=total_steps, + n_parameters=len(parameter_names), + ) + + if chain.ndim != 3 or chain.size == 0 or log_posterior.shape != chain.shape[:2]: + return self._failure_result( + message='emcee sampling did not return usable posterior samples.', + starting_values=starting_values, + starting_uncertainties=starting_uncertainties, + sampler_settings=sampler_settings, + raw_state=sampler.get_last_sample(), + sampler_completed=True, + ) + + finite_log_posterior = np.where(np.isfinite(log_posterior), log_posterior, -np.inf) + if not np.any(np.isfinite(finite_log_posterior)): + return self._failure_result( + message='emcee sampling did not return any finite log-posterior values.', + starting_values=starting_values, + starting_uncertainties=starting_uncertainties, + sampler_settings=sampler_settings, + raw_state=sampler.get_last_sample(), + sampler_completed=True, + ) + + best_flat_index = int(np.argmax(finite_log_posterior)) + best_draw_index, best_walker_index = np.unravel_index( + best_flat_index, + finite_log_posterior.shape, + ) + best_sample_values = np.asarray(chain[best_draw_index, best_walker_index, :], dtype=float) + draw_index = np.arange(chain.shape[0], dtype=float) * self.thin + discard + 1 + posterior_samples = PosteriorSamples( + parameter_names=parameter_names, + parameter_samples=chain, + log_posterior=log_posterior, + draw_index=draw_index, + ) + convergence_diagnostics = self._convergence_diagnostics( + posterior_samples=posterior_samples, + sampler=sampler, + ) + posterior_parameter_summaries = summarize_posterior_parameters( + parameter_names=parameter_names, + posterior_samples=posterior_samples, + best_sample_values=best_sample_values, + parameter_display_names=parameter_display_names, + convergence_diagnostics=convergence_diagnostics, + ) + posterior_standard_deviations = standard_deviations_from_summaries( + posterior_parameter_summaries + ) + best_log_posterior = float(finite_log_posterior[best_draw_index, best_walker_index]) + self._track_sampler_completion( + total_steps=total_steps, + best_log_posterior=best_log_posterior, + ) + + return OptimizeResult( + x=best_sample_values, + dx=posterior_standard_deviations, + fun=-best_log_posterior, + success=True, + status=0, + message='emcee sampling completed', + var_names=parameter_names, + posterior_samples=posterior_samples, + posterior_parameter_summaries=posterior_parameter_summaries, + convergence_diagnostics=convergence_diagnostics, + sampler_settings=sampler_settings, + sampler_completed=True, + raw_state=sampler.get_last_sample(), + best_log_posterior=best_log_posterior, + starting_values=np.asarray(starting_values, dtype=float), + starting_uncertainties=list(starting_uncertainties), + ) + + def _convergence_diagnostics( + self, + *, + posterior_samples: PosteriorSamples, + sampler: emcee.EnsembleSampler, + ) -> dict[str, object]: + """Compute convergence diagnostics and add emcee acceptance rate.""" + try: + convergence_diagnostics = compute_convergence_diagnostics(posterior_samples) + except (TypeError, ValueError, RuntimeError) as error: + self._warn_after_tracking( + f'emcee convergence diagnostics could not be computed: {error}' + ) + convergence_diagnostics = { + 'converged': False, + 'r_hat_by_parameter': {}, + 'ess_bulk_by_parameter': {}, + 'max_r_hat': None, + 'min_ess_bulk': None, + 'n_draws': int(posterior_samples.parameter_samples.shape[0]), + 'n_chains': int(posterior_samples.parameter_samples.shape[1]), + 'n_parameters': len(posterior_samples.parameter_names), + } + + acceptance_fraction = np.asarray(sampler.acceptance_fraction, dtype=float) + finite_acceptance = acceptance_fraction[np.isfinite(acceptance_fraction)] + convergence_diagnostics['acceptance_rate_mean'] = ( + float(np.mean(finite_acceptance)) if finite_acceptance.size else None + ) + if not convergence_diagnostics.get('converged', True): + self._warn_after_tracking( + 'Convergence diagnostics indicate the posterior may be poorly mixed.' + ) + return convergence_diagnostics + + def _track_sampler_completion( + self, + *, + total_steps: int, + best_log_posterior: float, + ) -> None: + """Record one final sampler progress row.""" + reduced_chi2 = self.tracker.best_chi2 + if reduced_chi2 is None: + reduced_chi2 = np.nan + self.tracker.track_sampler_progress( + SamplerProgressUpdate( + iteration=max(1, total_steps), + total_iterations=max(1, total_steps), + phase='sampling', + progress_percent=100.0, + log_posterior=best_log_posterior, + reduced_chi2=float(reduced_chi2), + elapsed_time=self.tracker._current_elapsed_time(), + force_report=True, + ) + ) + + @staticmethod + def _sync_result_to_parameters( + parameters: list[object], + raw_result: object, + ) -> None: + """ + Sync proposed or best posterior values to live parameters. + """ + if isinstance(raw_result, dict): + for parameter in parameters: + value = raw_result.get(parameter.unique_name) + if value is None: + value = raw_result.get(getattr(parameter, '_minimizer_uid', '')) + if value is not None: + parameter._set_value_from_minimizer(float(value)) + return + + if hasattr(raw_result, 'x'): + if getattr(raw_result, 'success', False): + values = raw_result.x + uncertainties = getattr(raw_result, 'dx', None) + else: + values = getattr(raw_result, 'starting_values', raw_result.x) + uncertainties = getattr(raw_result, 'starting_uncertainties', None) + else: + values = raw_result + uncertainties = None + + if values is None: + return + + for index, parameter in enumerate(parameters): + parameter._set_value_from_minimizer(float(values[index])) + if uncertainties is None: + parameter.uncertainty = None + continue + + uncertainty = uncertainties[index] + parameter.uncertainty = None if uncertainty is None else float(uncertainty) + + def _build_fit_results( + self, + *, + parameters: list[object], + raw_result: object, + success: bool, + ) -> BayesianFitResults: + """ + Build the Bayesian fit result container. + """ + fit_results = BayesianFitResults( + success=success, + parameters=parameters, + reduced_chi_square=self.tracker.best_chi2, + engine_result=getattr(raw_result, 'raw_state', raw_result), + starting_parameters=parameters, + fitting_time=self.tracker.fitting_time, + sampler_name='emcee', + point_estimate_name='best_sample', + posterior_samples=getattr(raw_result, 'posterior_samples', None), + posterior_parameter_summaries=getattr(raw_result, 'posterior_parameter_summaries', []), + posterior_predictive={}, + credible_interval_levels=(0.68, 0.95), + sampler_settings=getattr(raw_result, 'sampler_settings', {}), + convergence_diagnostics=getattr(raw_result, 'convergence_diagnostics', {}), + sampler_completed=getattr(raw_result, 'sampler_completed', False), + best_log_posterior=getattr(raw_result, 'best_log_posterior', None), + ) + fit_results.message = getattr(raw_result, 'message', '') + fit_results.iterations = int(fit_results.sampler_settings.get('steps', self.nsteps)) + fit_results.result = raw_result + return fit_results + + def _check_success(self, raw_result: object) -> bool: + """Determine success from normalized emcee result.""" + return bool(getattr(raw_result, 'success', False)) From c03c5e4ca1a9ec65bd2219ffe87ff371be5f2f3a Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sun, 24 May 2026 22:15:52 +0200 Subject: [PATCH 13/65] Wire emcee resume through fit stack --- docs/dev/plans/emcee-minimizer.md | 2 +- src/easydiffraction/analysis/analysis.py | 125 ++++++++++++++++-- .../analysis/categories/minimizer/base.py | 4 + src/easydiffraction/analysis/fitting.py | 45 ++++++- .../analysis/minimizers/base.py | 13 ++ 5 files changed, 177 insertions(+), 12 deletions(-) diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index 7de1b22ab..cc1de119d 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -491,7 +491,7 @@ Mark `[x]` as each step lands. Commit: `Add EmceeMinimizer engine class` -- [ ] **P1.5 — Wire `fit(resume=True, extra_steps=N)` end-to-end.** The +- [x] **P1.5 — Wire `fit(resume=True, extra_steps=N)` end-to-end.** The current fit stack does not accept `resume` / `extra_steps` anywhere. Every signature and call site listed below must be updated in this step. Each item is one short edit; the step lands diff --git a/src/easydiffraction/analysis/analysis.py b/src/easydiffraction/analysis/analysis.py index 9a01d8385..795b1ce48 100644 --- a/src/easydiffraction/analysis/analysis.py +++ b/src/easydiffraction/analysis/analysis.py @@ -926,20 +926,74 @@ def _get_params_as_dataframe( df.columns = pd.MultiIndex.from_tuples(df.columns) return df - def fit(self) -> None: + def fit( + self, + *, + resume: bool = False, + extra_steps: int | None = None, + ) -> None: """Execute fitting for the currently selected fitting mode.""" mode = FitModeEnum(self._fitting_mode.type) + self._validate_fit_request( + mode=mode, + resume=resume, + extra_steps=extra_steps, + ) if mode is FitModeEnum.SINGLE: - self._run_single() + self._run_single(resume=resume, extra_steps=extra_steps) elif mode is FitModeEnum.JOINT: self._prepare_joint_fit() - self._run_joint() + self._run_joint(resume=resume, extra_steps=extra_steps) elif mode is FitModeEnum.SEQUENTIAL: self._run_sequential() else: # pragma: no cover msg = f'Unknown fit mode: {mode!r}' raise ValueError(msg) + def _validate_fit_request( + self, + *, + mode: FitModeEnum, + resume: bool, + extra_steps: int | None, + ) -> None: + """Validate fit options before dispatching to a fitting mode.""" + if extra_steps is not None and not resume: + msg = 'extra_steps is only valid when resume=True.' + raise ValueError(msg) + if resume and mode is not FitModeEnum.SINGLE: + msg = 'Resume is supported in single fit mode only.' + raise ValueError(msg) + + is_emcee = self.minimizer.type == MinimizerTypeEnum.EMCEE.value + if resume and not is_emcee: + msg = "Resume is supported only when analysis.minimizer.type = 'emcee'." + raise ValueError(msg) + if is_emcee and self.project.info.path is None: + msg = ( + 'emcee requires a saved project; call project.save_as() ' + 'before analysis.fit().' + ) + raise ValueError(msg) + if resume: + self._validate_resume_extra_steps(extra_steps) + + @staticmethod + def _validate_resume_extra_steps(extra_steps: int | None) -> None: + """Validate the emcee resume step count.""" + if extra_steps is None or isinstance(extra_steps, bool): + msg = 'extra_steps must be a positive integer when resume=True.' + raise ValueError(msg) + + try: + integer_steps = int(extra_steps) + except (TypeError, ValueError): + msg = 'extra_steps must be a positive integer when resume=True.' + raise ValueError(msg) from None + if integer_steps != extra_steps or integer_steps < 1: + msg = 'extra_steps must be a positive integer when resume=True.' + raise ValueError(msg) + def _warn_results_sidecar_overwrite(self) -> None: """Warn before persisted sidecar arrays are overwritten.""" project_path = self.project.info.path @@ -1134,8 +1188,9 @@ def _warn_about_minimizer_swap_defaults( def _sync_engine_from_minimizer_category(self) -> None: """Apply minimizer category settings to the live engine.""" engine = self.fitter.minimizer + skip_keys = type(self.minimizer)._engine_sync_skip_keys for key, value in self.minimizer._native_kwargs().items(): - if key == 'random_seed': + if key in skip_keys: continue if not hasattr(engine, key): log.warning( @@ -1766,7 +1821,11 @@ def _resolve_sequential_data_dir(self) -> Path: return project_path / data_dir - def _prepare_fit_run(self) -> tuple[VerbosityEnum, object, object] | None: + def _prepare_fit_run( + self, + *, + resume: bool = False, + ) -> tuple[VerbosityEnum, object, object] | None: """Resolve common inputs for single and joint fitting.""" verb = VerbosityEnum(self.project.verbosity.fit.value) structures = self.project.structures @@ -1779,7 +1838,8 @@ def _prepare_fit_run(self) -> tuple[VerbosityEnum, object, object] | None: log.warning('No experiments found in the project. Cannot run fit.') return None - self._warn_results_sidecar_overwrite() + if not resume: + self._warn_results_sidecar_overwrite() # Apply constraints before fitting so that user-constrained # parameters are marked and excluded from the free parameter @@ -1789,11 +1849,16 @@ def _prepare_fit_run(self) -> tuple[VerbosityEnum, object, object] | None: return verb, structures, experiments - def _run_single(self) -> None: + def _run_single( + self, + *, + resume: bool = False, + extra_steps: int | None = None, + ) -> None: """ Execute single-mode fitting with current project verbosity. """ - prepared = self._prepare_fit_run() + prepared = self._prepare_fit_run(resume=resume) if prepared is None: return @@ -1804,14 +1869,25 @@ def _run_single(self) -> None: experiments, use_physical_limits=False, random_seed=None, + resume=resume, + extra_steps=extra_steps, ) if self.project.info.path is not None: self.project.save() - def _run_joint(self) -> None: + def _run_joint( + self, + *, + resume: bool = False, + extra_steps: int | None = None, + ) -> None: """Execute joint-mode fitting with current project verbosity.""" - prepared = self._prepare_fit_run() + if resume: + msg = 'Resume is supported in single fit mode only.' + raise ValueError(msg) + + prepared = self._prepare_fit_run(resume=resume) if prepared is None: return @@ -1822,6 +1898,8 @@ def _run_joint(self) -> None: experiments, use_physical_limits=False, random_seed=None, + resume=resume, + extra_steps=extra_steps, ) if self.project.info.path is not None: @@ -1872,6 +1950,8 @@ def _fit_joint( *, use_physical_limits: bool, random_seed: int | None, + resume: bool = False, + extra_steps: int | None = None, ) -> None: """ Run joint fitting across all experiments with weights. @@ -1888,7 +1968,15 @@ def _fit_joint( Whether to use physical limits as fit bounds. random_seed : int | None Optional random seed passed to stochastic minimizers. + resume : bool, default=False + Whether to resume a sampler state. + extra_steps : int | None, default=None + Additional sampler steps for resume-capable minimizers. """ + if resume: + msg = 'Resume is supported in single fit mode only.' + raise ValueError(msg) + mode = FitModeEnum.JOINT # Auto-populate joint_fit if empty if not len(self._joint_fit): @@ -1910,6 +1998,8 @@ def _fit_joint( verbosity=verb, use_physical_limits=use_physical_limits, random_seed=self._resolved_fit_random_seed(random_seed), + resume=resume, + extra_steps=extra_steps, ) # After fitting, get the results @@ -1923,6 +2013,8 @@ def _fit_single( *, use_physical_limits: bool, random_seed: int | None, + resume: bool = False, + extra_steps: int | None = None, ) -> None: """ Run single-mode fitting for each experiment independently. @@ -1939,9 +2031,16 @@ def _fit_single( Whether to use physical limits as fit bounds. random_seed : int | None Optional random seed passed to stochastic minimizers. + resume : bool, default=False + Whether to resume a sampler state. + extra_steps : int | None, default=None + Additional sampler steps for resume-capable minimizers. """ mode = FitModeEnum.SINGLE expt_names = experiments.names + if resume and len(expt_names) != 1: + msg = 'Resume is supported for one single-fit experiment at a time.' + raise ValueError(msg) short_display_handle = self._fit_single_print_header(verb, expt_names, mode) short_rows: list[list[str]] = [] @@ -1954,6 +2053,8 @@ def _fit_single( experiments, use_physical_limits=use_physical_limits, random_seed=random_seed, + resume=resume, + extra_steps=extra_steps, short_state=(short_rows, short_display_handle), ) finally: @@ -1972,6 +2073,8 @@ def _fit_single_experiments( *, use_physical_limits: bool, random_seed: int | None, + resume: bool, + extra_steps: int | None, short_state: tuple[list[list[str]], object], ) -> None: """Run the per-experiment loop for single-fit mode.""" @@ -1991,6 +2094,8 @@ def _fit_single_experiments( verbosity=verb, use_physical_limits=use_physical_limits, random_seed=self._resolved_fit_random_seed(random_seed), + resume=resume, + extra_steps=extra_steps, ) results = self.fitter.results diff --git a/src/easydiffraction/analysis/categories/minimizer/base.py b/src/easydiffraction/analysis/categories/minimizer/base.py index fb583964d..a57696ef4 100644 --- a/src/easydiffraction/analysis/categories/minimizer/base.py +++ b/src/easydiffraction/analysis/categories/minimizer/base.py @@ -24,6 +24,10 @@ class MinimizerCategoryBase(CategoryItem, SwitchableCategoryBase): _owner_attr_name = 'minimizer' _swap_method_name = '_swap_minimizer' _native_key_map: ClassVar[dict[str, str]] = {} + _engine_sync_skip_keys: ClassVar[frozenset[str]] = frozenset({ + 'random_seed', + 'parallel_workers', + }) _setting_descriptor_names: ClassVar[tuple[str, ...]] = () _result_descriptor_names: ClassVar[tuple[str, ...]] = () _fit_result_class: ClassVar[type[FitResultBase]] = FitResultBase diff --git a/src/easydiffraction/analysis/fitting.py b/src/easydiffraction/analysis/fitting.py index c7c79917f..80a024965 100644 --- a/src/easydiffraction/analysis/fitting.py +++ b/src/easydiffraction/analysis/fitting.py @@ -147,6 +147,8 @@ def fit( *, use_physical_limits: bool = False, random_seed: int | None = None, + resume: bool = False, + extra_steps: int | None = None, ) -> None: """ Run the fitting process. @@ -175,6 +177,10 @@ def fit( unbounded. random_seed : int | None, default=None Optional random seed passed to stochastic minimizers. + resume : bool, default=False + Whether to resume a sampler state instead of starting a new fit. + extra_steps : int | None, default=None + Additional sampler steps for resume-capable minimizers. """ # Enforce symmetry constraints (e.g. ADP) before collecting # free parameters so that components fixed by site symmetry are @@ -186,6 +192,9 @@ def fit( params = self._collect_fit_parameters(structures, experiments) if not params: + if resume: + msg = 'Resume requires the same free parameters used by the saved emcee chain.' + raise ValueError(msg) if analysis is not None: analysis._clear_persisted_fit_state() analysis.fit_results = None @@ -193,8 +202,10 @@ def fit( print('⚠️ No parameters selected for fitting.') return - if analysis is not None: + if analysis is not None and not resume: analysis._capture_fit_parameter_state(params) + if analysis is not None and resume: + self._validate_resume_parameter_set(params=params, analysis=analysis) for param in params: param._fit_start_value = param.value @@ -207,6 +218,8 @@ def fit( analysis=analysis, ) + self._set_minimizer_sidecar_path(analysis) + try: # Keep tracker finalization in this layer so post-processing # can run before the live display is closed. @@ -217,6 +230,8 @@ def fit( finalize_tracking=False, use_physical_limits=use_physical_limits, random_seed=random_seed, + resume=resume, + extra_steps=extra_steps, ) # Stop the timer and backfill results.fitting_time now so # post-processing projects a real duration into persisted @@ -231,6 +246,34 @@ def fit( finally: self.minimizer._stop_tracking() + def _set_minimizer_sidecar_path(self, analysis: object) -> None: + """Set the analysis results sidecar path on engines that use it.""" + if analysis is None or not hasattr(self.minimizer, '_sidecar_path'): + return + + project_info = getattr(getattr(analysis, 'project', None), 'info', None) + project_path = getattr(project_info, 'path', None) + sidecar_path = None if project_path is None else project_path / 'analysis' / 'results.h5' + self.minimizer._sidecar_path = sidecar_path + + @staticmethod + def _validate_resume_parameter_set( + *, + params: list[Parameter], + analysis: object, + ) -> None: + """Ensure resume uses the same persisted free-parameter set.""" + persisted_names = [ + item.param_unique_name.value for item in getattr(analysis, 'fit_parameters', []) + ] + if not persisted_names: + return + + current_names = [param.unique_name for param in params] + if persisted_names != current_names: + msg = 'Resume parameter set differs from the saved emcee chain; start a fresh run.' + raise ValueError(msg) + def _process_fit_results( self, structures: Structures, diff --git a/src/easydiffraction/analysis/minimizers/base.py b/src/easydiffraction/analysis/minimizers/base.py index 438ed9eaa..a5ebb7020 100644 --- a/src/easydiffraction/analysis/minimizers/base.py +++ b/src/easydiffraction/analysis/minimizers/base.py @@ -357,6 +357,8 @@ def fit( finalize_tracking: bool = True, use_physical_limits: bool = False, random_seed: int | None = None, + resume: bool = False, + extra_steps: int | None = None, ) -> FitResults: """ Run the full minimization workflow. @@ -378,12 +380,23 @@ def fit( unbounded. random_seed : int | None, default=None Optional random seed passed to stochastic minimizers. + resume : bool, default=False + Whether to resume an existing sampler state. Unsupported by + the base minimizer implementation. + extra_steps : int | None, default=None + Additional sampler steps for resume-capable minimizers. Returns ------- FitResults FitResults with success flag, best chi2 and timing. """ + del extra_steps + if resume: + minimizer_name = self.name or self.__class__.__name__ + msg = f"Minimizer '{minimizer_name}' does not support resume." + raise NotImplementedError(msg) + if use_physical_limits: self._apply_physical_limits(parameters) From 1d3908aee734740770a6c6ed45780eebce27e889 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sun, 24 May 2026 22:17:22 +0200 Subject: [PATCH 14/65] Append-on-save plus explicit truncate-on-new-fit prep --- docs/dev/plans/emcee-minimizer.md | 2 +- src/easydiffraction/analysis/analysis.py | 12 ++++---- src/easydiffraction/io/results_sidecar.py | 37 +++++++++++++++++++++-- 3 files changed, 41 insertions(+), 10 deletions(-) diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index cc1de119d..e97f49eab 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -580,7 +580,7 @@ Mark `[x]` as each step lands. Commit: `Wire emcee resume through fit stack` -- [ ] **P1.5a — Make `results.h5` append-on-save and add an explicit +- [x] **P1.5a — Make `results.h5` append-on-save and add an explicit truncate-on-new-fit prep step.** Two coordinated changes that land in a single commit because they jointly preserve the ADR lifecycle (resume keeps `/emcee_chain`; new fit removes it). diff --git a/src/easydiffraction/analysis/analysis.py b/src/easydiffraction/analysis/analysis.py index 795b1ce48..67987d4b8 100644 --- a/src/easydiffraction/analysis/analysis.py +++ b/src/easydiffraction/analysis/analysis.py @@ -994,17 +994,17 @@ def _validate_resume_extra_steps(extra_steps: int | None) -> None: msg = 'extra_steps must be a positive integer when resume=True.' raise ValueError(msg) - def _warn_results_sidecar_overwrite(self) -> None: - """Warn before persisted sidecar arrays are overwritten.""" + def _prepare_results_sidecar_for_new_fit(self) -> None: + """Warn and remove persisted sidecar arrays before a fresh fit.""" project_path = self.project.info.path if project_path is None: return from easydiffraction.io.results_sidecar import ( # noqa: PLC0415 - warn_analysis_results_sidecar_overwrite, + prepare_analysis_results_sidecar_for_new_fit, ) - warn_analysis_results_sidecar_overwrite(analysis_dir=project_path / 'analysis') + prepare_analysis_results_sidecar_for_new_fit(analysis_dir=project_path / 'analysis') def _prepare_joint_fit(self) -> None: """ @@ -1839,7 +1839,7 @@ def _prepare_fit_run( return None if not resume: - self._warn_results_sidecar_overwrite() + self._prepare_results_sidecar_for_new_fit() # Apply constraints before fitting so that user-constrained # parameters are marked and excluded from the free parameter @@ -1913,7 +1913,7 @@ def _run_sequential(self) -> None: self._set_fitting_mode_type(FitModeEnum.SEQUENTIAL.value) self._update_categories() - self._warn_results_sidecar_overwrite() + self._prepare_results_sidecar_for_new_fit() self._clear_persisted_fit_state() max_workers_value = self._sequential_fit.max_workers.value diff --git a/src/easydiffraction/io/results_sidecar.py b/src/easydiffraction/io/results_sidecar.py index db7ed17cf..79abeceff 100644 --- a/src/easydiffraction/io/results_sidecar.py +++ b/src/easydiffraction/io/results_sidecar.py @@ -22,6 +22,12 @@ _DISTRIBUTION_CACHE_GROUP = '/distribution_cache' _PAIR_CACHE_GROUP = '/pair_cache' _PREDICTIVE_GROUP = '/predictive' +_CANONICAL_GROUPS = ( + 'posterior', + 'distribution_cache', + 'pair_cache', + 'predictive', +) _POSTERIOR_SAMPLE_NDIM = 3 @@ -54,9 +60,8 @@ def _delete_stale_sidecar(sidecar_path: Path) -> None: sidecar_path.unlink() -def warn_analysis_results_sidecar_overwrite(*, analysis_dir: Path) -> None: +def _warn_existing_sidecar_overwrite(sidecar_path: Path) -> None: """Warn when a new fit will overwrite existing sidecar arrays.""" - sidecar_path = _sidecar_path(analysis_dir=analysis_dir) if not sidecar_path.is_file() or sidecar_path.stat().st_size == 0: return @@ -66,6 +71,19 @@ def warn_analysis_results_sidecar_overwrite(*, analysis_dir: Path) -> None: ) +def warn_analysis_results_sidecar_overwrite(*, analysis_dir: Path) -> None: + """Warn when a new fit will overwrite existing sidecar arrays.""" + _warn_existing_sidecar_overwrite(_sidecar_path(analysis_dir=analysis_dir)) + + +def prepare_analysis_results_sidecar_for_new_fit(*, analysis_dir: Path) -> None: + """Warn and remove the results sidecar before a fresh fit starts.""" + sidecar_path = _sidecar_path(analysis_dir=analysis_dir) + _warn_existing_sidecar_overwrite(sidecar_path) + if sidecar_path.is_file(): + sidecar_path.unlink() + + def _create_dataset(handle: object, path: str, data: np.ndarray) -> None: """Create or replace one dataset in an open HDF5 file.""" normalized_path = _normalized_hdf5_path(path) @@ -76,6 +94,18 @@ def _create_dataset(handle: object, path: str, data: np.ndarray) -> None: group.create_dataset(dataset_name, data=data) +def _delete_group_if_present(handle: object, group_name: str) -> None: + """Delete one top-level group from an open HDF5 file when present.""" + if group_name in handle: + del handle[group_name] + + +def _delete_canonical_groups(handle: object) -> None: + """Delete EasyDiffraction-owned top-level groups before append writes.""" + for group_name in _CANONICAL_GROUPS: + _delete_group_if_present(handle, group_name) + + def _read_dataset(handle: object, path: str) -> np.ndarray | None: """Read one dataset from an open HDF5 file when it exists.""" normalized_path = _normalized_hdf5_path(path) @@ -299,7 +329,8 @@ def write_analysis_results_sidecar( import h5py # noqa: PLC0415 analysis_dir.mkdir(parents=True, exist_ok=True) - with h5py.File(sidecar_path, 'w') as handle: + with h5py.File(sidecar_path, 'a') as handle: + _delete_canonical_groups(handle) wrote_any = _write_posterior_payload(handle, analysis) wrote_any = _write_distribution_caches(handle, analysis) or wrote_any wrote_any = _write_pair_caches(handle, analysis) or wrote_any From 67a1b06a99ee9c838b3df257d406db0dcfae6175 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sun, 24 May 2026 22:22:46 +0200 Subject: [PATCH 15/65] Route emcee posterior through fit_result and sidecar --- docs/dev/issues/closed.md | 28 ++++++++ docs/dev/issues/open.md | 67 ------------------ docs/dev/plans/emcee-minimizer.md | 21 +++--- src/easydiffraction/analysis/analysis.py | 69 +++++++++++-------- .../analysis/fit_helpers/bayesian.py | 11 +++ src/easydiffraction/display/plotting.py | 16 +---- src/easydiffraction/project/display.py | 5 +- 7 files changed, 98 insertions(+), 119 deletions(-) diff --git a/docs/dev/issues/closed.md b/docs/dev/issues/closed.md index 4b46b130e..fbe4578c4 100644 --- a/docs/dev/issues/closed.md +++ b/docs/dev/issues/closed.md @@ -4,6 +4,34 @@ Issues that have been fully resolved. Kept for historical reference. --- +## 103. Make `_sync_engine_from_minimizer_category` Skip-Keys Declarative + +Closed by +[`emcee-minimizer.md`](../plans/emcee-minimizer.md). Minimizer +categories now declare `_engine_sync_skip_keys`, and analysis sync +filters against that set instead of hardcoding skipped keys. + +--- + +## 101. Remove Dead Branch in `_fit_state_categories` + +Closed by +[`emcee-minimizer.md`](../plans/emcee-minimizer.md). The deterministic +branch that returned the same category list as the fallthrough path was +removed while preserving unsupported `result_kind` warning behavior. + +--- + +## 100. Collapse Duplicate Predictive-Cache-Key Helpers + +Closed by +[`emcee-minimizer.md`](../plans/emcee-minimizer.md). +`posterior_predictive_cache_key()` in `analysis.fit_helpers.bayesian` +is now the single helper used by analysis, plotting, and project +display code. + +--- + ## 77. Add Help Methods to Public Discovery Facades Added consistent `help()` methods for plain user-facing facade classes diff --git a/docs/dev/issues/open.md b/docs/dev/issues/open.md index 1e0bfe340..bf13134ce 100644 --- a/docs/dev/issues/open.md +++ b/docs/dev/issues/open.md @@ -1668,52 +1668,6 @@ sampler progress displays — any fix should keep their visuals consistent --- -## 100. 🟢 Collapse Duplicate Predictive-Cache-Key Helpers - -**Type:** Refactor / drift risk **Source:** Review 8 finding F1. -**Recommended:** fold into the emcee-minimizer plan while the -surrounding code is being touched. - -`Analysis._predictive_cache_key` -([analysis.py:478-487](../../../src/easydiffraction/analysis/analysis.py)) -and `Plotter._posterior_predictive_key` -([plotting.py:3795-3804](../../../src/easydiffraction/display/plotting.py)) -both return `f'{name}:{x_axis_name}:{suffix}'`. The strings are -identical today; a future refactor that changes one will silently break -lookup against the other. - -**Fix:** collapse to a single helper — either move the canonical helper -to a shared module (e.g. `analysis/fit_helpers/bayesian.py`), or have -`Analysis._store_posterior_predictive_projection` and -`_restored_predictive_summaries` reuse -`Plotter._posterior_predictive_key` from `project.rendering.plotter` -(already accessed nearby). - -**Depends on:** nothing. - ---- - -## 101. 🟢 Remove Dead Branch in `_fit_state_categories` - -**Type:** Dead code **Source:** Review 8 finding F4. **Recommended:** -fold into the emcee-minimizer plan. - -`Analysis._fit_state_categories` -([analysis.py:1135-1148](../../../src/easydiffraction/analysis/analysis.py)) -has -`if result_kind is FitResultKindEnum.DETERMINISTIC: return categories` -followed by `return categories`. Both branches return the same list -since P1.10 absorbed Bayesian-only categories. - -**Fix:** simplify to an unconditional `return categories`. Keep the -preceding `try/except` for its warning side-effect; extract it so the -function body reads cleanly. If a future Bayesian-only category list is -expected, add a TODO instead. - -**Depends on:** nothing. - ---- - ## 102. 🟢 Drop Compute-and-Ignore `result_kind` Validation in CIF Restore **Type:** Dead code / clarity **Source:** Review 8 finding F7. @@ -1735,27 +1689,6 @@ ignore" pattern. --- -## 103. 🟢 Make `_sync_engine_from_minimizer_category` Skip-Keys Declarative - -**Type:** Refactor / discoverability **Source:** Review 8 finding F10. -**Recommended:** fold into the emcee-minimizer plan (it adds -`proposal_moves` which is also engine-level). - -`Analysis._sync_engine_from_minimizer_category` -([analysis.py:1077-1089](../../../src/easydiffraction/analysis/analysis.py)) -hardcodes `if key == 'random_seed': continue` to keep call-time seed -threading via `_resolved_fit_random_seed`. A second ambient key joining -it (emcee `proposal_moves`) will need the same treatment. - -**Fix:** declare -`_engine_sync_skip_keys: ClassVar[frozenset[str]] = frozenset({'random_seed'})` -on `MinimizerCategoryBase` (or per family) and filter against it. Adds -declarative, growing coverage. - -**Depends on:** nothing. - ---- - ## 104. 🟢 Tighten `FitParameterItem.posterior_summary` NaN Behaviour **Type:** Robustness / partial-data edge case **Source:** Review 8 diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index e97f49eab..96836b3ea 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -175,10 +175,10 @@ reconciliation), stop and ask before proceeding. ## Cleanup opportunities inherited from earlier work -The input/output-split work left four cleanup items still open in -[`docs/dev/issues/open.md`](../issues/open.md) that touch code this plan -modifies. Fold them in opportunistically while the surrounding code is -already being edited; the plan does not block on them. +The input/output-split work left four cleanup items open when this plan +was written. They touch code this plan modifies, so fold them in +opportunistically while the surrounding code is already being edited; +the plan does not block on them. - **#100 — Collapse duplicate predictive-cache-key helpers.** `Analysis._predictive_cache_key` @@ -187,17 +187,18 @@ already being edited; the plan does not block on them. ([plotting.py:3823](../../../src/easydiffraction/display/plotting.py)) build the identical string; keep one canonical helper. P1.6 may touch the predictive plotting path while validating emcee posterior - emission. + emission. Resolved in P1.6. - **#101 — Remove dead branch in `Analysis._fit_state_categories`.** Both branches return the same list since the Bayesian categories were absorbed. One-line fix at [analysis.py:1184-1205](../../../src/easydiffraction/analysis/analysis.py). + Resolved in P1.6. - **#102 — Drop compute-and-ignore `result_kind` validation.** `_restore_persisted_fit_state` ([serialize.py:590-606](../../../src/easydiffraction/io/cif/serialize.py)) calls `FitResultKindEnum(result_kind_value)` for its side effect only. Move the warning into the descriptor setter, or extract a validator - helper. + helper. Still open. - **#103 — Make `_sync_engine_from_minimizer_category` skip-keys declarative.** This plan adds `proposal_moves` as a second engine-level "ambient" key (alongside `random_seed`). The current @@ -206,7 +207,7 @@ already being edited; the plan does not block on them. should become a class-level `_engine_sync_skip_keys: ClassVar[frozenset[str]] = frozenset(...)` on `MinimizerCategoryBase` before the second member lands. Recommend - addressing as part of P1.5. + addressing as part of P1.5. Resolved in P1.5. When the matching open-issue is fully resolved, move it to [`closed.md`](../issues/closed.md) and update @@ -569,7 +570,7 @@ Mark `[x]` as each step lands. the `Analysis` flow — the `Analysis.fit` guard above fires first in normal use. - **Open issue #103 cleanup.** Introduce + **Issue #103 cleanup.** Introduce `_engine_sync_skip_keys: ClassVar[frozenset[str]] = frozenset({'random_seed', 'parallel_workers'})` on `MinimizerCategoryBase`, and update `_sync_engine_from_minimizer_category` @@ -631,7 +632,7 @@ Mark `[x]` as each step lands. Commit: `Append-on-save plus explicit truncate-on-new-fit prep` -- [ ] **P1.6 — Route emcee outputs into the existing fit_result and +- [x] **P1.6 — Route emcee outputs into the existing fit_result and sidecar pipeline.** Verify the existing `_store_posterior_fit_projection` ([analysis.py](../../../src/easydiffraction/analysis/analysis.py)) @@ -644,7 +645,7 @@ Mark `[x]` as each step lands. vs the DREAM extraction helper). Cache derivations (KDE, pair grids) reuse the existing pipeline. - Opportunistic cleanup: address open issue #100 if the + Opportunistic cleanup: address issue #100 if the predictive plotting path is being touched anyway. Collapse `Analysis._predictive_cache_key` and `Plotter._posterior_predictive_key` into one canonical helper. diff --git a/src/easydiffraction/analysis/analysis.py b/src/easydiffraction/analysis/analysis.py index 67987d4b8..04c620480 100644 --- a/src/easydiffraction/analysis/analysis.py +++ b/src/easydiffraction/analysis/analysis.py @@ -32,6 +32,7 @@ from easydiffraction.analysis.fit_helpers.bayesian import BayesianFitResults from easydiffraction.analysis.fit_helpers.bayesian import PosteriorPredictiveSummary from easydiffraction.analysis.fit_helpers.bayesian import PosteriorSamples +from easydiffraction.analysis.fit_helpers.bayesian import posterior_predictive_cache_key from easydiffraction.analysis.fit_helpers.reporting import FitResults from easydiffraction.analysis.fitting import Fitter from easydiffraction.analysis.minimizers.enums import MinimizerTypeEnum @@ -524,17 +525,6 @@ def _swap_fitting_mode(self, new_type: str) -> None: """Switch the active fitting-mode category.""" self._replace_fitting_mode(new_type, announce=True) - @staticmethod - def _predictive_cache_key( - experiment_name: str, - x_axis_name: str, - *, - include_draws: bool = True, - ) -> str: - """Return the runtime cache key for one predictive summary.""" - key_suffix = 'draws' if include_draws else 'band' - return f'{experiment_name}:{x_axis_name}:{key_suffix}' - def _live_parameter_map(self) -> dict[str, Parameter]: """Return live parameters keyed by unique name.""" all_parameters = self.project.structures.parameters + self.project.experiments.parameters @@ -668,7 +658,7 @@ def _restored_predictive_summaries(self) -> dict[str, PosteriorPredictiveSummary ) restored_predictive[experiment_name] = summary restored_predictive[ - self._predictive_cache_key( + posterior_predictive_cache_key( experiment_name, x_axis_name, include_draws=False, @@ -676,7 +666,7 @@ def _restored_predictive_summaries(self) -> dict[str, PosteriorPredictiveSummary ] = summary if summary.draws is not None: restored_predictive[ - self._predictive_cache_key( + posterior_predictive_cache_key( experiment_name, x_axis_name, include_draws=True, @@ -744,15 +734,7 @@ def _restore_fit_results_from_projection(self) -> object | None: float(self.fit_result.credible_interval_inner.value), float(self.fit_result.credible_interval_outer.value), ), - sampler_settings={ - 'steps': int(sampler_settings.get('steps', 0)), - 'burn': int(sampler_settings.get('burn', 0)), - 'thin': int(sampler_settings.get('thin', 0)), - 'pop': int(sampler_settings.get('pop', 0)), - 'parallel': int(sampler_settings.get('parallel', 0)), - 'init': str(sampler_settings.get('init', '')), - 'random_seed': sampler_settings.get('random_seed'), - }, + sampler_settings=self._restored_bayesian_sampler_settings(sampler_settings), convergence_diagnostics={ 'converged': False, 'max_r_hat': self.fit_result.gelman_rubin_max.value, @@ -796,6 +778,42 @@ def _restore_fit_results_from_projection(self) -> object | None: self.fit_results = restored_results return restored_results + def _restored_bayesian_sampler_settings( + self, + sampler_settings: dict[str, object], + ) -> dict[str, object]: + """Return display-oriented sampler settings for restored results.""" + if self.minimizer.type == MinimizerTypeEnum.EMCEE.value: + return { + 'steps': self._int_sampler_setting(sampler_settings, 'nsteps'), + 'burn': self._int_sampler_setting(sampler_settings, 'nburn'), + 'thin': self._int_sampler_setting(sampler_settings, 'thin'), + 'pop': self._int_sampler_setting(sampler_settings, 'nwalkers'), + 'parallel': self._int_sampler_setting(sampler_settings, 'parallel_workers'), + 'init': str(sampler_settings.get('initialization_method', '')), + 'proposal_moves': str(sampler_settings.get('proposal_moves', '')), + 'random_seed': sampler_settings.get('random_seed'), + } + + return { + 'steps': self._int_sampler_setting(sampler_settings, 'steps'), + 'burn': self._int_sampler_setting(sampler_settings, 'burn'), + 'thin': self._int_sampler_setting(sampler_settings, 'thin'), + 'pop': self._int_sampler_setting(sampler_settings, 'pop'), + 'parallel': self._int_sampler_setting(sampler_settings, 'parallel'), + 'init': str(sampler_settings.get('init', '')), + 'random_seed': sampler_settings.get('random_seed'), + } + + @staticmethod + def _int_sampler_setting( + sampler_settings: dict[str, object], + key: str, + ) -> int: + """Return an integer sampler setting with a zero fallback.""" + value = sampler_settings.get(key, 0) + return 0 if value is None else int(value) + def help(self) -> None: """Print a summary of analysis properties and methods.""" cls = type(self) @@ -1245,7 +1263,7 @@ def _fit_state_categories(self) -> list[object]: ] try: - result_kind = FitResultKindEnum(self.fit_result.result_kind.value) + FitResultKindEnum(self.fit_result.result_kind.value) except ValueError: log.warning( 'Unsupported fit_result.result_kind while serializing analysis CIF: ' @@ -1254,9 +1272,6 @@ def _fit_state_categories(self) -> list[object]: ) return categories - if result_kind is FitResultKindEnum.DETERMINISTIC: - return categories - return categories def _clear_persisted_fit_state(self) -> None: @@ -1640,7 +1655,7 @@ def _store_posterior_predictive_projection( results.posterior_predictive[summary.experiment_name] = summary results.posterior_predictive[ - self._predictive_cache_key( + posterior_predictive_cache_key( summary.experiment_name, str(x_axis_name), include_draws=False, diff --git a/src/easydiffraction/analysis/fit_helpers/bayesian.py b/src/easydiffraction/analysis/fit_helpers/bayesian.py index 20dd78a36..2f7523074 100644 --- a/src/easydiffraction/analysis/fit_helpers/bayesian.py +++ b/src/easydiffraction/analysis/fit_helpers/bayesian.py @@ -32,6 +32,17 @@ DiagnosticsMap = dict[str, object] | None +def posterior_predictive_cache_key( + experiment_name: str, + x_axis_name: str, + *, + include_draws: bool = True, +) -> str: + """Return the cache key for one posterior predictive summary.""" + key_suffix = 'draws' if include_draws else 'band' + return f'{experiment_name}:{x_axis_name}:{key_suffix}' + + @dataclass(slots=True) class PosteriorPredictiveSummary: """ diff --git a/src/easydiffraction/display/plotting.py b/src/easydiffraction/display/plotting.py index f96c812a4..200843411 100644 --- a/src/easydiffraction/display/plotting.py +++ b/src/easydiffraction/display/plotting.py @@ -19,6 +19,7 @@ from easydiffraction.analysis.enums import FitCorrelationSourceEnum from easydiffraction.analysis.enums import FitResultKindEnum from easydiffraction.analysis.fit_helpers.bayesian import PosteriorPredictiveSummary +from easydiffraction.analysis.fit_helpers.bayesian import posterior_predictive_cache_key from easydiffraction.datablocks.experiment.item.base import intensity_category_for from easydiffraction.datablocks.experiment.item.enums import SampleFormEnum from easydiffraction.datablocks.experiment.item.enums import ScatteringTypeEnum @@ -3542,12 +3543,12 @@ def _get_or_build_posterior_predictive_summary( return None x_axis_name = getattr(x_axis, 'value', x_axis) - draw_cache_key = self._posterior_predictive_key( + draw_cache_key = posterior_predictive_cache_key( expt_name, str(x_axis_name), include_draws=True, ) - band_cache_key = self._posterior_predictive_key( + band_cache_key = posterior_predictive_cache_key( expt_name, str(x_axis_name), include_draws=False, @@ -3819,17 +3820,6 @@ def _posterior_predictive_draw_indices(n_draws: int) -> np.ndarray: ) ) - @staticmethod - def _posterior_predictive_key( - expt_name: str, - x_axis_name: str, - *, - include_draws: bool = True, - ) -> str: - """Return the cache key for a posterior predictive summary.""" - key_suffix = 'draws' if include_draws else 'band' - return f'{expt_name}:{x_axis_name}:{key_suffix}' - def _get_posterior_inference_data( self, ) -> tuple[object | None, object | None]: diff --git a/src/easydiffraction/project/display.py b/src/easydiffraction/project/display.py index 5067e40d3..d6067d1ec 100644 --- a/src/easydiffraction/project/display.py +++ b/src/easydiffraction/project/display.py @@ -8,6 +8,7 @@ from dataclasses import dataclass from typing import TYPE_CHECKING +from easydiffraction.analysis.fit_helpers.bayesian import posterior_predictive_cache_key from easydiffraction.datablocks.experiment.item.base import intensity_category_for from easydiffraction.datablocks.experiment.item.enums import SampleFormEnum from easydiffraction.datablocks.experiment.item.enums import ScatteringTypeEnum @@ -219,8 +220,8 @@ def _predictive_needs_processing_indicator( return True cache_keys = [ - plotter._posterior_predictive_key(expt_name, x_axis_name, include_draws=True), - plotter._posterior_predictive_key(expt_name, x_axis_name, include_draws=False), + posterior_predictive_cache_key(expt_name, x_axis_name, include_draws=True), + posterior_predictive_cache_key(expt_name, x_axis_name, include_draws=False), expt_name, ] for cache_key in cache_keys: From 26e6b3cc2661b153e3d3da9d1f90967170dd5808 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sun, 24 May 2026 22:46:02 +0200 Subject: [PATCH 16/65] Add ed-25 emcee tutorial --- docs/dev/plans/emcee-minimizer.md | 2 +- docs/docs/tutorials/ed-25.ipynb | 573 ++++++++++++++++++++++++++++++ docs/docs/tutorials/ed-25.py | 242 +++++++++++++ docs/docs/tutorials/index.md | 4 + docs/mkdocs.yml | 1 + 5 files changed, 821 insertions(+), 1 deletion(-) create mode 100644 docs/docs/tutorials/ed-25.ipynb create mode 100644 docs/docs/tutorials/ed-25.py diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index 96836b3ea..827c3e97b 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -652,7 +652,7 @@ Mark `[x]` as each step lands. Commit: `Route emcee posterior through fit_result and sidecar` -- [ ] **P1.7 — Add `ed-25.py` tutorial.** Verify first that +- [x] **P1.7 — Add `ed-25.py` tutorial.** Verify first that `docs/docs/tutorials/ed-25.py` is unused. `ed-23.py` is the "Co2SiO4 Sequential Fit" tutorial and `ed-24.py` is the "LBCO Bayesian Display" tutorial — do **not** overwrite either. If diff --git a/docs/docs/tutorials/ed-25.ipynb b/docs/docs/tutorials/ed-25.ipynb new file mode 100644 index 000000000..e4f899efa --- /dev/null +++ b/docs/docs/tutorials/ed-25.ipynb @@ -0,0 +1,573 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "id": "0", + "metadata": { + "tags": [ + "hide-in-docs" + ] + }, + "outputs": [], + "source": [ + "# Check whether easydiffraction is installed; install it if needed.\n", + "# Required for remote environments such as Google Colab.\n", + "import importlib.util\n", + "\n", + "if importlib.util.find_spec('easydiffraction') is None:\n", + " %pip install easydiffraction" + ] + }, + { + "cell_type": "markdown", + "id": "1", + "metadata": {}, + "source": [ + "# Bayesian Analysis with emcee: LBCO, HRPT\n", + "\n", + "This tutorial demonstrates how to run Bayesian sampling with the\n", + "emcee minimizer and then resume the same chain from the saved project.\n", + "\n", + "The workflow uses the same La0.5Ba0.5CoO3 powder diffraction example\n", + "as the DREAM Bayesian tutorial:\n", + "\n", + "- run a short local refinement,\n", + "- derive finite fit bounds for the sampled parameters,\n", + "- switch to emcee and sample the posterior,\n", + "- save the project with the emcee chain,\n", + "- resume the chain with additional steps,\n", + "- inspect posterior plots after each sampling stage." + ] + }, + { + "cell_type": "markdown", + "id": "2", + "metadata": {}, + "source": [ + "## Import Library" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3", + "metadata": {}, + "outputs": [], + "source": [ + "import easydiffraction as ed" + ] + }, + { + "cell_type": "markdown", + "id": "4", + "metadata": {}, + "source": [ + "## Create a Project Container\n", + "\n", + "The project is saved before sampling because emcee stores its chain in\n", + "the project's analysis sidecar file." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5", + "metadata": {}, + "outputs": [], + "source": [ + "project = ed.Project()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6", + "metadata": {}, + "outputs": [], + "source": [ + "project.save_as('projects/lbco_hrpt_emcee')" + ] + }, + { + "cell_type": "markdown", + "id": "7", + "metadata": {}, + "source": [ + "## Build the Structural Model\n", + "\n", + "Define a compact cubic perovskite model for La0.5Ba0.5CoO3." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8", + "metadata": {}, + "outputs": [], + "source": [ + "project.structures.create(name='lbco')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9", + "metadata": {}, + "outputs": [], + "source": [ + "structure = project.structures['lbco']" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "10", + "metadata": {}, + "outputs": [], + "source": [ + "structure.space_group.name_h_m = 'P m -3 m'\n", + "structure.space_group.it_coordinate_system_code = '1'" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "11", + "metadata": {}, + "outputs": [], + "source": [ + "structure.cell.length_a = 3.88" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "12", + "metadata": {}, + "outputs": [], + "source": [ + "structure.atom_sites.create(\n", + " label='La',\n", + " type_symbol='La',\n", + " fract_x=0,\n", + " fract_y=0,\n", + " fract_z=0,\n", + " wyckoff_letter='a',\n", + " adp_type='Biso',\n", + " adp_iso=0.5151,\n", + " occupancy=0.5,\n", + ")\n", + "structure.atom_sites.create(\n", + " label='Ba',\n", + " type_symbol='Ba',\n", + " fract_x=0,\n", + " fract_y=0,\n", + " fract_z=0,\n", + " wyckoff_letter='a',\n", + " adp_type='Biso',\n", + " adp_iso=0.5151,\n", + " occupancy=0.5,\n", + ")\n", + "structure.atom_sites.create(\n", + " label='Co',\n", + " type_symbol='Co',\n", + " fract_x=0.5,\n", + " fract_y=0.5,\n", + " fract_z=0.5,\n", + " wyckoff_letter='b',\n", + " adp_type='Biso',\n", + " adp_iso=0.2190,\n", + ")\n", + "structure.atom_sites.create(\n", + " label='O',\n", + " type_symbol='O',\n", + " fract_x=0,\n", + " fract_y=0.5,\n", + " fract_z=0.5,\n", + " wyckoff_letter='c',\n", + " adp_type='Biso',\n", + " adp_iso=1.3916,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "13", + "metadata": {}, + "source": [ + "## Define the Diffraction Experiment\n", + "\n", + "Download the HRPT powder pattern, create a neutron powder experiment,\n", + "and set the key instrument, peak-profile, and background values." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "14", + "metadata": {}, + "outputs": [], + "source": [ + "data_path = ed.download_data(id=3, destination='data')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "15", + "metadata": {}, + "outputs": [], + "source": [ + "project.experiments.add_from_data_path(\n", + " name='hrpt',\n", + " data_path=data_path,\n", + " sample_form='powder',\n", + " beam_mode='constant wavelength',\n", + " radiation_probe='neutron',\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "16", + "metadata": {}, + "outputs": [], + "source": [ + "experiment = project.experiments['hrpt']" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "17", + "metadata": {}, + "outputs": [], + "source": [ + "experiment.linked_phases.create(id='lbco', scale=9.1351)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "18", + "metadata": {}, + "outputs": [], + "source": [ + "experiment.instrument.setup_wavelength = 1.494\n", + "experiment.instrument.calib_twotheta_offset = 0.0" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "19", + "metadata": {}, + "outputs": [], + "source": [ + "experiment.peak.broad_gauss_u = 0.1\n", + "experiment.peak.broad_gauss_v = -0.1\n", + "experiment.peak.broad_gauss_w = 0.1204\n", + "experiment.peak.broad_lorentz_y = 0.0844" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "20", + "metadata": {}, + "outputs": [], + "source": [ + "experiment.background.create(id='1', x=10, y=168.5585)\n", + "experiment.background.create(id='2', x=30, y=164.3357)\n", + "experiment.background.create(id='3', x=50, y=166.8881)\n", + "experiment.background.create(id='4', x=110, y=175.4006)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "21", + "metadata": {}, + "outputs": [], + "source": [ + "experiment.excluded_regions.create(id='1', start=0, end=10)\n", + "experiment.excluded_regions.create(id='2', start=100, end=180)" + ] + }, + { + "cell_type": "markdown", + "id": "22", + "metadata": {}, + "source": [ + "## Run a Local Refinement First\n", + "\n", + "The local fit provides starting values and uncertainties that are used\n", + "to build finite bounds for emcee." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "23", + "metadata": {}, + "outputs": [], + "source": [ + "structure.cell.length_a.free = True" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "24", + "metadata": {}, + "outputs": [], + "source": [ + "experiment.linked_phases['lbco'].scale.free = True\n", + "experiment.peak.broad_gauss_u.free = True\n", + "experiment.peak.broad_gauss_v.free = True\n", + "experiment.instrument.calib_twotheta_offset.free = True" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "25", + "metadata": {}, + "outputs": [], + "source": [ + "project.analysis.minimizer.type = 'bumps (lm)'" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "26", + "metadata": {}, + "outputs": [], + "source": [ + "project.analysis.fit()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "27", + "metadata": {}, + "outputs": [], + "source": [ + "project.display.fit.results()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "28", + "metadata": {}, + "outputs": [], + "source": [ + "for param in project.free_parameters:\n", + " param.set_fit_bounds_from_uncertainty()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "29", + "metadata": {}, + "outputs": [], + "source": [ + "project.display.parameters.free()" + ] + }, + { + "cell_type": "markdown", + "id": "30", + "metadata": {}, + "source": [ + "## Run emcee Sampling\n", + "\n", + "The sampling settings are intentionally small for tutorial runtime.\n", + "Use more steps and inspect convergence diagnostics for production\n", + "analysis." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "31", + "metadata": {}, + "outputs": [], + "source": [ + "project.analysis.minimizer.type = 'emcee'" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "32", + "metadata": {}, + "outputs": [], + "source": [ + "project.analysis.minimizer.sampling_steps = 1000\n", + "project.analysis.minimizer.burn_in_steps = 200\n", + "project.analysis.minimizer.thinning_interval = 10\n", + "project.analysis.minimizer.population_size = 32\n", + "project.analysis.minimizer.initialization_method = 'ball'\n", + "project.analysis.minimizer.random_seed = 12345" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "33", + "metadata": {}, + "outputs": [], + "source": [ + "project.analysis.fit()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "34", + "metadata": {}, + "outputs": [], + "source": [ + "project.display.fit.results()" + ] + }, + { + "cell_type": "markdown", + "id": "35", + "metadata": {}, + "source": [ + "## Inspect the Posterior\n", + "\n", + "The posterior distribution plot shows the sampled marginal\n", + "distributions after the first emcee run." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "36", + "metadata": {}, + "outputs": [], + "source": [ + "project.display.posterior.distribution()" + ] + }, + { + "cell_type": "markdown", + "id": "37", + "metadata": {}, + "source": [ + "The posterior predictive plot propagates the sampled parameter\n", + "uncertainty into the calculated diffraction pattern." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "38", + "metadata": {}, + "outputs": [], + "source": [ + "project.display.posterior.predictive(expt_name='hrpt')" + ] + }, + { + "cell_type": "markdown", + "id": "39", + "metadata": {}, + "source": [ + "## Save the Sampled Project\n", + "\n", + "Saving persists both the analysis state and the emcee chain sidecar so\n", + "the same chain can be resumed later." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "40", + "metadata": {}, + "outputs": [], + "source": [ + "project.save()" + ] + }, + { + "cell_type": "markdown", + "id": "41", + "metadata": {}, + "source": [ + "## Resume emcee Sampling\n", + "\n", + "Resume from the saved backend and append 500 more emcee steps to the\n", + "existing chain." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "42", + "metadata": {}, + "outputs": [], + "source": [ + "project.analysis.fit(resume=True, extra_steps=500)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "43", + "metadata": {}, + "outputs": [], + "source": [ + "project.display.fit.results()" + ] + }, + { + "cell_type": "markdown", + "id": "44", + "metadata": {}, + "source": [ + "## Inspect the Resumed Posterior\n", + "\n", + "After resume, the posterior plots use the extended chain." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "45", + "metadata": {}, + "outputs": [], + "source": [ + "project.display.posterior.distribution()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "46", + "metadata": {}, + "outputs": [], + "source": [ + "project.display.posterior.predictive(expt_name='hrpt')" + ] + } + ], + "metadata": { + "jupytext": { + "cell_metadata_filter": "-all", + "main_language": "python", + "notebook_metadata_filter": "-all" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/docs/tutorials/ed-25.py b/docs/docs/tutorials/ed-25.py new file mode 100644 index 000000000..34965d2b4 --- /dev/null +++ b/docs/docs/tutorials/ed-25.py @@ -0,0 +1,242 @@ +# %% [markdown] +# # Bayesian Analysis with emcee: LBCO, HRPT +# +# This tutorial demonstrates how to run Bayesian sampling with the +# emcee minimizer and then resume the same chain from the saved project. +# +# The workflow uses the same La0.5Ba0.5CoO3 powder diffraction example +# as the DREAM Bayesian tutorial: +# +# - run a short local refinement, +# - derive finite fit bounds for the sampled parameters, +# - switch to emcee and sample the posterior, +# - save the project with the emcee chain, +# - resume the chain with additional steps, +# - inspect posterior plots after each sampling stage. + +# %% [markdown] +# ## Import Library + +# %% +import easydiffraction as ed + +# %% [markdown] +# ## Create a Project Container +# +# The project is saved before sampling because emcee stores its chain in +# the project's analysis sidecar file. + +# %% +project = ed.Project() + +# %% +project.save_as('projects/lbco_hrpt_emcee') + +# %% [markdown] +# ## Build the Structural Model +# +# Define a compact cubic perovskite model for La0.5Ba0.5CoO3. + +# %% +project.structures.create(name='lbco') + +# %% +structure = project.structures['lbco'] + +# %% +structure.space_group.name_h_m = 'P m -3 m' +structure.space_group.it_coordinate_system_code = '1' + +# %% +structure.cell.length_a = 3.88 + +# %% +structure.atom_sites.create( + label='La', + type_symbol='La', + fract_x=0, + fract_y=0, + fract_z=0, + wyckoff_letter='a', + adp_type='Biso', + adp_iso=0.5151, + occupancy=0.5, +) +structure.atom_sites.create( + label='Ba', + type_symbol='Ba', + fract_x=0, + fract_y=0, + fract_z=0, + wyckoff_letter='a', + adp_type='Biso', + adp_iso=0.5151, + occupancy=0.5, +) +structure.atom_sites.create( + label='Co', + type_symbol='Co', + fract_x=0.5, + fract_y=0.5, + fract_z=0.5, + wyckoff_letter='b', + adp_type='Biso', + adp_iso=0.2190, +) +structure.atom_sites.create( + label='O', + type_symbol='O', + fract_x=0, + fract_y=0.5, + fract_z=0.5, + wyckoff_letter='c', + adp_type='Biso', + adp_iso=1.3916, +) + +# %% [markdown] +# ## Define the Diffraction Experiment +# +# Download the HRPT powder pattern, create a neutron powder experiment, +# and set the key instrument, peak-profile, and background values. + +# %% +data_path = ed.download_data(id=3, destination='data') + +# %% +project.experiments.add_from_data_path( + name='hrpt', + data_path=data_path, + sample_form='powder', + beam_mode='constant wavelength', + radiation_probe='neutron', +) + +# %% +experiment = project.experiments['hrpt'] + +# %% +experiment.linked_phases.create(id='lbco', scale=9.1351) + +# %% +experiment.instrument.setup_wavelength = 1.494 +experiment.instrument.calib_twotheta_offset = 0.0 + +# %% +experiment.peak.broad_gauss_u = 0.1 +experiment.peak.broad_gauss_v = -0.1 +experiment.peak.broad_gauss_w = 0.1204 +experiment.peak.broad_lorentz_y = 0.0844 + +# %% +experiment.background.create(id='1', x=10, y=168.5585) +experiment.background.create(id='2', x=30, y=164.3357) +experiment.background.create(id='3', x=50, y=166.8881) +experiment.background.create(id='4', x=110, y=175.4006) + +# %% +experiment.excluded_regions.create(id='1', start=0, end=10) +experiment.excluded_regions.create(id='2', start=100, end=180) + +# %% [markdown] +# ## Run a Local Refinement First +# +# The local fit provides starting values and uncertainties that are used +# to build finite bounds for emcee. + +# %% +structure.cell.length_a.free = True + +# %% +experiment.linked_phases['lbco'].scale.free = True +experiment.peak.broad_gauss_u.free = True +experiment.peak.broad_gauss_v.free = True +experiment.instrument.calib_twotheta_offset.free = True + +# %% +project.analysis.minimizer.type = 'bumps (lm)' + +# %% +project.analysis.fit() + +# %% +project.display.fit.results() + +# %% +for param in project.free_parameters: + param.set_fit_bounds_from_uncertainty() + +# %% +project.display.parameters.free() + +# %% [markdown] +# ## Run emcee Sampling +# +# The sampling settings are intentionally small for tutorial runtime. +# Use more steps and inspect convergence diagnostics for production +# analysis. + +# %% +project.analysis.minimizer.type = 'emcee' + +# %% +project.analysis.minimizer.sampling_steps = 1000 +project.analysis.minimizer.burn_in_steps = 200 +project.analysis.minimizer.thinning_interval = 10 +project.analysis.minimizer.population_size = 32 +project.analysis.minimizer.initialization_method = 'ball' +project.analysis.minimizer.random_seed = 12345 + +# %% +project.analysis.fit() + +# %% +project.display.fit.results() + +# %% [markdown] +# ## Inspect the Posterior +# +# The posterior distribution plot shows the sampled marginal +# distributions after the first emcee run. + +# %% +project.display.posterior.distribution() + +# %% [markdown] +# The posterior predictive plot propagates the sampled parameter +# uncertainty into the calculated diffraction pattern. + +# %% +project.display.posterior.predictive(expt_name='hrpt') + +# %% [markdown] +# ## Save the Sampled Project +# +# Saving persists both the analysis state and the emcee chain sidecar so +# the same chain can be resumed later. + +# %% +project.save() + +# %% [markdown] +# ## Resume emcee Sampling +# +# Resume from the saved backend and append 500 more emcee steps to the +# existing chain. + +# %% +project.analysis.fit(resume=True, extra_steps=500) + +# %% +project.display.fit.results() + +# %% [markdown] +# ## Inspect the Resumed Posterior +# +# After resume, the posterior plots use the extended chain. + +# %% +project.display.posterior.distribution() + +# %% +project.display.posterior.predictive(expt_name='hrpt') diff --git a/docs/docs/tutorials/index.md b/docs/docs/tutorials/index.md index b789645d0..13ed5d11a 100644 --- a/docs/docs/tutorials/index.md +++ b/docs/docs/tutorials/index.md @@ -115,6 +115,10 @@ The tutorials are organized into the following categories: tutorial covers the use of Markov Chain Monte Carlo (MCMC) sampling to explore the posterior distribution of the refined parameters, providing insights into parameter uncertainties and correlations. +- [LBCO emcee Resume](ed-25.ipynb) – Demonstrates how to run Bayesian + sampling with the emcee minimizer, save the project with its chain, + resume sampling with additional steps, and inspect the posterior + before and after resume. - [Tb2TiO7 Bayesian](ed-22.ipynb) – Another example of a Bayesian analysis. This tutorial focuses on the Tb2TiO7 crystal structure using constant wavelength neutron single crystal diffraction data from HEiDi diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 02f862023..72985eec5 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -221,6 +221,7 @@ nav: - BEER McStas: tutorials/ed-20.ipynb - Bayesian Analysis: - LBCO Bayesian: tutorials/ed-21.ipynb + - LBCO emcee Resume: tutorials/ed-25.ipynb - Tb2TiO7 Bayesian: tutorials/ed-22.ipynb - Workshops & Schools: - DMSC Summer School: tutorials/ed-13.ipynb From 54674e5777c79b9fb35565ba66b288ad46c2bad2 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sun, 24 May 2026 22:46:34 +0200 Subject: [PATCH 17/65] Mark emcee phase 1 review gate --- docs/dev/plans/emcee-minimizer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index 827c3e97b..247d57f4e 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -695,7 +695,7 @@ Mark `[x]` as each step lands. Commit: `Add ed-25 emcee tutorial` -- [ ] **P1.8 — Phase 1 review gate.** No code change. Stop and request +- [x] **P1.8 — Phase 1 review gate.** No code change. Stop and request user review. After approval, proceed to Phase 2. ## Verification (Phase 2) From fd29decca639b48744a923349b9fae13368772eb Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sun, 24 May 2026 23:24:57 +0200 Subject: [PATCH 18/65] Improve emcee sampler progress reporting --- docs/dev/plans/emcee-minimizer.md | 2 +- .../analysis/categories/minimizer/emcee.py | 2 +- .../analysis/fit_helpers/bayesian.py | 10 +- .../analysis/minimizers/emcee.py | 355 +++++++++++++++--- .../categories/minimizer/test_emcee.py | 18 + .../analysis/fit_helpers/test_bayesian.py | 20 + .../analysis/minimizers/test_emcee.py | 125 ++++++ 7 files changed, 469 insertions(+), 63 deletions(-) create mode 100644 tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py create mode 100644 tests/unit/easydiffraction/analysis/minimizers/test_emcee.py diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index 247d57f4e..9138f0b18 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -296,7 +296,7 @@ Mark `[x]` as each step lands. native kwargs (see §"Decisions already made" point 3). - Class-level defaults for emcee-specific values: `sampling_steps=5000`, `burn_in_steps=1000`, `thinning_interval=5`, - `population_size=32`, `parallel_workers=0`, + `population_size=32`, `parallel_workers=1`, `proposal_moves='stretch'`. - `__init__` constructs descriptors via the inherited helpers (`_sampling_steps_descriptor(default)`, etc. from diff --git a/src/easydiffraction/analysis/categories/minimizer/emcee.py b/src/easydiffraction/analysis/categories/minimizer/emcee.py index e71d8b1c7..bd0ac6196 100644 --- a/src/easydiffraction/analysis/categories/minimizer/emcee.py +++ b/src/easydiffraction/analysis/categories/minimizer/emcee.py @@ -20,7 +20,7 @@ DEFAULT_BURN_IN_STEPS = 1000 DEFAULT_THINNING_INTERVAL = 5 DEFAULT_POPULATION_SIZE = 32 -DEFAULT_PARALLEL_WORKERS = 0 +DEFAULT_PARALLEL_WORKERS = 1 DEFAULT_INITIALIZATION_METHOD = InitializationMethodEnum.BALL DEFAULT_PROPOSAL_MOVES = 'stretch' SUPPORTED_PROPOSAL_MOVES = ('stretch', 'de', 'de_snooker', 'walk') diff --git a/src/easydiffraction/analysis/fit_helpers/bayesian.py b/src/easydiffraction/analysis/fit_helpers/bayesian.py index 2f7523074..b248a92fc 100644 --- a/src/easydiffraction/analysis/fit_helpers/bayesian.py +++ b/src/easydiffraction/analysis/fit_helpers/bayesian.py @@ -4,6 +4,7 @@ from __future__ import annotations +import warnings from dataclasses import dataclass import arviz as az @@ -158,7 +159,14 @@ def to_arviz(self) -> object: if sample_stats is not None: data['sample_stats'] = sample_stats - return az.from_dict(data) + with warnings.catch_warnings(): + warnings.filterwarnings( + 'ignore', + message='Found chain dimension to be longer than draw dimension.*', + category=UserWarning, + module='arviz_base.base', + ) + return az.from_dict(data) SummaryList = list[PosteriorParameterSummary] | None diff --git a/src/easydiffraction/analysis/minimizers/emcee.py b/src/easydiffraction/analysis/minimizers/emcee.py index ade6868c4..ef06ec323 100644 --- a/src/easydiffraction/analysis/minimizers/emcee.py +++ b/src/easydiffraction/analysis/minimizers/emcee.py @@ -7,8 +7,8 @@ import multiprocessing import os import pickle # noqa: S403 - used only to test whether multiprocessing can serialize a callable. -from collections.abc import Callable from pathlib import Path +from typing import TYPE_CHECKING import emcee import numpy as np @@ -32,12 +32,184 @@ DEFAULT_NBURN = 1000 DEFAULT_THIN = 5 DEFAULT_NWALKERS = 32 -DEFAULT_PARALLEL_WORKERS = 0 +DEFAULT_PARALLEL_WORKERS = 1 DEFAULT_INITIALIZATION_METHOD = InitializationMethodEnum.BALL DEFAULT_PROPOSAL_MOVES = 'stretch' MAX_RANDOM_SEED = int(np.iinfo(np.uint32).max) EMCEE_CHAIN_GROUP = 'emcee_chain' EMCEE_FAILURES = (ArithmeticError, RuntimeError, TypeError, ValueError) +EMCEE_SAMPLE_ARRAY_NDIM = 3 +TOTAL_PROGRESS_POINTS = 25 +SUPPORTED_INITIALIZATION_METHODS = ( + InitializationMethodEnum.BALL, + InitializationMethodEnum.UNIFORM, + InitializationMethodEnum.PRIOR, +) +SUPPORTED_INITIALIZATION_METHOD_SET = frozenset(SUPPORTED_INITIALIZATION_METHODS) + +if TYPE_CHECKING: + from collections.abc import Callable + + +class _EmceeProgressReporter: + """ + Translate emcee iteration states into sampler progress rows. + """ + + def __init__( + self, + *, + tracker: object, + total_steps: int, + burn_steps: int, + ) -> None: + self._tracker = tracker + self._total_steps = max(1, total_steps) + self._burn_steps = min(max(0, burn_steps), self._total_steps) + burn_target_count, sampling_target_count = self._phase_progress_point_counts( + total_steps=self._total_steps, + burn_steps=self._burn_steps, + ) + self._burn_targets = self._progress_targets( + start=1, + stop=self._burn_steps, + target_count=burn_target_count, + ) + self._sampling_targets = self._progress_targets( + start=self._burn_steps + 1, + stop=self._total_steps, + target_count=sampling_target_count, + ) + self._next_burn_target_index = 0 + self._next_sampling_target_index = 0 + + def report(self, *, iteration: int, state: object) -> None: + """ + Forward one emcee state when it reaches a report target. + """ + clamped_iteration = min(max(1, iteration), self._total_steps) + if not self._should_report(clamped_iteration): + return + + self._tracker.track_sampler_progress( + SamplerProgressUpdate( + iteration=clamped_iteration, + total_iterations=self._total_steps, + phase=self._phase_name(clamped_iteration), + progress_percent=self._progress_percent(clamped_iteration), + log_posterior=self._log_posterior_from_state(state), + reduced_chi2=self._reduced_chi2_from_tracker(), + elapsed_time=self._tracker._current_elapsed_time(), + force_report=True, + ) + ) + + @staticmethod + def _phase_progress_point_counts( + *, + total_steps: int, + burn_steps: int, + ) -> tuple[int, int]: + """Return proportional burn and sampling progress counts.""" + total_points = min(TOTAL_PROGRESS_POINTS, max(1, total_steps)) + burn_steps = min(max(0, burn_steps), total_steps) + sampling_steps = max(total_steps - burn_steps, 0) + + if burn_steps == 0: + return 0, total_points + if sampling_steps == 0: + return total_points, 0 + + burn_target_count = round(total_points * burn_steps / total_steps) + burn_target_count = min( + max(burn_target_count, 1), + burn_steps, + total_points - 1, + ) + sampling_target_count = min( + max(total_points - burn_target_count, 1), + sampling_steps, + ) + return burn_target_count, sampling_target_count + + @staticmethod + def _progress_targets( + *, + start: int, + stop: int, + target_count: int, + ) -> list[int]: + """Return monotonically increasing reporting targets.""" + if target_count < 1 or stop < start: + return [] + + targets = np.linspace(start, stop, num=target_count) + rounded = np.rint(targets).astype(int) + unique_targets = sorted({int(value) for value in rounded if start <= value <= stop}) + if start not in unique_targets: + unique_targets.insert(0, start) + if stop not in unique_targets: + unique_targets.append(stop) + return unique_targets + + def _should_report(self, iteration: int) -> bool: + """Return whether this iteration should be rendered.""" + if self._phase_name(iteration) == 'burn-in': + return self._consume_progress_target( + iteration, + phase_targets=self._burn_targets, + target_index_name='_next_burn_target_index', + ) + + return self._consume_progress_target( + iteration, + phase_targets=self._sampling_targets, + target_index_name='_next_sampling_target_index', + ) + + def _consume_progress_target( + self, + iteration: int, + *, + phase_targets: list[int], + target_index_name: str, + ) -> bool: + """Advance a phase target pointer when iteration reaches it.""" + target_index = getattr(self, target_index_name) + should_report = False + while target_index < len(phase_targets) and iteration >= phase_targets[target_index]: + target_index += 1 + should_report = True + setattr(self, target_index_name, target_index) + return should_report + + def _phase_name(self, iteration: int) -> str: + """Return the current emcee phase name.""" + if iteration <= self._burn_steps: + return 'burn-in' + return 'sampling' + + def _progress_percent(self, iteration: int) -> float: + """Return emcee progress as a percentage.""" + return 100.0 * min(iteration, self._total_steps) / self._total_steps + + @staticmethod + def _log_posterior_from_state(state: object) -> float: + """Return the best finite log posterior from an emcee state.""" + log_probability = getattr(state, 'log_prob', None) + if log_probability is None: + return float('-inf') + + values = np.asarray(log_probability, dtype=float) + finite_values = values[np.isfinite(values)] + if finite_values.size == 0: + return float('-inf') + return float(np.max(finite_values)) + + def _reduced_chi2_from_tracker(self) -> float: + """Return the current best reduced chi-square, if available.""" + best_chi2 = getattr(self._tracker, 'best_chi2', None) + return float(best_chi2) if best_chi2 is not None else float('nan') @MinimizerFactory.register @@ -91,7 +263,7 @@ def nburn(self, value: int) -> None: @property def thin(self) -> int: - """emcee thinning interval.""" + """Emcee thinning interval.""" return self._thin @thin.setter @@ -109,7 +281,9 @@ def nwalkers(self, value: int) -> None: @property def parallel_workers(self) -> int: - """Worker count; ``0`` asks for all CPUs and ``1`` runs serially.""" + """ + Worker count; ``0`` asks for all CPUs and ``1`` runs serially. + """ return self._parallel_workers @parallel_workers.setter @@ -118,7 +292,7 @@ def parallel_workers(self, value: int) -> None: @property def initialization_method(self) -> InitializationMethodEnum: - """emcee walker initialization method.""" + """Emcee walker initialization method.""" return self._initialization_method @initialization_method.setter @@ -127,14 +301,14 @@ def initialization_method(self, value: InitializationMethodEnum | str) -> None: @property def proposal_moves(self) -> str: - """emcee proposal move name.""" + """Emcee proposal move name.""" return self._proposal_moves @proposal_moves.setter def proposal_moves(self, value: str) -> None: self._proposal_moves = self._validated_proposal_moves(value) - def fit( + def fit( # noqa: PLR0913 self, parameters: list[object], objective_function: Callable[..., object], @@ -223,7 +397,9 @@ def _validate_sampled_parameter_bounds( cls, parameters: list[object], ) -> None: - """Validate finite ordered bounds for sampled emcee parameters.""" + """ + Validate finite ordered bounds for sampled emcee parameters. + """ issues: list[str] = [] for parameter in parameters: parameter_name = cls._parameter_name_for_bound_validation(parameter) @@ -309,7 +485,9 @@ def _validated_positive_integer(name: str, value: float) -> int: @staticmethod def _validated_non_negative_integer(name: str, value: float) -> int: - """Validate an emcee setting that must be a non-negative integer.""" + """ + Validate an emcee setting that must be a non-negative integer. + """ if isinstance(value, bool): msg = f"emcee setting '{name}' must be a non-negative integer." raise TypeError(msg) @@ -333,28 +511,14 @@ def _validated_initialization_method( method = InitializationMethodEnum(value) except ValueError: valid_values = ', '.join( - initialization.value - for initialization in ( - InitializationMethodEnum.BALL, - InitializationMethodEnum.UNIFORM, - InitializationMethodEnum.PRIOR, - ) + initialization.value for initialization in SUPPORTED_INITIALIZATION_METHODS ) msg = f"emcee setting 'initialization_method' must be one of: {valid_values}." raise ValueError(msg) from None - if method not in ( - InitializationMethodEnum.BALL, - InitializationMethodEnum.UNIFORM, - InitializationMethodEnum.PRIOR, - ): + if method not in SUPPORTED_INITIALIZATION_METHOD_SET: valid_values = ', '.join( - initialization.value - for initialization in ( - InitializationMethodEnum.BALL, - InitializationMethodEnum.UNIFORM, - InitializationMethodEnum.PRIOR, - ) + initialization.value for initialization in SUPPORTED_INITIALIZATION_METHODS ) msg = f"emcee setting 'initialization_method' must be one of: {valid_values}." raise ValueError(msg) @@ -381,7 +545,7 @@ def _run_solver( parameters = list(kwargs['parameters']) parameter_names = list(kwargs['parameter_names']) random_seed = int(kwargs['random_seed']) - resume = bool(kwargs.get('resume', False)) + resume = bool(kwargs.get('resume')) extra_steps = kwargs.get('extra_steps') self._validate_walker_count(n_parameters=len(parameter_names)) @@ -408,35 +572,17 @@ def _run_solver( ) pool = self._build_pool(log_prob) try: - if resume: - self._validate_resume( - backend=backend, - n_parameters=len(parameter_names), - extra_steps=extra_steps, - ) - else: - backend.reset(self.nwalkers, len(parameter_names)) - - sampler = emcee.EnsembleSampler( - nwalkers=self.nwalkers, - ndim=len(parameter_names), - log_prob_fn=log_prob, - pool=pool, - moves=self._resolve_moves(self.proposal_moves), + sampler = self._run_sampler( backend=backend, + log_prob=log_prob, + pool=pool, + parameters=parameters, + n_parameters=len(parameter_names), + random_seed=random_seed, + resume=resume, + extra_steps=extra_steps, + total_iterations=total_iterations, ) - self._sampler = sampler - - if resume: - sampler.run_mcmc( - None, - nsteps=int(extra_steps), - skip_initial_state_check=True, - progress=False, - ) - else: - initial_state = self._initial_state(parameters, random_seed=random_seed) - sampler.run_mcmc(initial_state, nsteps=self.nsteps, progress=False) except EMCEE_FAILURES as error: return self._failure_result( message=f'emcee sampling failed: {error}', @@ -466,6 +612,87 @@ def _run_solver( starting_uncertainties=kwargs['starting_uncertainties'], ) + def _run_sampler( # noqa: PLR0913 + self, + *, + backend: emcee.backends.HDFBackend, + log_prob: Callable[[np.ndarray], float], + pool: object | None, + parameters: list[object], + n_parameters: int, + random_seed: int, + resume: bool, + extra_steps: object, + total_iterations: int, + ) -> emcee.EnsembleSampler: + """Configure emcee, run sampling, and return the sampler.""" + if resume: + self._validate_resume( + backend=backend, + n_parameters=n_parameters, + extra_steps=extra_steps, + ) + else: + backend.reset(self.nwalkers, n_parameters) + + sampler = emcee.EnsembleSampler( + nwalkers=self.nwalkers, + ndim=n_parameters, + log_prob_fn=log_prob, + pool=pool, + moves=self._resolve_moves(self.proposal_moves), + backend=backend, + ) + self._sampler = sampler + + reporter = _EmceeProgressReporter( + tracker=self.tracker, + total_steps=total_iterations, + burn_steps=0 if resume else self.nburn, + ) + if resume: + self._sample_with_progress( + sampler=sampler, + initial_state=None, + iterations=int(extra_steps), + reporter=reporter, + skip_initial_state_check=True, + ) + return sampler + + initial_state = self._initial_state(parameters, random_seed=random_seed) + self._sample_with_progress( + sampler=sampler, + initial_state=initial_state, + iterations=self.nsteps, + reporter=reporter, + skip_initial_state_check=False, + ) + return sampler + + @staticmethod + def _sample_with_progress( + *, + sampler: emcee.EnsembleSampler, + initial_state: object | None, + iterations: int, + reporter: _EmceeProgressReporter, + skip_initial_state_check: bool, + ) -> None: + """ + Run emcee one iteration at a time and report sampler progress. + """ + for iteration, state in enumerate( + sampler.sample( + initial_state, + iterations=iterations, + skip_initial_state_check=skip_initial_state_check, + progress=False, + ), + start=1, + ): + reporter.report(iteration=iteration, state=state) + @staticmethod def _backend_iteration(backend: object) -> int: """Return backend iteration count, or zero when unavailable.""" @@ -566,7 +793,7 @@ def _build_pool(self, log_prob: Callable[[np.ndarray], float]) -> object | None: return None try: - pickle.dumps(log_prob) # noqa: S301 - no untrusted data is deserialized. + pickle.dumps(log_prob) except (AttributeError, TypeError, pickle.PickleError): self._warn_after_tracking( 'emcee parallel evaluation requires a picklable objective; ' @@ -648,7 +875,9 @@ def _failure_result( raw_state: object, sampler_completed: bool, ) -> OptimizeResult: - """Build a normalized failure result for an incomplete emcee run.""" + """ + Build a normalized failure result for an incomplete emcee run. + """ return OptimizeResult( x=np.asarray(starting_values, dtype=float), dx=None, @@ -668,7 +897,7 @@ def _failure_result( starting_uncertainties=list(starting_uncertainties), ) - def _build_success_result( + def _build_success_result( # noqa: PLR0914 self, *, sampler: emcee.EnsembleSampler, @@ -693,7 +922,11 @@ def _build_success_result( n_parameters=len(parameter_names), ) - if chain.ndim != 3 or chain.size == 0 or log_posterior.shape != chain.shape[:2]: + if ( + chain.ndim != EMCEE_SAMPLE_ARRAY_NDIM + or chain.size == 0 + or log_posterior.shape != chain.shape[:2] + ): return self._failure_result( message='emcee sampling did not return usable posterior samples.', starting_values=starting_values, @@ -772,7 +1005,9 @@ def _convergence_diagnostics( posterior_samples: PosteriorSamples, sampler: emcee.EnsembleSampler, ) -> dict[str, object]: - """Compute convergence diagnostics and add emcee acceptance rate.""" + """ + Compute convergence diagnostics and add emcee acceptance rate. + """ try: convergence_diagnostics = compute_convergence_diagnostics(posterior_samples) except (TypeError, ValueError, RuntimeError) as error: @@ -897,6 +1132,6 @@ def _build_fit_results( fit_results.result = raw_result return fit_results - def _check_success(self, raw_result: object) -> bool: + def _check_success(self, raw_result: object) -> bool: # noqa: PLR6301 """Determine success from normalized emcee result.""" return bool(getattr(raw_result, 'success', False)) diff --git a/tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py b/tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py new file mode 100644 index 000000000..c0491721e --- /dev/null +++ b/tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py @@ -0,0 +1,18 @@ +# SPDX-FileCopyrightText: 2026 EasyScience contributors +# SPDX-License-Identifier: BSD-3-Clause +"""Tests for the emcee minimizer category.""" + +from __future__ import annotations + + +def test_emcee_minimizer_category_defaults_to_serial_parallel_workers(): + from easydiffraction.analysis.categories.minimizer.emcee import ( + DEFAULT_PARALLEL_WORKERS, + ) + from easydiffraction.analysis.categories.minimizer.emcee import EmceeMinimizer + + minimizer = EmceeMinimizer() + + assert DEFAULT_PARALLEL_WORKERS == 1 + assert minimizer.parallel_workers.value == 1 + assert minimizer._native_kwargs()['parallel_workers'] == 1 diff --git a/tests/unit/easydiffraction/analysis/fit_helpers/test_bayesian.py b/tests/unit/easydiffraction/analysis/fit_helpers/test_bayesian.py index 348f380f3..554131130 100644 --- a/tests/unit/easydiffraction/analysis/fit_helpers/test_bayesian.py +++ b/tests/unit/easydiffraction/analysis/fit_helpers/test_bayesian.py @@ -3,6 +3,8 @@ from __future__ import annotations +import warnings + import numpy as np import pytest @@ -57,6 +59,24 @@ def test_posterior_samples_flatten_and_to_arviz(): assert inference_data.sample_stats['lp'].shape == (2, 2) +def test_posterior_samples_to_arviz_allows_more_chains_than_draws_without_warning(): + from easydiffraction.analysis.fit_helpers.bayesian import PosteriorSamples + + posterior_samples = PosteriorSamples( + parameter_names=['a'], + parameter_samples=np.ones((2, 32, 1), dtype=float), + log_posterior=np.ones((2, 32), dtype=float), + ) + + with warnings.catch_warnings(record=True) as caught_warnings: + warnings.simplefilter('always') + inference_data = posterior_samples.to_arviz() + + warning_messages = [str(warning.message) for warning in caught_warnings] + assert not any('Found chain dimension' in message for message in warning_messages) + assert inference_data.posterior['a'].shape == (32, 2) + + def test_posterior_samples_to_arviz_validates_shapes(): from easydiffraction.analysis.fit_helpers.bayesian import PosteriorSamples diff --git a/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py b/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py new file mode 100644 index 000000000..cbd66b198 --- /dev/null +++ b/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py @@ -0,0 +1,125 @@ +# SPDX-FileCopyrightText: 2026 EasyScience contributors +# SPDX-License-Identifier: BSD-3-Clause +"""Tests for the emcee minimizer engine.""" + +from __future__ import annotations + +from types import SimpleNamespace + +import numpy as np +import pytest + + +class _FakeTracker: + """Progress tracker test double.""" + + best_chi2 = 1.23 + + def __init__(self) -> None: + self.updates = [] + + def _current_elapsed_time(self) -> float: + return float(len(self.updates) + 1) + + def track_sampler_progress(self, update: object) -> None: + self.updates.append(update) + + +class _FakeSampler: + """Minimal sampler test double for the emcee sample loop.""" + + def __init__(self) -> None: + self.calls = [] + + def sample( + self, + initial_state: object, + *, + iterations: int, + skip_initial_state_check: bool, + progress: bool, + ) -> object: + self.calls.append( + { + 'initial_state': initial_state, + 'iterations': iterations, + 'skip_initial_state_check': skip_initial_state_check, + 'progress': progress, + } + ) + for index in range(iterations): + yield SimpleNamespace(log_prob=np.array([float(index)], dtype=float)) + + +def test_emcee_minimizer_defaults_to_serial_parallel_workers(): + from easydiffraction.analysis.minimizers.emcee import DEFAULT_PARALLEL_WORKERS + from easydiffraction.analysis.minimizers.emcee import EmceeMinimizer + + minimizer = EmceeMinimizer() + + assert DEFAULT_PARALLEL_WORKERS == 1 + assert minimizer.parallel_workers == 1 + + +def test_emcee_progress_reporter_emits_burn_in_and_sampling_updates(): + from easydiffraction.analysis.minimizers.emcee import _EmceeProgressReporter + + tracker = _FakeTracker() + reporter = _EmceeProgressReporter(tracker=tracker, total_steps=10, burn_steps=2) + state = SimpleNamespace(log_prob=np.array([-10.0, -5.0], dtype=float)) + + for iteration in range(1, 11): + reporter.report(iteration=iteration, state=state) + + assert len(tracker.updates) > 2 + assert tracker.updates[0].iteration == 1 + assert tracker.updates[0].phase == 'burn-in' + assert any(update.phase == 'sampling' for update in tracker.updates) + assert tracker.updates[-1].iteration == 10 + assert tracker.updates[-1].progress_percent == pytest.approx(100.0) + assert tracker.updates[-1].log_posterior == pytest.approx(-5.0) + assert tracker.updates[-1].reduced_chi2 == pytest.approx(1.23) + + +def test_emcee_progress_reporter_treats_resume_as_sampling_only(): + from easydiffraction.analysis.minimizers.emcee import _EmceeProgressReporter + + tracker = _FakeTracker() + reporter = _EmceeProgressReporter(tracker=tracker, total_steps=5, burn_steps=0) + state = SimpleNamespace(log_prob=np.array([-2.0, -1.0], dtype=float)) + + for iteration in range(1, 6): + reporter.report(iteration=iteration, state=state) + + assert tracker.updates + assert {update.phase for update in tracker.updates} == {'sampling'} + + +def test_sample_with_progress_iterates_sampler_and_reports_each_state(): + from easydiffraction.analysis.minimizers.emcee import EmceeMinimizer + + sampler = _FakeSampler() + reporter = SimpleNamespace(updates=[]) + + def report(*, iteration: int, state: object) -> None: + reporter.updates.append((iteration, state.log_prob.copy())) + + reporter.report = report + + EmceeMinimizer._sample_with_progress( + sampler=sampler, + initial_state=None, + iterations=3, + reporter=reporter, + skip_initial_state_check=True, + ) + + assert sampler.calls == [ + { + 'initial_state': None, + 'iterations': 3, + 'skip_initial_state_check': True, + 'progress': False, + } + ] + assert [iteration for iteration, _log_prob in reporter.updates] == [1, 2, 3] From 59bc73e49f6dc67004947f6f7499d841c23110a1 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sun, 24 May 2026 23:29:41 +0200 Subject: [PATCH 19/65] Restore emcee default parallel sampling --- docs/dev/plans/emcee-minimizer.md | 2 +- .../analysis/categories/minimizer/emcee.py | 2 +- .../analysis/minimizers/emcee.py | 175 +++++++++++++----- .../categories/minimizer/test_emcee.py | 8 +- .../analysis/minimizers/test_emcee.py | 64 ++++++- 5 files changed, 198 insertions(+), 53 deletions(-) diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index 9138f0b18..247d57f4e 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -296,7 +296,7 @@ Mark `[x]` as each step lands. native kwargs (see §"Decisions already made" point 3). - Class-level defaults for emcee-specific values: `sampling_steps=5000`, `burn_in_steps=1000`, `thinning_interval=5`, - `population_size=32`, `parallel_workers=1`, + `population_size=32`, `parallel_workers=0`, `proposal_moves='stretch'`. - `__init__` constructs descriptors via the inherited helpers (`_sampling_steps_descriptor(default)`, etc. from diff --git a/src/easydiffraction/analysis/categories/minimizer/emcee.py b/src/easydiffraction/analysis/categories/minimizer/emcee.py index bd0ac6196..e71d8b1c7 100644 --- a/src/easydiffraction/analysis/categories/minimizer/emcee.py +++ b/src/easydiffraction/analysis/categories/minimizer/emcee.py @@ -20,7 +20,7 @@ DEFAULT_BURN_IN_STEPS = 1000 DEFAULT_THINNING_INTERVAL = 5 DEFAULT_POPULATION_SIZE = 32 -DEFAULT_PARALLEL_WORKERS = 1 +DEFAULT_PARALLEL_WORKERS = 0 DEFAULT_INITIALIZATION_METHOD = InitializationMethodEnum.BALL DEFAULT_PROPOSAL_MOVES = 'stretch' SUPPORTED_PROPOSAL_MOVES = ('stretch', 'de', 'de_snooker', 'walk') diff --git a/src/easydiffraction/analysis/minimizers/emcee.py b/src/easydiffraction/analysis/minimizers/emcee.py index ef06ec323..0b031395b 100644 --- a/src/easydiffraction/analysis/minimizers/emcee.py +++ b/src/easydiffraction/analysis/minimizers/emcee.py @@ -7,6 +7,7 @@ import multiprocessing import os import pickle # noqa: S403 - used only to test whether multiprocessing can serialize a callable. +from dataclasses import dataclass from pathlib import Path from typing import TYPE_CHECKING @@ -32,7 +33,7 @@ DEFAULT_NBURN = 1000 DEFAULT_THIN = 5 DEFAULT_NWALKERS = 32 -DEFAULT_PARALLEL_WORKERS = 1 +DEFAULT_PARALLEL_WORKERS = 0 DEFAULT_INITIALIZATION_METHOD = InitializationMethodEnum.BALL DEFAULT_PROPOSAL_MOVES = 'stretch' MAX_RANDOM_SEED = int(np.iinfo(np.uint32).max) @@ -51,6 +52,67 @@ from collections.abc import Callable +@dataclass(frozen=True, slots=True) +class _EmceePoolContext: + """Resolved emcee pool and log-probability callable.""" + + pool: object | None + log_prob_fn: Callable[[np.ndarray], float] + + +class _EmceeLogProbability: + """Pickle-aware emcee log-probability adapter.""" + + def __init__( + self, + *, + parameters: list[object], + parameter_names: list[str], + objective_function: Callable[[dict[str, object]], object], + ) -> None: + self._parameter_names = parameter_names + self._objective_function = objective_function + self._bounds = { + name: (float(parameter.fit_min), float(parameter.fit_max)) + for name, parameter in zip(parameter_names, parameters, strict=True) + } + + def __call__(self, theta: np.ndarray) -> float: + """Return log posterior for one walker position.""" + for name, value in zip(self._parameter_names, theta, strict=True): + lower_bound, upper_bound = self._bounds[name] + if not lower_bound <= float(value) <= upper_bound: + return -np.inf + + engine_params = { + name: float(value) for name, value in zip(self._parameter_names, theta, strict=True) + } + try: + residuals = np.asarray(self._objective_function(engine_params), dtype=float) + except Exception: # noqa: BLE001 - calculator failures make this proposal invalid. + return -np.inf + if residuals.size == 0 or not np.all(np.isfinite(residuals)): + return -np.inf + return -0.5 * float(np.sum(residuals**2)) + + +_EMCEE_WORKER_LOG_PROB: _EmceeLogProbability | None = None + + +def _set_emcee_worker_log_prob(log_prob: _EmceeLogProbability | None) -> None: + """Set the fork-inherited emcee worker log-probability callable.""" + global _EMCEE_WORKER_LOG_PROB # noqa: PLW0603 + _EMCEE_WORKER_LOG_PROB = log_prob + + +def _emcee_log_prob_worker(theta: np.ndarray) -> float: + """Evaluate log probability in an emcee multiprocessing worker.""" + if _EMCEE_WORKER_LOG_PROB is None: + msg = 'emcee worker log-probability callable has not been initialized.' + raise RuntimeError(msg) + return _EMCEE_WORKER_LOG_PROB(theta) + + class _EmceeProgressReporter: """ Translate emcee iteration states into sampler progress rows. @@ -570,12 +632,12 @@ def _run_solver( parameter_names=parameter_names, objective_function=objective_function, ) - pool = self._build_pool(log_prob) + pool_context = self._build_pool_context(log_prob) try: sampler = self._run_sampler( backend=backend, - log_prob=log_prob, - pool=pool, + log_prob=pool_context.log_prob_fn, + pool=pool_context.pool, parameters=parameters, n_parameters=len(parameter_names), random_seed=random_seed, @@ -597,9 +659,7 @@ def _run_solver( sampler_completed=False, ) finally: - if pool is not None: - pool.close() - pool.join() + self._close_pool_context(pool_context) self.tracker.start_sampler_post_processing() return self._build_success_result( @@ -707,31 +767,13 @@ def _build_log_probability( parameters: list[object], parameter_names: list[str], objective_function: Callable[[dict[str, object]], object], - ) -> Callable[[np.ndarray], float]: + ) -> _EmceeLogProbability: """Return an emcee log-probability adapter.""" - bounds = { - name: (float(parameter.fit_min), float(parameter.fit_max)) - for name, parameter in zip(parameter_names, parameters, strict=True) - } - - def log_prob(theta: np.ndarray) -> float: - for name, value in zip(parameter_names, theta, strict=True): - lower_bound, upper_bound = bounds[name] - if not lower_bound <= float(value) <= upper_bound: - return -np.inf - - engine_params = { - name: float(value) for name, value in zip(parameter_names, theta, strict=True) - } - try: - residuals = np.asarray(objective_function(engine_params), dtype=float) - except Exception: # noqa: BLE001 - calculator failures make this proposal invalid. - return -np.inf - if residuals.size == 0 or not np.all(np.isfinite(residuals)): - return -np.inf - return -0.5 * float(np.sum(residuals**2)) - - return log_prob + return _EmceeLogProbability( + parameters=parameters, + parameter_names=parameter_names, + objective_function=objective_function, + ) def _resolved_sidecar_path(self) -> Path: """Return the HDF sidecar path required by the emcee backend.""" @@ -786,25 +828,70 @@ def _validate_resume( ) raise ValueError(msg) - def _build_pool(self, log_prob: Callable[[np.ndarray], float]) -> object | None: - """Build an emcee map pool for picklable objectives.""" + def _build_pool_context(self, log_prob: _EmceeLogProbability) -> _EmceePoolContext: + """ + Build an emcee map pool for the configured parallel setting. + """ workers = self.parallel_workers if workers == 1: - return None + return _EmceePoolContext(pool=None, log_prob_fn=log_prob) + + worker_count = os.cpu_count() if workers == 0 else workers + if worker_count is None or worker_count <= 1: + return _EmceePoolContext(pool=None, log_prob_fn=log_prob) + + if self._can_pickle(log_prob): + return _EmceePoolContext( + pool=multiprocessing.Pool(worker_count), + log_prob_fn=log_prob, + ) + + if self._fork_context_available(): + try: + _set_emcee_worker_log_prob(log_prob) + pool = multiprocessing.get_context('fork').Pool(worker_count) + except (OSError, RuntimeError): + _set_emcee_worker_log_prob(None) + else: + return _EmceePoolContext( + pool=pool, + log_prob_fn=_emcee_log_prob_worker, + ) + + self._warn_after_tracking( + 'emcee parallel evaluation requires either a picklable objective ' + 'or fork-based multiprocessing; falling back to serial execution.' + ) + return _EmceePoolContext(pool=None, log_prob_fn=log_prob) + @staticmethod + def _close_pool_context(pool_context: _EmceePoolContext) -> None: + """ + Close a resolved emcee pool and clear inherited worker state. + """ + pool = pool_context.pool + try: + if pool is not None: + pool.close() + pool.join() + finally: + _set_emcee_worker_log_prob(None) + + @staticmethod + def _can_pickle(value: object) -> bool: + """ + Return whether a value can be serialized by multiprocessing. + """ try: - pickle.dumps(log_prob) + pickle.dumps(value) except (AttributeError, TypeError, pickle.PickleError): - self._warn_after_tracking( - 'emcee parallel evaluation requires a picklable objective; ' - 'falling back to serial execution.' - ) - return None + return False + return True - worker_count = os.cpu_count() if workers == 0 else workers - if worker_count is None or worker_count <= 1: - return None - return multiprocessing.Pool(worker_count) + @staticmethod + def _fork_context_available() -> bool: + """Return whether fork-based multiprocessing is available.""" + return os.name != 'nt' and 'fork' in multiprocessing.get_all_start_methods() @staticmethod def _resolve_moves(proposal_moves: str) -> object: diff --git a/tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py b/tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py index c0491721e..a90e34349 100644 --- a/tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py +++ b/tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py @@ -5,7 +5,7 @@ from __future__ import annotations -def test_emcee_minimizer_category_defaults_to_serial_parallel_workers(): +def test_emcee_minimizer_category_defaults_to_max_parallel_workers(): from easydiffraction.analysis.categories.minimizer.emcee import ( DEFAULT_PARALLEL_WORKERS, ) @@ -13,6 +13,6 @@ def test_emcee_minimizer_category_defaults_to_serial_parallel_workers(): minimizer = EmceeMinimizer() - assert DEFAULT_PARALLEL_WORKERS == 1 - assert minimizer.parallel_workers.value == 1 - assert minimizer._native_kwargs()['parallel_workers'] == 1 + assert DEFAULT_PARALLEL_WORKERS == 0 + assert minimizer.parallel_workers.value == 0 + assert minimizer._native_kwargs()['parallel_workers'] == 0 diff --git a/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py b/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py index cbd66b198..9a57eac45 100644 --- a/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py +++ b/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py @@ -51,14 +51,72 @@ def sample( yield SimpleNamespace(log_prob=np.array([float(index)], dtype=float)) -def test_emcee_minimizer_defaults_to_serial_parallel_workers(): +def test_emcee_minimizer_defaults_to_max_parallel_workers(): from easydiffraction.analysis.minimizers.emcee import DEFAULT_PARALLEL_WORKERS from easydiffraction.analysis.minimizers.emcee import EmceeMinimizer minimizer = EmceeMinimizer() - assert DEFAULT_PARALLEL_WORKERS == 1 - assert minimizer.parallel_workers == 1 + assert DEFAULT_PARALLEL_WORKERS == 0 + assert minimizer.parallel_workers == 0 + + +def test_emcee_pool_context_uses_fork_worker_for_unpicklable_objective(monkeypatch): + from easydiffraction.analysis.minimizers.emcee import EmceeMinimizer + from easydiffraction.analysis.minimizers.emcee import _emcee_log_prob_worker + + class FakePool: + def __init__(self) -> None: + self.closed = False + self.joined = False + + def close(self) -> None: + self.closed = True + + def join(self) -> None: + self.joined = True + + class FakeContext: + def __init__(self) -> None: + self.pool = FakePool() + self.worker_count = None + + def Pool(self, worker_count: int) -> FakePool: # noqa: N802 + self.worker_count = worker_count + return self.pool + + fake_context = FakeContext() + minimizer = EmceeMinimizer() + minimizer.parallel_workers = 2 + + monkeypatch.setattr( + EmceeMinimizer, + '_fork_context_available', + staticmethod(lambda: True), + ) + monkeypatch.setattr( + 'easydiffraction.analysis.minimizers.emcee.multiprocessing.get_context', + lambda name: fake_context, + ) + + log_prob = minimizer._build_log_probability( + parameters=[SimpleNamespace(fit_min=-10.0, fit_max=10.0)], + parameter_names=['p'], + objective_function=lambda values: np.array([values['p']], dtype=float), + ) + pool_context = minimizer._build_pool_context(log_prob) + + assert fake_context.worker_count == 2 + assert pool_context.pool is fake_context.pool + assert pool_context.log_prob_fn is _emcee_log_prob_worker + assert pool_context.log_prob_fn(np.array([2.0], dtype=float)) == pytest.approx(-2.0) + + minimizer._close_pool_context(pool_context) + + assert fake_context.pool.closed is True + assert fake_context.pool.joined is True + with pytest.raises(RuntimeError, match='has not been initialized'): + _emcee_log_prob_worker(np.array([2.0], dtype=float)) def test_emcee_progress_reporter_emits_burn_in_and_sampling_updates(): From e48c459ca5249ec486f0efc2a93a6dd5cc1d42e3 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sun, 24 May 2026 23:33:13 +0200 Subject: [PATCH 20/65] Align emcee sampling progress totals --- .../analysis/minimizers/emcee.py | 16 +++++----- .../analysis/minimizers/test_emcee.py | 30 +++++++++++++++++++ 2 files changed, 39 insertions(+), 7 deletions(-) diff --git a/src/easydiffraction/analysis/minimizers/emcee.py b/src/easydiffraction/analysis/minimizers/emcee.py index 0b031395b..cc7209317 100644 --- a/src/easydiffraction/analysis/minimizers/emcee.py +++ b/src/easydiffraction/analysis/minimizers/emcee.py @@ -661,8 +661,7 @@ def _run_solver( finally: self._close_pool_context(pool_context) - self.tracker.start_sampler_post_processing() - return self._build_success_result( + result = self._build_success_result( sampler=sampler, backend=backend, parameter_names=parameter_names, @@ -671,6 +670,8 @@ def _run_solver( starting_values=kwargs['starting_values'], starting_uncertainties=kwargs['starting_uncertainties'], ) + self.tracker.start_sampler_post_processing() + return result def _run_sampler( # noqa: PLR0913 self, @@ -724,7 +725,7 @@ def _run_sampler( # noqa: PLR0913 self._sample_with_progress( sampler=sampler, initial_state=initial_state, - iterations=self.nsteps, + iterations=total_iterations, reporter=reporter, skip_initial_state_check=False, ) @@ -790,7 +791,7 @@ def _resolved_total_iterations( ) -> int: """Return the total iterations expected for progress display.""" if not resume: - return self.nsteps + return self.nsteps + self.nburn + 1 return self._validated_positive_integer('extra_steps', extra_steps) def _validate_walker_count(self, *, n_parameters: int) -> None: @@ -934,10 +935,10 @@ def _sampler_settings( n_parameters: int, ) -> dict[str, object]: """Build sampler settings recorded in results.""" - samples = total_steps * self.nwalkers * n_parameters + samples = self.nsteps * self.nwalkers * n_parameters return { 'random_seed': int(random_seed), - 'steps': int(total_steps), + 'steps': int(self.nsteps), 'burn': int(self.nburn), 'thin': int(self.thin), 'pop': int(self.nwalkers), @@ -945,7 +946,8 @@ def _sampler_settings( 'init': self.initialization_method.value, 'proposal_moves': self.proposal_moves, 'samples': int(samples), - 'nsteps': int(total_steps), + 'total_steps': int(total_steps), + 'nsteps': int(self.nsteps), 'nburn': int(self.nburn), 'nwalkers': int(self.nwalkers), 'parallel_workers': int(self.parallel_workers), diff --git a/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py b/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py index 9a57eac45..4d1a151f1 100644 --- a/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py +++ b/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py @@ -153,6 +153,36 @@ def test_emcee_progress_reporter_treats_resume_as_sampling_only(): assert {update.phase for update in tracker.updates} == {'sampling'} +def test_emcee_total_iterations_adds_burn_in_and_initial_generation(): + from easydiffraction.analysis.minimizers.emcee import EmceeMinimizer + + minimizer = EmceeMinimizer() + minimizer.nsteps = 100 + minimizer.nburn = 20 + + assert minimizer._resolved_total_iterations(resume=False, extra_steps=None) == 121 + assert minimizer._resolved_total_iterations(resume=True, extra_steps=50) == 50 + + +def test_emcee_sampler_settings_record_sampling_and_total_steps(): + from easydiffraction.analysis.minimizers.emcee import EmceeMinimizer + + minimizer = EmceeMinimizer() + minimizer.nsteps = 100 + minimizer.nburn = 20 + + settings = minimizer._sampler_settings( + random_seed=123, + total_steps=121, + n_parameters=2, + ) + + assert settings['steps'] == 100 + assert settings['burn'] == 20 + assert settings['total_steps'] == 121 + assert settings['samples'] == 100 * minimizer.nwalkers * 2 + + def test_sample_with_progress_iterates_sampler_and_reports_each_state(): from easydiffraction.analysis.minimizers.emcee import EmceeMinimizer From 6dd7690b64fcfb707d6ea591b24580d4826f1221 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sun, 24 May 2026 23:42:58 +0200 Subject: [PATCH 21/65] Trim deterministic fit-parameter CIF columns --- .../adrs/accepted/analysis-cif-fit-state.md | 4 ++ .../categories/fit_parameters/default.py | 41 +++++++++++++ src/easydiffraction/io/cif/serialize.py | 24 +++++--- .../categories/test_fit_parameters.py | 59 +++++++++++++++++++ 4 files changed, 120 insertions(+), 8 deletions(-) diff --git a/docs/dev/adrs/accepted/analysis-cif-fit-state.md b/docs/dev/adrs/accepted/analysis-cif-fit-state.md index 0cc5e658a..c5a03b7be 100644 --- a/docs/dev/adrs/accepted/analysis-cif-fit-state.md +++ b/docs/dev/adrs/accepted/analysis-cif-fit-state.md @@ -66,6 +66,10 @@ pre-fit scalar snapshots: - `fit_bounds_uncertainty_multiplier` - `start_value` - `start_uncertainty` + +For Bayesian fit projections, `_fit_parameter` also stores per-parameter +posterior summaries: + - `posterior_best_sample_value` - `posterior_median` - `posterior_uncertainty` diff --git a/src/easydiffraction/analysis/categories/fit_parameters/default.py b/src/easydiffraction/analysis/categories/fit_parameters/default.py index 2eed6ee1f..02662ca6e 100644 --- a/src/easydiffraction/analysis/categories/fit_parameters/default.py +++ b/src/easydiffraction/analysis/categories/fit_parameters/default.py @@ -4,9 +4,12 @@ from __future__ import annotations +from typing import ClassVar + import numpy as np from easydiffraction.analysis.categories.fit_parameters.factory import FitParametersFactory +from easydiffraction.analysis.enums import FitResultKindEnum from easydiffraction.core.category import CategoryCollection from easydiffraction.core.category import CategoryItem from easydiffraction.core.metadata import TypeInfo @@ -23,6 +26,25 @@ class FitParameterItem(CategoryItem): _category_code = 'fit_parameter' _category_entry_name = 'param_unique_name' + _control_descriptor_names: ClassVar[tuple[str, ...]] = ( + 'param_unique_name', + 'fit_min', + 'fit_max', + 'fit_bounds_uncertainty_multiplier', + 'start_value', + 'start_uncertainty', + ) + _posterior_descriptor_names: ClassVar[tuple[str, ...]] = ( + 'posterior_best_sample_value', + 'posterior_median', + 'posterior_uncertainty', + 'posterior_interval_68_low', + 'posterior_interval_68_high', + 'posterior_interval_95_low', + 'posterior_interval_95_high', + 'posterior_gelman_rubin', + 'posterior_effective_sample_size_bulk', + ) def __init__(self) -> None: super().__init__() @@ -332,6 +354,25 @@ class FitParameters(CategoryCollection): def __init__(self) -> None: super().__init__(item_type=FitParameterItem) + def _include_posterior_cif_descriptors(self) -> bool: + """Return whether CIF output includes posterior columns.""" + parent = getattr(self, '_parent', None) + fit_result = getattr(parent, 'fit_result', None) + result_kind = getattr(getattr(fit_result, 'result_kind', None), 'value', None) + if result_kind is not None: + return result_kind == FitResultKindEnum.BAYESIAN.value + return any(item.has_posterior_summary() for item in self) + + def _cif_loop_parameters(self, item: FitParameterItem) -> list[object]: + """Return CIF loop descriptors for the current fit kind.""" + descriptor_names = FitParameterItem._control_descriptor_names + if self._include_posterior_cif_descriptors(): + descriptor_names = ( + *descriptor_names, + *FitParameterItem._posterior_descriptor_names, + ) + return [getattr(item, name) for name in descriptor_names] + def create( self, *, diff --git a/src/easydiffraction/io/cif/serialize.py b/src/easydiffraction/io/cif/serialize.py index fcd209d08..5ffad9faf 100644 --- a/src/easydiffraction/io/cif/serialize.py +++ b/src/easydiffraction/io/cif/serialize.py @@ -179,11 +179,11 @@ def category_item_to_cif(item: object) -> str: def _validate_loop_tags( - item: object, + parameters: list[GenericDescriptorBase], header_tags: list[str], ) -> None: """Log an error if any row tag disagrees with *header_tags*.""" - for col, p in enumerate(item.parameters): + for col, p in enumerate(parameters): tag = p._cif_handler.names[0] # type: ignore[attr-defined] if tag != header_tags[col]: log.error( @@ -197,6 +197,7 @@ def _validate_loop_tags( def _emit_loop_rows( items: list, row_fn: object, + row_parameters_fn: object, header_tags: list[str], max_display: int | None, ) -> list[str]: @@ -205,15 +206,15 @@ def _emit_loop_rows( if max_display is not None and len(items) > max_display: half = max_display // 2 for item in items[:half]: - _validate_loop_tags(item, header_tags) + _validate_loop_tags(row_parameters_fn(item), header_tags) lines.append(' '.join(row_fn(item))) lines.append('...') for item in items[-half:]: - _validate_loop_tags(item, header_tags) + _validate_loop_tags(row_parameters_fn(item), header_tags) lines.append(' '.join(row_fn(item))) else: for item in items: - _validate_loop_tags(item, header_tags) + _validate_loop_tags(row_parameters_fn(item), header_tags) lines.append(' '.join(row_fn(item))) return lines @@ -253,11 +254,18 @@ def category_collection_to_cif( if not len(collection): return '\n'.join(lines) + loop_parameters_hook = getattr(collection, '_cif_loop_parameters', None) + + def _loop_parameters(item: object) -> list[GenericDescriptorBase]: + if loop_parameters_hook is not None: + return list(loop_parameters_hook(item)) + return list(item.parameters) + # Header — use first item's CIF tag names as the canonical columns first_item = next(iter(collection.values())) lines.append('loop_') header_tags: list[str] = [] - for p in first_item.parameters: + for p in _loop_parameters(first_item): tags = p._cif_handler.names # type: ignore[attr-defined] header_tags.append(tags[0]) lines.append(tags[0]) @@ -270,10 +278,10 @@ def _row(item: object) -> list[str]: override = row_hook(item) if override is not None: return override - return [format_param_value(p) for p in item.parameters] + return [format_param_value(p) for p in _loop_parameters(item)] items = list(collection.values()) - lines.extend(_emit_loop_rows(items, _row, header_tags, max_display)) + lines.extend(_emit_loop_rows(items, _row, _loop_parameters, header_tags, max_display)) return '\n'.join(lines) diff --git a/tests/unit/easydiffraction/analysis/categories/test_fit_parameters.py b/tests/unit/easydiffraction/analysis/categories/test_fit_parameters.py index e3d0cedde..b96bbe67d 100644 --- a/tests/unit/easydiffraction/analysis/categories/test_fit_parameters.py +++ b/tests/unit/easydiffraction/analysis/categories/test_fit_parameters.py @@ -2,6 +2,8 @@ # SPDX-License-Identifier: BSD-3-Clause """Tests for analysis/categories/fit_parameters/.""" +from types import SimpleNamespace + def test_fit_parameters_factory_create(): from easydiffraction.analysis.categories.fit_parameters.default import FitParameters @@ -11,3 +13,60 @@ def test_fit_parameters_factory_create(): assert FitParametersFactory.default_tag() == 'default' assert isinstance(collection, FitParameters) + + +def _fit_parameters_with_parent_result_kind(result_kind: str): + from easydiffraction.analysis.categories.fit_parameters.default import FitParameters + + collection = FitParameters() + collection.create( + param_unique_name='cosio.cell.length_a', + fit_min=-1.0, + fit_max=1.0, + start_value=10.3, + start_uncertainty=None, + ) + collection._parent = SimpleNamespace( + fit_result=SimpleNamespace( + result_kind=SimpleNamespace(value=result_kind), + ), + ) + return collection + + +def test_fit_parameters_cif_omits_posterior_columns_for_deterministic_result(): + from easydiffraction.analysis.enums import FitResultKindEnum + + collection = _fit_parameters_with_parent_result_kind(FitResultKindEnum.DETERMINISTIC.value) + + cif_text = collection.as_cif + + assert '_fit_parameter.start_value' in cif_text + assert '_fit_parameter.posterior_median' not in cif_text + assert '_fit_parameter.posterior_effective_sample_size_bulk' not in cif_text + + +def test_fit_parameters_cif_keeps_posterior_columns_for_bayesian_result(): + from easydiffraction.analysis.enums import FitResultKindEnum + from easydiffraction.core.posterior import PosteriorParameterSummary + + collection = _fit_parameters_with_parent_result_kind(FitResultKindEnum.BAYESIAN.value) + collection.set_posterior_summary( + PosteriorParameterSummary( + unique_name='cosio.cell.length_a', + display_name='a', + best_sample_value=10.1, + median=10.2, + standard_deviation=0.3, + interval_68=(9.9, 10.5), + interval_95=(9.7, 10.7), + ess_bulk=80.0, + r_hat=1.01, + ) + ) + + cif_text = collection.as_cif + + assert '_fit_parameter.posterior_median' in cif_text + assert '_fit_parameter.posterior_effective_sample_size_bulk' in cif_text + assert '10.2' in cif_text From 40ca4623cc300a70bb14cb3859fecc3a5792ffb5 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sun, 24 May 2026 23:48:12 +0200 Subject: [PATCH 22/65] Omit empty fit bounds multiplier column --- .../dev/adrs/accepted/analysis-cif-fit-state.md | 6 +++++- .../categories/fit_parameters/default.py | 17 ++++++++++++++++- .../analysis/categories/test_fit_parameters.py | 13 +++++++++++++ 3 files changed, 34 insertions(+), 2 deletions(-) diff --git a/docs/dev/adrs/accepted/analysis-cif-fit-state.md b/docs/dev/adrs/accepted/analysis-cif-fit-state.md index c5a03b7be..db4366764 100644 --- a/docs/dev/adrs/accepted/analysis-cif-fit-state.md +++ b/docs/dev/adrs/accepted/analysis-cif-fit-state.md @@ -63,10 +63,14 @@ pre-fit scalar snapshots: - `param_unique_name` - `fit_min` - `fit_max` -- `fit_bounds_uncertainty_multiplier` - `start_value` - `start_uncertainty` +When any row has uncertainty-derived bounds, `_fit_parameter` also stores +the provenance field: + +- `fit_bounds_uncertainty_multiplier` + For Bayesian fit projections, `_fit_parameter` also stores per-parameter posterior summaries: diff --git a/src/easydiffraction/analysis/categories/fit_parameters/default.py b/src/easydiffraction/analysis/categories/fit_parameters/default.py index 02662ca6e..ab0dbc1fb 100644 --- a/src/easydiffraction/analysis/categories/fit_parameters/default.py +++ b/src/easydiffraction/analysis/categories/fit_parameters/default.py @@ -30,10 +30,12 @@ class FitParameterItem(CategoryItem): 'param_unique_name', 'fit_min', 'fit_max', - 'fit_bounds_uncertainty_multiplier', 'start_value', 'start_uncertainty', ) + _optional_control_descriptor_names: ClassVar[tuple[str, ...]] = ( + 'fit_bounds_uncertainty_multiplier', + ) _posterior_descriptor_names: ClassVar[tuple[str, ...]] = ( 'posterior_best_sample_value', 'posterior_median', @@ -363,9 +365,22 @@ def _include_posterior_cif_descriptors(self) -> bool: return result_kind == FitResultKindEnum.BAYESIAN.value return any(item.has_posterior_summary() for item in self) + def _include_uncertainty_multiplier_cif_descriptor(self) -> bool: + """Return whether CIF output includes the bounds multiplier.""" + return any( + item.fit_bounds_uncertainty_multiplier.value is not None + for item in self + ) + def _cif_loop_parameters(self, item: FitParameterItem) -> list[object]: """Return CIF loop descriptors for the current fit kind.""" descriptor_names = FitParameterItem._control_descriptor_names + if self._include_uncertainty_multiplier_cif_descriptor(): + descriptor_names = ( + *descriptor_names[:3], + *FitParameterItem._optional_control_descriptor_names, + *descriptor_names[3:], + ) if self._include_posterior_cif_descriptors(): descriptor_names = ( *descriptor_names, diff --git a/tests/unit/easydiffraction/analysis/categories/test_fit_parameters.py b/tests/unit/easydiffraction/analysis/categories/test_fit_parameters.py index b96bbe67d..63b4cfa85 100644 --- a/tests/unit/easydiffraction/analysis/categories/test_fit_parameters.py +++ b/tests/unit/easydiffraction/analysis/categories/test_fit_parameters.py @@ -42,10 +42,23 @@ def test_fit_parameters_cif_omits_posterior_columns_for_deterministic_result(): cif_text = collection.as_cif assert '_fit_parameter.start_value' in cif_text + assert '_fit_parameter.fit_bounds_uncertainty_multiplier' not in cif_text assert '_fit_parameter.posterior_median' not in cif_text assert '_fit_parameter.posterior_effective_sample_size_bulk' not in cif_text +def test_fit_parameters_cif_keeps_uncertainty_multiplier_when_populated(): + from easydiffraction.analysis.enums import FitResultKindEnum + + collection = _fit_parameters_with_parent_result_kind(FitResultKindEnum.DETERMINISTIC.value) + collection['cosio.cell.length_a']._set_fit_bounds_uncertainty_multiplier(4.0) + + cif_text = collection.as_cif + + assert '_fit_parameter.fit_bounds_uncertainty_multiplier' in cif_text + assert '4.' in cif_text + + def test_fit_parameters_cif_keeps_posterior_columns_for_bayesian_result(): from easydiffraction.analysis.enums import FitResultKindEnum from easydiffraction.core.posterior import PosteriorParameterSummary From c4ea21c384f0dbabcbd389dd18232f4928aac468 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sun, 24 May 2026 23:51:58 +0200 Subject: [PATCH 23/65] Suppress duplicate LSQ exit reason --- .../adrs/accepted/analysis-cif-fit-state.md | 4 +++ .../analysis/categories/fit_result/lsq.py | 15 +++++++++ src/easydiffraction/io/cif/serialize.py | 4 ++- .../categories/fit_result/test_lsq.py | 33 +++++++++++++++++++ 4 files changed, 55 insertions(+), 1 deletion(-) diff --git a/docs/dev/adrs/accepted/analysis-cif-fit-state.md b/docs/dev/adrs/accepted/analysis-cif-fit-state.md index db4366764..3d4f1ee21 100644 --- a/docs/dev/adrs/accepted/analysis-cif-fit-state.md +++ b/docs/dev/adrs/accepted/analysis-cif-fit-state.md @@ -112,6 +112,10 @@ Deterministic fit-result classes add compact fit output counts: - `degrees_of_freedom` - `covariance_available` - `correlation_available` + +When the LSQ backend provides a termination reason that differs from the +common `_fit_result.message`, deterministic fit results also store: + - `exit_reason` Do not persist a `_deterministic_parameter_result` category. Final diff --git a/src/easydiffraction/analysis/categories/fit_result/lsq.py b/src/easydiffraction/analysis/categories/fit_result/lsq.py index 9e8e31ec9..e997ffa66 100644 --- a/src/easydiffraction/analysis/categories/fit_result/lsq.py +++ b/src/easydiffraction/analysis/categories/fit_result/lsq.py @@ -211,6 +211,21 @@ def correlation_available(self) -> BoolDescriptor: def _set_correlation_available(self, *, value: bool | None) -> None: self._correlation_available.value = value + def _include_exit_reason_cif_descriptor(self) -> bool: + """Return whether exit_reason adds distinct information.""" + exit_reason = self.exit_reason.value + if exit_reason is None: + return False + return exit_reason != self.message.value + + def _cif_parameters(self) -> list[object]: + """Return LSQ fit-result descriptors active for CIF output.""" + return [ + descriptor + for descriptor in self.parameters + if descriptor is not self.exit_reason or self._include_exit_reason_cif_descriptor() + ] + @property def exit_reason(self) -> StringDescriptor: """Backend exit reason for the persisted deterministic fit.""" diff --git a/src/easydiffraction/io/cif/serialize.py b/src/easydiffraction/io/cif/serialize.py index 5ffad9faf..873aac0ee 100644 --- a/src/easydiffraction/io/cif/serialize.py +++ b/src/easydiffraction/io/cif/serialize.py @@ -174,7 +174,9 @@ def category_item_to_cif(item: object) -> str: Expects ``item.parameters`` iterable of params with ``_cif_handler.names`` and ``value``. """ - lines: list[str] = [param_to_cif(p) for p in item.parameters] + parameters_hook = getattr(item, '_cif_parameters', None) + parameters = parameters_hook() if parameters_hook is not None else item.parameters + lines: list[str] = [param_to_cif(p) for p in parameters] return '\n'.join(lines) diff --git a/tests/unit/easydiffraction/analysis/categories/fit_result/test_lsq.py b/tests/unit/easydiffraction/analysis/categories/fit_result/test_lsq.py index bce6e3670..52cd075e2 100644 --- a/tests/unit/easydiffraction/analysis/categories/fit_result/test_lsq.py +++ b/tests/unit/easydiffraction/analysis/categories/fit_result/test_lsq.py @@ -53,3 +53,36 @@ def test_least_squares_fit_result_round_trips_cif_outputs(): assert restored.covariance_available.value is True assert restored.correlation_available.value is False assert restored.exit_reason.value == 'converged' + + +def test_least_squares_fit_result_omits_duplicate_exit_reason(): + from easydiffraction.analysis.categories.fit_result.lsq import ( + LeastSquaresFitResult, + ) + + fit_result = LeastSquaresFitResult() + fit_result._set_message('Fit succeeded.') + fit_result._set_exit_reason('Fit succeeded.') + + cif_text = fit_result.as_cif + + assert '_fit_result.message "Fit succeeded."' in cif_text + assert '_fit_result.exit_reason' not in cif_text + + +def test_least_squares_fit_result_keeps_distinct_exit_reason(): + from easydiffraction.analysis.categories.fit_result.lsq import ( + LeastSquaresFitResult, + ) + + fit_result = LeastSquaresFitResult() + fit_result._set_message('Fit failed.') + fit_result._set_exit_reason('maximum number of evaluations reached') + + cif_text = fit_result.as_cif + + assert '_fit_result.message "Fit failed."' in cif_text + assert ( + '_fit_result.exit_reason "maximum number of evaluations reached"' + in cif_text + ) From e9ff3dd6bcf6a014c70b81b0b942171f2e5cd8c7 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 00:18:56 +0200 Subject: [PATCH 24/65] Persist Bayesian runtime seed and timing --- .../adrs/accepted/analysis-cif-fit-state.md | 7 +++- .../accepted/minimizer-input-output-split.md | 2 +- src/easydiffraction/analysis/analysis.py | 35 ++++++++++++++--- .../categories/fit_result/bayesian.py | 39 +++++++++++++++++++ src/easydiffraction/analysis/fitting.py | 23 +++++++---- .../categories/fit_result/test_bayesian.py | 29 ++++++++++++++ .../easydiffraction/analysis/test_analysis.py | 35 ++++++++++++++++- .../easydiffraction/analysis/test_fitting.py | 14 +++++-- 8 files changed, 166 insertions(+), 18 deletions(-) diff --git a/docs/dev/adrs/accepted/analysis-cif-fit-state.md b/docs/dev/adrs/accepted/analysis-cif-fit-state.md index 3d4f1ee21..3bd4fa388 100644 --- a/docs/dev/adrs/accepted/analysis-cif-fit-state.md +++ b/docs/dev/adrs/accepted/analysis-cif-fit-state.md @@ -139,11 +139,16 @@ Bayesian fit-result classes store scalar outputs under `_fit_result.*`: - `sampler_completed` - `credible_interval_inner` - `credible_interval_outer` -- `acceptance_rate_mean` +- `resolved_random_seed` - `gelman_rubin_max` - `effective_sample_size_min` - `best_log_posterior` +When the backend reports an acceptance rate, Bayesian fit results also +store: + +- `acceptance_rate_mean` + Bayesian per-parameter posterior summaries are stored on the corresponding `_fit_parameter` rows. Their row order defines the saved posterior parameter order. diff --git a/docs/dev/adrs/accepted/minimizer-input-output-split.md b/docs/dev/adrs/accepted/minimizer-input-output-split.md index 13a33a5be..ee2656fc1 100644 --- a/docs/dev/adrs/accepted/minimizer-input-output-split.md +++ b/docs/dev/adrs/accepted/minimizer-input-output-split.md @@ -159,7 +159,7 @@ live on `FitResultBase`; family-specific fields on the concrete classes: `exit_reason`. - `BayesianFitResult` adds: `point_estimate_name`, `sampler_completed`, `credible_interval_inner`, `credible_interval_outer`, - `acceptance_rate_mean`, `gelman_rubin_max`, + `resolved_random_seed`, `acceptance_rate_mean`, `gelman_rubin_max`, `effective_sample_size_min`, `best_log_posterior`. The three overlapping pairs from §"Context" are resolved by **dropping diff --git a/src/easydiffraction/analysis/analysis.py b/src/easydiffraction/analysis/analysis.py index 04c620480..67d88a412 100644 --- a/src/easydiffraction/analysis/analysis.py +++ b/src/easydiffraction/analysis/analysis.py @@ -714,6 +714,7 @@ def _restore_fit_results_from_projection(self) -> object | None: else (0, 0, 0) ) sampler_settings = self.minimizer._native_kwargs() + resolved_random_seed = self._restored_bayesian_random_seed(sampler_settings) sampler_name = ( 'dream' if self.minimizer.type == MinimizerTypeEnum.BUMPS_DREAM.value @@ -734,7 +735,10 @@ def _restore_fit_results_from_projection(self) -> object | None: float(self.fit_result.credible_interval_inner.value), float(self.fit_result.credible_interval_outer.value), ), - sampler_settings=self._restored_bayesian_sampler_settings(sampler_settings), + sampler_settings=self._restored_bayesian_sampler_settings( + sampler_settings, + random_seed=resolved_random_seed, + ), convergence_diagnostics={ 'converged': False, 'max_r_hat': self.fit_result.gelman_rubin_max.value, @@ -781,8 +785,10 @@ def _restore_fit_results_from_projection(self) -> object | None: def _restored_bayesian_sampler_settings( self, sampler_settings: dict[str, object], + *, + random_seed: object | None = None, ) -> dict[str, object]: - """Return display-oriented sampler settings for restored results.""" + """Return display settings for restored Bayesian results.""" if self.minimizer.type == MinimizerTypeEnum.EMCEE.value: return { 'steps': self._int_sampler_setting(sampler_settings, 'nsteps'), @@ -792,7 +798,7 @@ def _restored_bayesian_sampler_settings( 'parallel': self._int_sampler_setting(sampler_settings, 'parallel_workers'), 'init': str(sampler_settings.get('initialization_method', '')), 'proposal_moves': str(sampler_settings.get('proposal_moves', '')), - 'random_seed': sampler_settings.get('random_seed'), + 'random_seed': random_seed, } return { @@ -802,9 +808,19 @@ def _restored_bayesian_sampler_settings( 'pop': self._int_sampler_setting(sampler_settings, 'pop'), 'parallel': self._int_sampler_setting(sampler_settings, 'parallel'), 'init': str(sampler_settings.get('init', '')), - 'random_seed': sampler_settings.get('random_seed'), + 'random_seed': random_seed, } + def _restored_bayesian_random_seed( + self, + sampler_settings: dict[str, object], + ) -> object | None: + """Return persisted runtime or configured sampler seed.""" + resolved_seed = self.fit_result.resolved_random_seed.value + if resolved_seed is not None: + return int(resolved_seed) + return sampler_settings.get('random_seed') + @staticmethod def _int_sampler_setting( sampler_settings: dict[str, object], @@ -1013,7 +1029,7 @@ def _validate_resume_extra_steps(extra_steps: int | None) -> None: raise ValueError(msg) def _prepare_results_sidecar_for_new_fit(self) -> None: - """Warn and remove persisted sidecar arrays before a fresh fit.""" + """Remove persisted sidecar arrays before a fresh fit.""" project_path = self.project.info.path if project_path is None: return @@ -1758,6 +1774,9 @@ def _store_posterior_fit_projection(self, results: BayesianFitResults) -> None: self.fit_result._set_best_log_posterior(results.best_log_posterior) self.fit_result._set_credible_interval_inner(credible_interval_inner) self.fit_result._set_credible_interval_outer(credible_interval_outer) + self.fit_result._set_resolved_random_seed( + self._bayesian_result_random_seed(results) + ) self.fit_result._set_gelman_rubin_max(convergence.get('max_r_hat')) self.fit_result._set_effective_sample_size_min(convergence.get('min_ess_bulk')) self.fit_result._set_acceptance_rate_mean(convergence.get('acceptance_rate_mean')) @@ -1790,6 +1809,12 @@ def _store_posterior_fit_projection(self, results: BayesianFitResults) -> None: source_kind=FitCorrelationSourceEnum.POSTERIOR, ) + @staticmethod + def _bayesian_result_random_seed(results: BayesianFitResults) -> int | None: + """Return the runtime seed from Bayesian result settings.""" + seed = results.sampler_settings.get('random_seed') + return None if seed is None else int(seed) + def _store_fit_result_projection( self, results: FitResults, diff --git a/src/easydiffraction/analysis/categories/fit_result/bayesian.py b/src/easydiffraction/analysis/categories/fit_result/bayesian.py index 91e1310f1..85410ec33 100644 --- a/src/easydiffraction/analysis/categories/fit_result/bayesian.py +++ b/src/easydiffraction/analysis/categories/fit_result/bayesian.py @@ -11,6 +11,7 @@ from easydiffraction.core.metadata import TypeInfo from easydiffraction.core.validation import AttributeSpec from easydiffraction.core.variable import BoolDescriptor +from easydiffraction.core.variable import IntegerDescriptor from easydiffraction.core.variable import NumericDescriptor from easydiffraction.core.variable import StringDescriptor from easydiffraction.io.cif.handler import CifHandler @@ -30,11 +31,16 @@ class BayesianFitResult(FitResultBase): 'sampler_completed', 'credible_interval_inner', 'credible_interval_outer', + 'resolved_random_seed', 'acceptance_rate_mean', 'gelman_rubin_max', 'effective_sample_size_min', 'best_log_posterior', ) + _optional_result_descriptor_names: ClassVar[tuple[str, ...]] = ( + 'acceptance_rate_mean', + 'resolved_random_seed', + ) _expected_descriptor_names: ClassVar[tuple[str, ...]] = _result_descriptor_names def __init__(self) -> None: @@ -43,6 +49,7 @@ def __init__(self) -> None: self._sampler_completed = self._sampler_completed_descriptor() self._credible_interval_inner = self._credible_interval_inner_descriptor() self._credible_interval_outer = self._credible_interval_outer_descriptor() + self._resolved_random_seed = self._resolved_random_seed_descriptor() self._acceptance_rate_mean = self._acceptance_rate_mean_descriptor() self._gelman_rubin_max = self._gelman_rubin_max_descriptor() self._effective_sample_size_min = self._effective_sample_size_min_descriptor() @@ -98,6 +105,16 @@ def _acceptance_rate_mean_descriptor() -> NumericDescriptor: cif_handler=CifHandler(names=['_fit_result.acceptance_rate_mean']), ) + @staticmethod + def _resolved_random_seed_descriptor() -> IntegerDescriptor: + """Create a resolved-random-seed descriptor.""" + return IntegerDescriptor( + name='resolved_random_seed', + description='Runtime random seed used by the sampler.', + value_spec=AttributeSpec(default=None, allow_none=True), + cif_handler=CifHandler(names=['_fit_result.resolved_random_seed']), + ) + @staticmethod def _gelman_rubin_max_descriptor() -> NumericDescriptor: """Create a Gelman-Rubin descriptor.""" @@ -168,6 +185,15 @@ def _set_credible_interval_outer(self, value: float) -> None: """ self._credible_interval_outer.value = value + @property + def resolved_random_seed(self) -> IntegerDescriptor: + """Runtime random seed used by the sampler.""" + return self._resolved_random_seed + + def _set_resolved_random_seed(self, value: int | None) -> None: + """Set the resolved random seed for internal callers.""" + self._resolved_random_seed.value = value + @property def acceptance_rate_mean(self) -> NumericDescriptor: """Mean sampler acceptance rate.""" @@ -205,3 +231,16 @@ def best_log_posterior(self) -> NumericDescriptor: def _set_best_log_posterior(self, value: float | None) -> None: """Set the best log-posterior for internal callers.""" self._best_log_posterior.value = value + + def _cif_parameters(self) -> list[object]: + """Return Bayesian fit-result descriptors for CIF output.""" + optional_descriptor_ids = { + id(getattr(self, name)) + for name in self._optional_result_descriptor_names + if getattr(self, name).value is None + } + return [ + descriptor + for descriptor in self.parameters + if id(descriptor) not in optional_descriptor_ids + ] diff --git a/src/easydiffraction/analysis/fitting.py b/src/easydiffraction/analysis/fitting.py index 80a024965..fd47dc82c 100644 --- a/src/easydiffraction/analysis/fitting.py +++ b/src/easydiffraction/analysis/fitting.py @@ -178,7 +178,7 @@ def fit( random_seed : int | None, default=None Optional random seed passed to stochastic minimizers. resume : bool, default=False - Whether to resume a sampler state instead of starting a new fit. + Whether to resume a sampler state. extra_steps : int | None, default=None Additional sampler steps for resume-capable minimizers. """ @@ -233,21 +233,21 @@ def fit( resume=resume, extra_steps=extra_steps, ) - # Stop the timer and backfill results.fitting_time now so - # post-processing projects a real duration into persisted - # categories. The live display is still torn down in the - # finally below. - self.minimizer._finalize_timing() self._postprocess_fit_results( analysis=analysis, experiments=experiments, fitted_parameters=params, ) + # Keep the timer open through post-processing so the final + # sampler row and persisted fitting_time include the heavy + # Bayesian projection/cache work. + self.minimizer._finalize_timing() + self._backfill_persisted_fitting_time(analysis) finally: self.minimizer._stop_tracking() def _set_minimizer_sidecar_path(self, analysis: object) -> None: - """Set the analysis results sidecar path on engines that use it.""" + """Set the analysis results sidecar path when supported.""" if analysis is None or not hasattr(self.minimizer, '_sidecar_path'): return @@ -256,6 +256,15 @@ def _set_minimizer_sidecar_path(self, analysis: object) -> None: sidecar_path = None if project_path is None else project_path / 'analysis' / 'results.h5' self.minimizer._sidecar_path = sidecar_path + def _backfill_persisted_fitting_time(self, analysis: object) -> None: + """Update persisted fit-result time after post-processing.""" + if analysis is None or self.results is None: + return + fit_result = getattr(analysis, 'fit_result', None) + set_fitting_time = getattr(fit_result, '_set_fitting_time', None) + if callable(set_fitting_time): + set_fitting_time(self.results.fitting_time) + @staticmethod def _validate_resume_parameter_set( *, diff --git a/tests/unit/easydiffraction/analysis/categories/fit_result/test_bayesian.py b/tests/unit/easydiffraction/analysis/categories/fit_result/test_bayesian.py index 40c1a2657..c9e5e560d 100644 --- a/tests/unit/easydiffraction/analysis/categories/fit_result/test_bayesian.py +++ b/tests/unit/easydiffraction/analysis/categories/fit_result/test_bayesian.py @@ -18,6 +18,7 @@ def test_bayesian_fit_result_defaults_unknown_outputs_to_none(): assert fit_result.sampler_completed.value is None assert fit_result.credible_interval_inner.value == 0.68 assert fit_result.credible_interval_outer.value == 0.95 + assert fit_result.resolved_random_seed.value is None assert fit_result.acceptance_rate_mean.value is None assert fit_result.gelman_rubin_max.value is None assert fit_result.effective_sample_size_min.value is None @@ -34,6 +35,7 @@ def test_bayesian_fit_result_round_trips_cif_outputs(): fit_result._set_sampler_completed(value=True) fit_result._set_credible_interval_inner(0.5) fit_result._set_credible_interval_outer(0.9) + fit_result._set_resolved_random_seed(12345) fit_result._set_acceptance_rate_mean(0.42) fit_result._set_gelman_rubin_max(1.01) fit_result._set_effective_sample_size_min(80) @@ -46,7 +48,34 @@ def test_bayesian_fit_result_round_trips_cif_outputs(): assert restored.sampler_completed.value is True assert restored.credible_interval_inner.value == 0.5 assert restored.credible_interval_outer.value == 0.9 + assert restored.resolved_random_seed.value == 12345 assert restored.acceptance_rate_mean.value == 0.42 assert restored.gelman_rubin_max.value == 1.01 assert restored.effective_sample_size_min.value == 80 assert restored.best_log_posterior.value == -12.5 + + +def test_bayesian_fit_result_omits_optional_unknown_outputs(): + from easydiffraction.analysis.categories.fit_result.bayesian import ( + BayesianFitResult, + ) + + cif_text = BayesianFitResult().as_cif + + assert '_fit_result.acceptance_rate_mean' not in cif_text + assert '_fit_result.resolved_random_seed' not in cif_text + + +def test_bayesian_fit_result_keeps_optional_outputs_when_populated(): + from easydiffraction.analysis.categories.fit_result.bayesian import ( + BayesianFitResult, + ) + + fit_result = BayesianFitResult() + fit_result._set_resolved_random_seed(12345) + fit_result._set_acceptance_rate_mean(0.42) + + cif_text = fit_result.as_cif + + assert '_fit_result.resolved_random_seed 12345' in cif_text + assert '_fit_result.acceptance_rate_mean 0.42' in cif_text diff --git a/tests/unit/easydiffraction/analysis/test_analysis.py b/tests/unit/easydiffraction/analysis/test_analysis.py index 9d7e5b94d..779ae1fa5 100644 --- a/tests/unit/easydiffraction/analysis/test_analysis.py +++ b/tests/unit/easydiffraction/analysis/test_analysis.py @@ -121,6 +121,28 @@ def test_minimizer_type_invalid_assignment_raises_and_preserves_state(): assert a.minimizer.type == initial_type +def test_store_posterior_projection_persists_resolved_random_seed(): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.categories.fit_result.bayesian import BayesianFitResult + from easydiffraction.analysis.fit_helpers.bayesian import BayesianFitResults + + analysis = Analysis(project=_make_project_with_names([])) + analysis._fit_result._parent = None + analysis._fit_result = BayesianFitResult() + analysis._fit_result._parent = analysis + results = BayesianFitResults( + success=True, + convergence_diagnostics={}, + sampler_settings={'random_seed': 12345}, + posterior_samples=None, + posterior_parameter_summaries=[], + ) + + analysis._store_posterior_fit_projection(results) + + assert analysis.fit_result.resolved_random_seed.value == 12345 + + def test_fitting_mode_type_invalid_assignment_raises_and_preserves_state(): import pytest @@ -283,8 +305,19 @@ def fake_fit( verbosity: object, use_physical_limits: bool, random_seed: int | None, + resume: bool, + extra_steps: int | None, ) -> None: - del structures, experiments, analysis, verbosity, use_physical_limits, random_seed + del ( + structures, + experiments, + analysis, + verbosity, + use_physical_limits, + random_seed, + resume, + extra_steps, + ) analysis_obj = fake_fit.analysis_obj analysis_obj.fitter.results = SimpleNamespace( reduced_chi_square=1.23, diff --git a/tests/unit/easydiffraction/analysis/test_fitting.py b/tests/unit/easydiffraction/analysis/test_fitting.py index 72d2c7724..04b4f20ec 100644 --- a/tests/unit/easydiffraction/analysis/test_fitting.py +++ b/tests/unit/easydiffraction/analysis/test_fitting.py @@ -144,30 +144,38 @@ def __init__(self): self.fit_calls: list[dict[str, object]] = [] self.stop_calls = 0 self.tracker = SimpleNamespace(track=lambda residuals, parameters: residuals) + self.result = None def fit(self, params, obj, verbosity=None, **kwargs): del params, obj self.fit_calls.append({'verbosity': verbosity, **kwargs}) - return BayesianFitResults( + self.result = BayesianFitResults( success=True, reduced_chi_square=1.2, convergence_diagnostics={'converged': False}, sampler_settings={'steps': 300}, best_log_posterior=-10.0, ) + return self.result def _stop_tracking(self): + analysis_events.append('stop') self.stop_calls += 1 def _finalize_timing(self): - pass + analysis_events.append('finalize') + self.result.fitting_time = 12.5 analysis_events: list[str] = [] + fit_result = SimpleNamespace( + _set_fitting_time=lambda value: analysis_events.append(('time', value)), + ) analysis = SimpleNamespace( _capture_fit_parameter_state=lambda params: analysis_events.append('capture'), _store_fit_result_projection=lambda results, experiments, fitted_parameters: ( analysis_events.append('store') ), + fit_result=fit_result, ) fitter = Fitter() @@ -187,7 +195,7 @@ def _finalize_timing(self): assert fitter.minimizer.fit_calls[0]['finalize_tracking'] is False assert fitter.minimizer.stop_calls == 1 - assert analysis_events == ['capture', 'store'] + assert analysis_events == ['capture', 'store', 'finalize', ('time', 12.5), 'stop'] def test_fitter_fit_stops_tracking_when_minimizer_fit_raises(monkeypatch): From b6c42a7b612c19fa66c204c852739a23e48f991c Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 00:29:47 +0200 Subject: [PATCH 25/65] Add notebook stop control for fitting --- src/easydiffraction/analysis/analysis.py | 29 +++ .../analysis/minimizers/bumps_dream.py | 3 + .../analysis/minimizers/emcee.py | 27 ++- src/easydiffraction/display/progress.py | 186 ++++++++++++++++++ .../analysis/fit_helpers/test_tracking.py | 70 +++++++ .../analysis/minimizers/test_emcee.py | 105 ++++++++++ .../easydiffraction/analysis/test_analysis.py | 45 +++++ 7 files changed, 460 insertions(+), 5 deletions(-) diff --git a/src/easydiffraction/analysis/analysis.py b/src/easydiffraction/analysis/analysis.py index 67d88a412..0c86fe1d3 100644 --- a/src/easydiffraction/analysis/analysis.py +++ b/src/easydiffraction/analysis/analysis.py @@ -44,6 +44,7 @@ from easydiffraction.core.variable import StringDescriptor from easydiffraction.datablocks.experiment.item.base import intensity_category_for from easydiffraction.display.progress import make_display_handle +from easydiffraction.display.progress import notebook_fit_stop_control from easydiffraction.display.tables import TableRenderer from easydiffraction.io.cif.serialize import analysis_to_cif from easydiffraction.utils.enums import VerbosityEnum @@ -973,6 +974,25 @@ def fit( resume=resume, extra_steps=extra_steps, ) + verb = VerbosityEnum(self.project.verbosity.fit.value) + try: + with notebook_fit_stop_control(verbosity=verb): + self._run_fit_mode( + mode=mode, + resume=resume, + extra_steps=extra_steps, + ) + except KeyboardInterrupt: + self._handle_fit_interrupted(verbosity=verb) + + def _run_fit_mode( + self, + *, + mode: FitModeEnum, + resume: bool, + extra_steps: int | None, + ) -> None: + """Dispatch a validated fit request to the selected mode.""" if mode is FitModeEnum.SINGLE: self._run_single(resume=resume, extra_steps=extra_steps) elif mode is FitModeEnum.JOINT: @@ -984,6 +1004,15 @@ def fit( msg = f'Unknown fit mode: {mode!r}' raise ValueError(msg) + def _handle_fit_interrupted(self, *, verbosity: VerbosityEnum) -> None: + """Clean up in-memory fit state after a user interrupt.""" + self.fit_results = None + self.fitter.results = None + self._clear_persisted_fit_state() + self._prepare_results_sidecar_for_new_fit() + if verbosity is not VerbosityEnum.SILENT: + console.print('⏹️ Fitting stopped by user.') + def _validate_fit_request( self, *, diff --git a/src/easydiffraction/analysis/minimizers/bumps_dream.py b/src/easydiffraction/analysis/minimizers/bumps_dream.py index 225c7ea0d..7bcdd5766 100644 --- a/src/easydiffraction/analysis/minimizers/bumps_dream.py +++ b/src/easydiffraction/analysis/minimizers/bumps_dream.py @@ -737,6 +737,9 @@ def _build_driver( trim=DEFAULT_TRIM, ) driver.clip() + except KeyboardInterrupt: + MPMapper.stop_mapper() + raise except Exception: MPMapper.stop_mapper() raise diff --git a/src/easydiffraction/analysis/minimizers/emcee.py b/src/easydiffraction/analysis/minimizers/emcee.py index cc7209317..3251998fc 100644 --- a/src/easydiffraction/analysis/minimizers/emcee.py +++ b/src/easydiffraction/analysis/minimizers/emcee.py @@ -633,6 +633,7 @@ def _run_solver( objective_function=objective_function, ) pool_context = self._build_pool_context(log_prob) + terminate_pool = False try: sampler = self._run_sampler( backend=backend, @@ -645,6 +646,9 @@ def _run_solver( extra_steps=extra_steps, total_iterations=total_iterations, ) + except KeyboardInterrupt: + terminate_pool = True + raise except EMCEE_FAILURES as error: return self._failure_result( message=f'emcee sampling failed: {error}', @@ -659,7 +663,7 @@ def _run_solver( sampler_completed=False, ) finally: - self._close_pool_context(pool_context) + self._close_pool_context(pool_context, terminate=terminate_pool) result = self._build_success_result( sampler=sampler, @@ -866,18 +870,31 @@ def _build_pool_context(self, log_prob: _EmceeLogProbability) -> _EmceePoolConte return _EmceePoolContext(pool=None, log_prob_fn=log_prob) @staticmethod - def _close_pool_context(pool_context: _EmceePoolContext) -> None: + def _close_pool_context( + pool_context: _EmceePoolContext, + *, + terminate: bool = False, + ) -> None: """ Close a resolved emcee pool and clear inherited worker state. """ pool = pool_context.pool try: - if pool is not None: - pool.close() - pool.join() + EmceeMinimizer._shutdown_pool(pool, terminate=terminate) finally: _set_emcee_worker_log_prob(None) + @staticmethod + def _shutdown_pool(pool: object | None, *, terminate: bool) -> None: + """Close or terminate an emcee multiprocessing pool.""" + if pool is None: + return + if terminate: + pool.terminate() + else: + pool.close() + pool.join() + @staticmethod def _can_pickle(value: object) -> bool: """ diff --git a/src/easydiffraction/display/progress.py b/src/easydiffraction/display/progress.py index 6123e626e..800cd66e6 100644 --- a/src/easydiffraction/display/progress.py +++ b/src/easydiffraction/display/progress.py @@ -5,10 +5,12 @@ from __future__ import annotations import html +import uuid from contextlib import AbstractContextManager from contextlib import suppress from time import monotonic from typing import TYPE_CHECKING +from typing import Self if TYPE_CHECKING: from types import TracebackType @@ -16,9 +18,13 @@ try: from IPython.display import HTML from IPython.display import DisplayHandle + from IPython.display import Javascript + from IPython.display import display except ImportError: # pragma: no cover - optional dependency HTML = None DisplayHandle = None + Javascript = None + display = None from rich.console import Group from rich.live import Live @@ -489,3 +495,183 @@ def activity_indicator( on exit. """ return _ActivityIndicatorContext(label=label, verbosity=verbosity) + + +class NotebookFitStopControl(AbstractContextManager): + """Display a Jupyter stop button for fitting runs.""" + + def __init__(self, *, verbosity: VerbosityEnum) -> None: + self._verbosity = verbosity + self._display_handle: object | None = None + self._element_id = f'ed-fit-stop-{uuid.uuid4().hex}' + + def __enter__(self) -> Self: + """Show the stop button.""" + self.show() + return self + + def __exit__( + self, + exc_type: type[BaseException] | None, + exc_value: BaseException | None, + traceback: TracebackType | None, + ) -> None: + """Update or clear the stop button when leaving the context.""" + del exc_value + del traceback + interrupted = exc_type is not None and issubclass(exc_type, KeyboardInterrupt) + self.close(interrupted=interrupted) + + def show(self) -> None: + """Render the stop button when running in a notebook.""" + if not self._can_display(): + return + + handle = DisplayHandle() + self._display_handle = handle + with suppress(Exception): + handle.display(HTML(self._active_html())) + display(Javascript(self._interrupt_javascript())) + + def close(self, *, interrupted: bool = False) -> None: + """Clear or update the stop button when fitting ends.""" + if self._display_handle is None or HTML is None: + return + + html_content = self._stopped_html() if interrupted else '' + with suppress(Exception): + self._display_handle.update(HTML(html_content)) + self._display_handle = None + + def _can_display(self) -> bool: + return ( + self._verbosity is not VerbosityEnum.SILENT + and in_jupyter() + and DisplayHandle is not None + and HTML is not None + and Javascript is not None + and display is not None + ) + + def _active_html(self) -> str: + return ( + '' + f'
' + f'' + f'' + '
' + ) + + def _stopped_html(self) -> str: + return ( + f'
' + 'Fitting stopped.' + '
' + ) + + def _interrupt_javascript(self) -> str: + button_id = f'{self._element_id}-button' + status_id = f'{self._element_id}-status' + return f""" +(function() {{ + const button = document.getElementById({button_id!r}); + const status = document.getElementById({status_id!r}); + if (!button) {{ + return; + }} + + function setStatus(text) {{ + if (status) {{ + status.textContent = text; + }} + }} + + function executeCommand(commandId) {{ + const app = window.jupyterapp || window.JupyterLab || window.jupyterlab; + if (!app || !app.commands) {{ + return false; + }} + try {{ + app.commands.execute(commandId); + return true; + }} catch (error) {{ + return false; + }} + }} + + function clickInterruptButton() {{ + const selectors = [ + '[data-command="kernelmenu:interrupt"]', + '[data-command="notebook:interrupt-kernel"]', + 'button[title*="Interrupt"]', + 'button[aria-label*="Interrupt"]' + ]; + for (const selector of selectors) {{ + const element = document.querySelector(selector); + if (element && element !== button) {{ + element.click(); + return true; + }} + }} + return false; + }} + + button.addEventListener('click', function() {{ + button.disabled = true; + setStatus('Stopping...'); + let interrupted = false; + if (window.Jupyter && Jupyter.notebook && Jupyter.notebook.kernel) {{ + try {{ + Jupyter.notebook.kernel.interrupt(); + interrupted = true; + }} catch (error) {{ + interrupted = false; + }} + }} + interrupted = interrupted || + executeCommand('kernelmenu:interrupt') || + executeCommand('notebook:interrupt-kernel') || + clickInterruptButton(); + if (!interrupted) {{ + button.disabled = false; + setStatus('Use Kernel > Interrupt to stop this fit.'); + }} + }}); +}})(); +""" + + +def notebook_fit_stop_control( + *, + verbosity: VerbosityEnum, +) -> NotebookFitStopControl: + """Return a notebook stop-control context for fitting runs.""" + return NotebookFitStopControl(verbosity=verbosity) diff --git a/tests/unit/easydiffraction/analysis/fit_helpers/test_tracking.py b/tests/unit/easydiffraction/analysis/fit_helpers/test_tracking.py index ff9df3288..97b675969 100644 --- a/tests/unit/easydiffraction/analysis/fit_helpers/test_tracking.py +++ b/tests/unit/easydiffraction/analysis/fit_helpers/test_tracking.py @@ -1,6 +1,8 @@ # SPDX-FileCopyrightText: 2025 EasyScience contributors # SPDX-License-Identifier: BSD-3-Clause +from types import SimpleNamespace + import numpy as np @@ -124,3 +126,71 @@ def test_tracker_sampler_post_processing_adds_final_status_row(): assert tracker._df_rows[-1][1] == '' assert tracker._df_rows[-1][3] == '' assert tracker._df_rows[-1][4] == 'post-processing' + + +def test_notebook_fit_stop_control_renders_interrupt_button(monkeypatch): + import easydiffraction.display.progress as progress_mod + from easydiffraction.utils.enums import VerbosityEnum + + html_updates: list[str] = [] + javascript_outputs: list[str] = [] + + class FakeDisplayHandle: + def display(self, value: SimpleNamespace) -> None: + html_updates.append(value.data) + + def update(self, value: SimpleNamespace) -> None: + html_updates.append(value.data) + + monkeypatch.setattr(progress_mod, 'in_jupyter', lambda: True) + monkeypatch.setattr(progress_mod, 'HTML', lambda data: SimpleNamespace(data=data)) + monkeypatch.setattr( + progress_mod, + 'Javascript', + lambda data: SimpleNamespace(data=data), + ) + monkeypatch.setattr(progress_mod, 'DisplayHandle', FakeDisplayHandle) + monkeypatch.setattr( + progress_mod, + 'display', + lambda value: javascript_outputs.append(value.data), + ) + + with progress_mod.notebook_fit_stop_control(verbosity=VerbosityEnum.FULL): + pass + + assert 'Stop fitting' in html_updates[0] + assert "kernelmenu:interrupt" in javascript_outputs[0] + assert html_updates[-1] == '' + + +def test_notebook_fit_stop_control_marks_interrupted(monkeypatch): + import easydiffraction.display.progress as progress_mod + from easydiffraction.utils.enums import VerbosityEnum + + html_updates: list[str] = [] + + class FakeDisplayHandle: + def display(self, value: SimpleNamespace) -> None: + html_updates.append(value.data) + + def update(self, value: SimpleNamespace) -> None: + html_updates.append(value.data) + + monkeypatch.setattr(progress_mod, 'in_jupyter', lambda: True) + monkeypatch.setattr(progress_mod, 'HTML', lambda data: SimpleNamespace(data=data)) + monkeypatch.setattr( + progress_mod, + 'Javascript', + lambda data: SimpleNamespace(data=data), + ) + monkeypatch.setattr(progress_mod, 'DisplayHandle', FakeDisplayHandle) + monkeypatch.setattr(progress_mod, 'display', lambda value: None) + + try: + with progress_mod.notebook_fit_stop_control(verbosity=VerbosityEnum.FULL): + raise KeyboardInterrupt + except KeyboardInterrupt: + pass + + assert 'Fitting stopped.' in html_updates[-1] diff --git a/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py b/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py index 4d1a151f1..455e06555 100644 --- a/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py +++ b/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py @@ -119,6 +119,111 @@ def Pool(self, worker_count: int) -> FakePool: # noqa: N802 _emcee_log_prob_worker(np.array([2.0], dtype=float)) +def test_emcee_pool_context_terminates_on_interrupt_cleanup(): + from easydiffraction.analysis.minimizers.emcee import EmceeMinimizer + from easydiffraction.analysis.minimizers.emcee import _EmceePoolContext + + class FakePool: + def __init__(self) -> None: + self.closed = False + self.terminated = False + self.joined = False + + def close(self) -> None: + self.closed = True + + def terminate(self) -> None: + self.terminated = True + + def join(self) -> None: + self.joined = True + + fake_pool = FakePool() + pool_context = _EmceePoolContext(pool=fake_pool, log_prob_fn=lambda values: 0.0) + + EmceeMinimizer._close_pool_context(pool_context, terminate=True) + + assert fake_pool.terminated is True + assert fake_pool.closed is False + assert fake_pool.joined is True + + +def test_emcee_run_solver_terminates_pool_when_interrupted(monkeypatch, tmp_path): + import easydiffraction.analysis.minimizers.emcee as emcee_mod + from easydiffraction.analysis.minimizers.emcee import EmceeMinimizer + from easydiffraction.analysis.minimizers.emcee import _EmceePoolContext + + class FakeBackend: + iteration = 0 + + def __init__( + self, + path: str, + *, + name: str, + read_only: bool, + ) -> None: + del path, name, read_only + + class FakePool: + def __init__(self) -> None: + self.terminated = False + self.joined = False + + def terminate(self) -> None: + self.terminated = True + + def join(self) -> None: + self.joined = True + + fake_pool = FakePool() + minimizer = EmceeMinimizer() + minimizer.tracker = SimpleNamespace(start_sampler_pre_processing=lambda **kwargs: None) + + monkeypatch.setattr( + emcee_mod.emcee.backends, + 'HDFBackend', + FakeBackend, + ) + monkeypatch.setattr( + minimizer, + '_resolved_sidecar_path', + lambda: tmp_path / 'analysis' / 'results.h5', + ) + monkeypatch.setattr(minimizer, '_validate_walker_count', lambda **kwargs: None) + monkeypatch.setattr( + minimizer, + '_build_log_probability', + lambda **kwargs: object(), + ) + monkeypatch.setattr( + minimizer, + '_build_pool_context', + lambda log_prob: _EmceePoolContext(pool=fake_pool, log_prob_fn=lambda values: 0.0), + ) + monkeypatch.setattr( + minimizer, + '_run_sampler', + lambda **kwargs: (_ for _ in ()).throw(KeyboardInterrupt), + ) + + with pytest.raises(KeyboardInterrupt): + minimizer._run_solver( + lambda values: np.array([0.0], dtype=float), + parameters=[SimpleNamespace(fit_min=-1.0, fit_max=1.0)], + parameter_names=['p'], + parameter_display_names=['p'], + random_seed=1, + resume=False, + extra_steps=None, + starting_values=[0.0], + starting_uncertainties=[None], + ) + + assert fake_pool.terminated is True + assert fake_pool.joined is True + + def test_emcee_progress_reporter_emits_burn_in_and_sampling_updates(): from easydiffraction.analysis.minimizers.emcee import _EmceeProgressReporter diff --git a/tests/unit/easydiffraction/analysis/test_analysis.py b/tests/unit/easydiffraction/analysis/test_analysis.py index 779ae1fa5..ea51940be 100644 --- a/tests/unit/easydiffraction/analysis/test_analysis.py +++ b/tests/unit/easydiffraction/analysis/test_analysis.py @@ -143,6 +143,51 @@ def test_store_posterior_projection_persists_resolved_random_seed(): assert analysis.fit_result.resolved_random_seed.value == 12345 +def test_fit_interrupt_cleans_state_and_prints_message(monkeypatch, capsys): + from easydiffraction.analysis import analysis as analysis_mod + from easydiffraction.analysis.analysis import Analysis + + events: list[object] = [] + + class FakeStopControl: + def __enter__(self) -> object: + events.append('enter') + return self + + def __exit__(self, exc_type, exc_value, traceback) -> None: + del exc_value + del traceback + events.append(exc_type) + + analysis = Analysis(project=_make_project_with_names([])) + analysis.project.verbosity = SimpleNamespace(fit=SimpleNamespace(value='full')) + analysis.fit_results = object() + analysis.fitter.results = object() + + monkeypatch.setattr( + analysis_mod, + 'notebook_fit_stop_control', + lambda *, verbosity: FakeStopControl(), + ) + monkeypatch.setattr( + analysis, + '_run_single', + lambda **kwargs: (_ for _ in ()).throw(KeyboardInterrupt), + ) + monkeypatch.setattr( + analysis, + '_prepare_results_sidecar_for_new_fit', + lambda: events.append('sidecar-cleanup'), + ) + + analysis.fit() + + assert events == ['enter', KeyboardInterrupt, 'sidecar-cleanup'] + assert analysis.fit_results is None + assert analysis.fitter.results is None + assert 'Fitting stopped by user.' in capsys.readouterr().out + + def test_fitting_mode_type_invalid_assignment_raises_and_preserves_state(): import pytest From b80bcf76eea1366f986b9b4e4b1ab20c9a5b5553 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 00:33:19 +0200 Subject: [PATCH 26/65] Use Jupyter server interrupt for stop fitting --- src/easydiffraction/display/progress.py | 170 ++++++++++++++---- .../analysis/fit_helpers/test_tracking.py | 13 +- 2 files changed, 148 insertions(+), 35 deletions(-) diff --git a/src/easydiffraction/display/progress.py b/src/easydiffraction/display/progress.py index 800cd66e6..7d9affd3c 100644 --- a/src/easydiffraction/display/progress.py +++ b/src/easydiffraction/display/progress.py @@ -8,6 +8,7 @@ import uuid from contextlib import AbstractContextManager from contextlib import suppress +from pathlib import Path from time import monotonic from typing import TYPE_CHECKING from typing import Self @@ -504,6 +505,7 @@ def __init__(self, *, verbosity: VerbosityEnum) -> None: self._verbosity = verbosity self._display_handle: object | None = None self._element_id = f'ed-fit-stop-{uuid.uuid4().hex}' + self._kernel_id = self._current_kernel_id() def __enter__(self) -> Self: """Show the stop button.""" @@ -600,10 +602,12 @@ def _stopped_html(self) -> str: def _interrupt_javascript(self) -> str: button_id = f'{self._element_id}-button' status_id = f'{self._element_id}-status' + kernel_id = self._kernel_id return f""" (function() {{ const button = document.getElementById({button_id!r}); const status = document.getElementById({status_id!r}); + const kernelId = {kernel_id!r}; if (!button) {{ return; }} @@ -614,53 +618,114 @@ def _interrupt_javascript(self) -> str: }} }} - function executeCommand(commandId) {{ - const app = window.jupyterapp || window.JupyterLab || window.jupyterlab; - if (!app || !app.commands) {{ - return false; + function pageConfig() {{ + const element = document.getElementById('jupyter-config-data'); + if (!element || !element.textContent) {{ + return {{}}; }} try {{ - app.commands.execute(commandId); - return true; + return JSON.parse(element.textContent); }} catch (error) {{ - return false; + return {{}}; }} }} - function clickInterruptButton() {{ - const selectors = [ - '[data-command="kernelmenu:interrupt"]', - '[data-command="notebook:interrupt-kernel"]', - 'button[title*="Interrupt"]', - 'button[aria-label*="Interrupt"]' - ]; - for (const selector of selectors) {{ - const element = document.querySelector(selector); - if (element && element !== button) {{ - element.click(); - return true; + function baseUrl(config) {{ + const configured = config.baseUrl || config.base_url || + (window.Jupyter && Jupyter.notebook && Jupyter.notebook.base_url); + if (configured) {{ + return configured.endsWith('/') ? configured : configured + '/'; + }} + const markers = ['/lab/', '/notebooks/', '/tree/']; + for (const marker of markers) {{ + const index = window.location.pathname.indexOf(marker); + if (index >= 0) {{ + return window.location.pathname.slice(0, index + 1); + }} + }} + return '/'; + }} + + function token(config) {{ + return config.token || new URLSearchParams(window.location.search).get('token') || ''; + }} + + function cookie(name) {{ + const prefix = name + '='; + for (const part of document.cookie.split(';')) {{ + const trimmed = part.trim(); + if (trimmed.startsWith(prefix)) {{ + return decodeURIComponent(trimmed.slice(prefix.length)); }} }} - return false; + return ''; + }} + + function notebookPath() {{ + const decoded = decodeURIComponent(window.location.pathname); + const markers = ['/lab/tree/', '/notebooks/', '/tree/']; + for (const marker of markers) {{ + const index = decoded.indexOf(marker); + if (index >= 0) {{ + return decoded.slice(index + marker.length); + }} + }} + return ''; + }} + + async function kernelFromSessions(config) {{ + const url = new URL(baseUrl(config) + 'api/sessions', window.location.origin); + const authToken = token(config); + if (authToken) {{ + url.searchParams.set('token', authToken); + }} + const response = await fetch(url, {{credentials: 'same-origin'}}); + if (!response.ok) {{ + return ''; + }} + const sessions = await response.json(); + const path = notebookPath(); + const session = sessions.find((item) => item.path === path) || sessions[0]; + return session && session.kernel ? session.kernel.id : ''; + }} + + async function interruptKernel(config, resolvedKernelId) {{ + const url = new URL( + baseUrl(config) + 'api/kernels/' + resolvedKernelId + '/interrupt', + window.location.origin + ); + const authToken = token(config); + if (authToken) {{ + url.searchParams.set('token', authToken); + }} + const xsrfToken = cookie('_xsrf'); + const headers = {{}}; + if (xsrfToken) {{ + headers['X-XSRFToken'] = xsrfToken; + }} + const response = await fetch(url, {{ + method: 'POST', + credentials: 'same-origin', + headers: headers + }}); + return response.ok; }} - button.addEventListener('click', function() {{ + button.addEventListener('click', async function() {{ button.disabled = true; setStatus('Stopping...'); - let interrupted = false; - if (window.Jupyter && Jupyter.notebook && Jupyter.notebook.kernel) {{ - try {{ - Jupyter.notebook.kernel.interrupt(); - interrupted = true; - }} catch (error) {{ - interrupted = false; + const config = pageConfig(); + try {{ + const resolvedKernelId = kernelId || await kernelFromSessions(config); + if (!resolvedKernelId) {{ + throw new Error('Could not resolve the current kernel id.'); }} - }} - interrupted = interrupted || - executeCommand('kernelmenu:interrupt') || - executeCommand('notebook:interrupt-kernel') || - clickInterruptButton(); - if (!interrupted) {{ + const interrupted = await interruptKernel(config, resolvedKernelId); + if (!interrupted) {{ + throw new Error('Jupyter Server rejected the interrupt request.'); + }} + setStatus('Interrupt sent...'); + }} catch (error) {{ button.disabled = false; setStatus('Use Kernel > Interrupt to stop this fit.'); }} @@ -668,6 +733,43 @@ def _interrupt_javascript(self) -> str: }})(); """ + @staticmethod + def _current_kernel_id() -> str: + """Return the active ipykernel id when available.""" + try: + from IPython import get_ipython # type: ignore[import-not-found] # noqa: PLC0415 + except ImportError: # pragma: no cover - optional dependency + return '' + + shell = get_ipython() + kernel = getattr(shell, 'kernel', None) + kernel_id = getattr(kernel, 'kernel_id', None) + if kernel_id: + return str(kernel_id) + + try: + from ipykernel.connect import ( # type: ignore[import-not-found] # noqa: PLC0415 + get_connection_file, + ) + except ImportError: # pragma: no cover - optional dependency + return '' + + with suppress(Exception): + return NotebookFitStopControl._kernel_id_from_connection_file( + get_connection_file() + ) + return '' + + @staticmethod + def _kernel_id_from_connection_file(connection_file: str) -> str: + """Extract the kernel id from an ipykernel connection file.""" + file_name = Path(connection_file).name + prefix = 'kernel-' + suffix = '.json' + if not file_name.startswith(prefix) or not file_name.endswith(suffix): + return '' + return file_name[len(prefix) : -len(suffix)] + def notebook_fit_stop_control( *, diff --git a/tests/unit/easydiffraction/analysis/fit_helpers/test_tracking.py b/tests/unit/easydiffraction/analysis/fit_helpers/test_tracking.py index 97b675969..498238edc 100644 --- a/tests/unit/easydiffraction/analysis/fit_helpers/test_tracking.py +++ b/tests/unit/easydiffraction/analysis/fit_helpers/test_tracking.py @@ -160,7 +160,8 @@ def update(self, value: SimpleNamespace) -> None: pass assert 'Stop fitting' in html_updates[0] - assert "kernelmenu:interrupt" in javascript_outputs[0] + assert "api/kernels/" in javascript_outputs[0] + assert "Interrupt sent..." in javascript_outputs[0] assert html_updates[-1] == '' @@ -194,3 +195,13 @@ def update(self, value: SimpleNamespace) -> None: pass assert 'Fitting stopped.' in html_updates[-1] + + +def test_notebook_fit_stop_control_extracts_kernel_id_from_connection_file(): + from easydiffraction.display.progress import NotebookFitStopControl + + kernel_id = NotebookFitStopControl._kernel_id_from_connection_file( + 'kernel-abc-123.json' + ) + + assert kernel_id == 'abc-123' From 735eb5e9215ec964c5a7251fa8ea5222292e7cd2 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 00:39:36 +0200 Subject: [PATCH 27/65] Clear stop fitting button on interrupt --- src/easydiffraction/display/progress.py | 18 +++++------------- .../analysis/fit_helpers/test_tracking.py | 4 ++-- 2 files changed, 7 insertions(+), 15 deletions(-) diff --git a/src/easydiffraction/display/progress.py b/src/easydiffraction/display/progress.py index 7d9affd3c..010a76fd5 100644 --- a/src/easydiffraction/display/progress.py +++ b/src/easydiffraction/display/progress.py @@ -519,10 +519,10 @@ def __exit__( traceback: TracebackType | None, ) -> None: """Update or clear the stop button when leaving the context.""" + del exc_type del exc_value del traceback - interrupted = exc_type is not None and issubclass(exc_type, KeyboardInterrupt) - self.close(interrupted=interrupted) + self.close() def show(self) -> None: """Render the stop button when running in a notebook.""" @@ -535,14 +535,13 @@ def show(self) -> None: handle.display(HTML(self._active_html())) display(Javascript(self._interrupt_javascript())) - def close(self, *, interrupted: bool = False) -> None: - """Clear or update the stop button when fitting ends.""" + def close(self) -> None: + """Clear the stop button when fitting ends.""" if self._display_handle is None or HTML is None: return - html_content = self._stopped_html() if interrupted else '' with suppress(Exception): - self._display_handle.update(HTML(html_content)) + self._display_handle.update(HTML('')) self._display_handle = None def _can_display(self) -> bool: @@ -592,13 +591,6 @@ def _active_html(self) -> str: '' ) - def _stopped_html(self) -> str: - return ( - f'
' - 'Fitting stopped.' - '
' - ) - def _interrupt_javascript(self) -> str: button_id = f'{self._element_id}-button' status_id = f'{self._element_id}-status' diff --git a/tests/unit/easydiffraction/analysis/fit_helpers/test_tracking.py b/tests/unit/easydiffraction/analysis/fit_helpers/test_tracking.py index 498238edc..dc7af2447 100644 --- a/tests/unit/easydiffraction/analysis/fit_helpers/test_tracking.py +++ b/tests/unit/easydiffraction/analysis/fit_helpers/test_tracking.py @@ -165,7 +165,7 @@ def update(self, value: SimpleNamespace) -> None: assert html_updates[-1] == '' -def test_notebook_fit_stop_control_marks_interrupted(monkeypatch): +def test_notebook_fit_stop_control_clears_button_after_interrupt(monkeypatch): import easydiffraction.display.progress as progress_mod from easydiffraction.utils.enums import VerbosityEnum @@ -194,7 +194,7 @@ def update(self, value: SimpleNamespace) -> None: except KeyboardInterrupt: pass - assert 'Fitting stopped.' in html_updates[-1] + assert html_updates[-1] == '' def test_notebook_fit_stop_control_extracts_kernel_id_from_connection_file(): From cb32ec5131ea3b37e6e3c4a772a014f50950b4be Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 00:44:31 +0200 Subject: [PATCH 28/65] Default emcee resume steps from minimizer settings --- src/easydiffraction/analysis/analysis.py | 16 +++++-- .../easydiffraction/analysis/test_analysis.py | 42 +++++++++++++++++++ 2 files changed, 55 insertions(+), 3 deletions(-) diff --git a/src/easydiffraction/analysis/analysis.py b/src/easydiffraction/analysis/analysis.py index 0c86fe1d3..07c9dd8c8 100644 --- a/src/easydiffraction/analysis/analysis.py +++ b/src/easydiffraction/analysis/analysis.py @@ -974,13 +974,16 @@ def fit( resume=resume, extra_steps=extra_steps, ) + resolved_extra_steps = ( + self._resolved_resume_extra_steps(extra_steps) if resume else extra_steps + ) verb = VerbosityEnum(self.project.verbosity.fit.value) try: with notebook_fit_stop_control(verbosity=verb): self._run_fit_mode( mode=mode, resume=resume, - extra_steps=extra_steps, + extra_steps=resolved_extra_steps, ) except KeyboardInterrupt: self._handle_fit_interrupted(verbosity=verb) @@ -1038,11 +1041,11 @@ def _validate_fit_request( 'before analysis.fit().' ) raise ValueError(msg) - if resume: + if resume and extra_steps is not None: self._validate_resume_extra_steps(extra_steps) @staticmethod - def _validate_resume_extra_steps(extra_steps: int | None) -> None: + def _validate_resume_extra_steps(extra_steps: object) -> int: """Validate the emcee resume step count.""" if extra_steps is None or isinstance(extra_steps, bool): msg = 'extra_steps must be a positive integer when resume=True.' @@ -1056,6 +1059,13 @@ def _validate_resume_extra_steps(extra_steps: int | None) -> None: if integer_steps != extra_steps or integer_steps < 1: msg = 'extra_steps must be a positive integer when resume=True.' raise ValueError(msg) + return integer_steps + + def _resolved_resume_extra_steps(self, extra_steps: int | None) -> int: + """Return explicit or minimizer-default emcee resume steps.""" + if extra_steps is not None: + return self._validate_resume_extra_steps(extra_steps) + return self._validate_resume_extra_steps(self.minimizer.sampling_steps.value) def _prepare_results_sidecar_for_new_fit(self) -> None: """Remove persisted sidecar arrays before a fresh fit.""" diff --git a/tests/unit/easydiffraction/analysis/test_analysis.py b/tests/unit/easydiffraction/analysis/test_analysis.py index ea51940be..7a8ce53cd 100644 --- a/tests/unit/easydiffraction/analysis/test_analysis.py +++ b/tests/unit/easydiffraction/analysis/test_analysis.py @@ -188,6 +188,48 @@ def __exit__(self, exc_type, exc_value, traceback) -> None: assert 'Fitting stopped by user.' in capsys.readouterr().out +def test_fit_resume_defaults_extra_steps_to_sampling_steps(monkeypatch, tmp_path): + from easydiffraction.analysis.analysis import Analysis + + analysis = Analysis(project=_make_project_with_names(['e1'])) + analysis.project.verbosity = SimpleNamespace(fit=SimpleNamespace(value='silent')) + analysis.project.info = SimpleNamespace(path=tmp_path) + analysis.minimizer.type = 'emcee' + analysis.minimizer.sampling_steps = 123 + captured: dict[str, object] = {} + + monkeypatch.setattr( + analysis, + '_run_single', + lambda **kwargs: captured.update(kwargs), + ) + + analysis.fit(resume=True) + + assert captured == {'resume': True, 'extra_steps': 123} + + +def test_fit_resume_preserves_explicit_extra_steps(monkeypatch, tmp_path): + from easydiffraction.analysis.analysis import Analysis + + analysis = Analysis(project=_make_project_with_names(['e1'])) + analysis.project.verbosity = SimpleNamespace(fit=SimpleNamespace(value='silent')) + analysis.project.info = SimpleNamespace(path=tmp_path) + analysis.minimizer.type = 'emcee' + analysis.minimizer.sampling_steps = 123 + captured: dict[str, object] = {} + + monkeypatch.setattr( + analysis, + '_run_single', + lambda **kwargs: captured.update(kwargs), + ) + + analysis.fit(resume=True, extra_steps=10) + + assert captured == {'resume': True, 'extra_steps': 10} + + def test_fitting_mode_type_invalid_assignment_raises_and_preserves_state(): import pytest From c0682596554fe74a79e13e1030350fec6f6b3aac Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 00:51:44 +0200 Subject: [PATCH 29/65] Fallback to fresh emcee fit when resume chain is missing --- src/easydiffraction/analysis/analysis.py | 51 +++++++++++++++++-- .../easydiffraction/analysis/test_analysis.py | 29 +++++++++++ 2 files changed, 77 insertions(+), 3 deletions(-) diff --git a/src/easydiffraction/analysis/analysis.py b/src/easydiffraction/analysis/analysis.py index 07c9dd8c8..f746bec86 100644 --- a/src/easydiffraction/analysis/analysis.py +++ b/src/easydiffraction/analysis/analysis.py @@ -974,15 +974,16 @@ def fit( resume=resume, extra_steps=extra_steps, ) - resolved_extra_steps = ( - self._resolved_resume_extra_steps(extra_steps) if resume else extra_steps + resolved_resume, resolved_extra_steps = self._resolved_resume_request( + resume=resume, + extra_steps=extra_steps, ) verb = VerbosityEnum(self.project.verbosity.fit.value) try: with notebook_fit_stop_control(verbosity=verb): self._run_fit_mode( mode=mode, - resume=resume, + resume=resolved_resume, extra_steps=resolved_extra_steps, ) except KeyboardInterrupt: @@ -1016,6 +1017,25 @@ def _handle_fit_interrupted(self, *, verbosity: VerbosityEnum) -> None: if verbosity is not VerbosityEnum.SILENT: console.print('⏹️ Fitting stopped by user.') + def _resolved_resume_request( + self, + *, + resume: bool, + extra_steps: int | None, + ) -> tuple[bool, int | None]: + """Return executable resume flags for this fit request.""" + if not resume: + return False, extra_steps + + if not self._has_resumable_emcee_sidecar(): + log.warning( + 'resume=True requested, but no saved emcee chain was found; ' + 'starting a fresh fit instead.' + ) + return False, None + + return True, self._resolved_resume_extra_steps(extra_steps) + def _validate_fit_request( self, *, @@ -1067,6 +1087,31 @@ def _resolved_resume_extra_steps(self, extra_steps: int | None) -> int: return self._validate_resume_extra_steps(extra_steps) return self._validate_resume_extra_steps(self.minimizer.sampling_steps.value) + def _has_resumable_emcee_sidecar(self) -> bool: + """Return whether the saved project has a resumable chain.""" + project_path = self.project.info.path + if project_path is None: + return False + + sidecar_path = project_path / 'analysis' / 'results.h5' + if not sidecar_path.is_file(): + return False + + try: + import h5py # noqa: PLC0415 + + from easydiffraction.analysis.minimizers.emcee import ( # noqa: PLC0415 + EMCEE_CHAIN_GROUP, + ) + + with h5py.File(sidecar_path, 'r') as handle: + group = handle.get(EMCEE_CHAIN_GROUP) + if group is None: + return False + return int(group.attrs.get('iteration', 0)) > 0 + except (OSError, TypeError, ValueError): + return False + def _prepare_results_sidecar_for_new_fit(self) -> None: """Remove persisted sidecar arrays before a fresh fit.""" project_path = self.project.info.path diff --git a/tests/unit/easydiffraction/analysis/test_analysis.py b/tests/unit/easydiffraction/analysis/test_analysis.py index 7a8ce53cd..bc022506a 100644 --- a/tests/unit/easydiffraction/analysis/test_analysis.py +++ b/tests/unit/easydiffraction/analysis/test_analysis.py @@ -198,6 +198,7 @@ def test_fit_resume_defaults_extra_steps_to_sampling_steps(monkeypatch, tmp_path analysis.minimizer.sampling_steps = 123 captured: dict[str, object] = {} + monkeypatch.setattr(analysis, '_has_resumable_emcee_sidecar', lambda: True) monkeypatch.setattr( analysis, '_run_single', @@ -219,6 +220,7 @@ def test_fit_resume_preserves_explicit_extra_steps(monkeypatch, tmp_path): analysis.minimizer.sampling_steps = 123 captured: dict[str, object] = {} + monkeypatch.setattr(analysis, '_has_resumable_emcee_sidecar', lambda: True) monkeypatch.setattr( analysis, '_run_single', @@ -230,6 +232,33 @@ def test_fit_resume_preserves_explicit_extra_steps(monkeypatch, tmp_path): assert captured == {'resume': True, 'extra_steps': 10} +def test_fit_resume_missing_sidecar_warns_and_starts_fresh( + monkeypatch, + tmp_path, +): + from easydiffraction.analysis import analysis as analysis_mod + from easydiffraction.analysis.analysis import Analysis + + analysis = Analysis(project=_make_project_with_names(['e1'])) + analysis.project.verbosity = SimpleNamespace(fit=SimpleNamespace(value='silent')) + analysis.project.info = SimpleNamespace(path=tmp_path) + analysis.minimizer.type = 'emcee' + captured: dict[str, object] = {} + warnings: list[str] = [] + + monkeypatch.setattr(analysis_mod.log, 'warning', warnings.append) + monkeypatch.setattr( + analysis, + '_run_single', + lambda **kwargs: captured.update(kwargs), + ) + + analysis.fit(resume=True) + + assert captured == {'resume': False, 'extra_steps': None} + assert any('no saved emcee chain' in message for message in warnings) + + def test_fitting_mode_type_invalid_assignment_raises_and_preserves_state(): import pytest From 3efc2ab1d6bb6aa49e806a7ba73bcfc0cf5ee4b6 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 01:02:29 +0200 Subject: [PATCH 30/65] Resume emcee from persisted sampler state --- .../analysis/minimizers/emcee.py | 12 ++++- .../analysis/minimizers/test_emcee.py | 54 +++++++++++++++++++ 2 files changed, 65 insertions(+), 1 deletion(-) diff --git a/src/easydiffraction/analysis/minimizers/emcee.py b/src/easydiffraction/analysis/minimizers/emcee.py index 3251998fc..018006b05 100644 --- a/src/easydiffraction/analysis/minimizers/emcee.py +++ b/src/easydiffraction/analysis/minimizers/emcee.py @@ -716,9 +716,10 @@ def _run_sampler( # noqa: PLR0913 burn_steps=0 if resume else self.nburn, ) if resume: + initial_state = self._resume_initial_state(backend) self._sample_with_progress( sampler=sampler, - initial_state=None, + initial_state=initial_state, iterations=int(extra_steps), reporter=reporter, skip_initial_state_check=True, @@ -766,6 +767,15 @@ def _backend_iteration(backend: object) -> int: except (AttributeError, TypeError, ValueError): return 0 + @staticmethod + def _resume_initial_state(backend: object) -> object: + """Return the last persisted emcee state for resume runs.""" + try: + return backend.get_last_sample() + except AttributeError as exc: + msg = 'Existing emcee chain has no last sample; start a fresh run.' + raise ValueError(msg) from exc + @staticmethod def _build_log_probability( *, diff --git a/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py b/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py index 455e06555..12cb2b5c2 100644 --- a/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py +++ b/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py @@ -269,6 +269,60 @@ def test_emcee_total_iterations_adds_burn_in_and_initial_generation(): assert minimizer._resolved_total_iterations(resume=True, extra_steps=50) == 50 +def test_emcee_run_sampler_resumes_from_backend_last_sample(monkeypatch): + import easydiffraction.analysis.minimizers.emcee as emcee_mod + from easydiffraction.analysis.minimizers.emcee import EmceeMinimizer + + class FakeEnsembleSampler: + def __init__(self, **kwargs) -> None: + del kwargs + self.fake_sampler = _FakeSampler() + + def sample(self, *args, **kwargs): + return self.fake_sampler.sample(*args, **kwargs) + + backend = SimpleNamespace( + shape=(4, 2), + iteration=3, + reset=lambda *args: (_ for _ in ()).throw(AssertionError('no reset')), + ) + last_sample = SimpleNamespace(log_prob=np.array([-1.0, -2.0], dtype=float)) + backend.get_last_sample = lambda: last_sample + minimizer = EmceeMinimizer() + minimizer.nwalkers = 4 + minimizer.tracker = _FakeTracker() + created_samplers: list[FakeEnsembleSampler] = [] + + def fake_sampler_factory(**kwargs): + sampler = FakeEnsembleSampler(**kwargs) + created_samplers.append(sampler) + return sampler + + monkeypatch.setattr(emcee_mod.emcee, 'EnsembleSampler', fake_sampler_factory) + + minimizer._run_sampler( + backend=backend, + log_prob=lambda values: -1.0, + pool=None, + parameters=[], + n_parameters=2, + random_seed=1, + resume=True, + extra_steps=2, + total_iterations=2, + ) + + assert created_samplers[0].fake_sampler.calls == [ + { + 'initial_state': last_sample, + 'iterations': 2, + 'skip_initial_state_check': True, + 'progress': False, + } + ] + assert [update.iteration for update in minimizer.tracker.updates] == [1, 2] + + def test_emcee_sampler_settings_record_sampling_and_total_steps(): from easydiffraction.analysis.minimizers.emcee import EmceeMinimizer From b3277e39e283d3d0ef10196530930878ffde6e84 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 16:13:33 +0200 Subject: [PATCH 31/65] Update emcee default proposal moves and thinning --- .../minimizer-category-consolidation.md | 6 +++--- docs/dev/plans/emcee-minimizer.md | 8 ++++---- .../analysis/categories/minimizer/emcee.py | 6 +++--- .../analysis/minimizers/emcee.py | 6 +++--- .../categories/minimizer/test_emcee.py | 19 +++++++++++++++++++ .../analysis/minimizers/test_emcee.py | 13 +++++++++++++ 6 files changed, 45 insertions(+), 13 deletions(-) diff --git a/docs/dev/adrs/accepted/minimizer-category-consolidation.md b/docs/dev/adrs/accepted/minimizer-category-consolidation.md index 8d41ae6af..189b564cd 100644 --- a/docs/dev/adrs/accepted/minimizer-category-consolidation.md +++ b/docs/dev/adrs/accepted/minimizer-category-consolidation.md @@ -280,7 +280,7 @@ differs. class EmceeMinimizer(BayesianMinimizerBase): _default_sampling_steps = 5000 _default_population_size = 32 - _default_proposal_moves = 'stretch' + _default_proposal_moves = 'de' ``` When `analysis.minimizer_type` changes, the underlying instance is @@ -339,9 +339,9 @@ _fitting.minimizer_type emcee _minimizer.sampling_steps 5000 _minimizer.burn_in_steps 1000 -_minimizer.thinning_interval 5 +_minimizer.thinning_interval 1 _minimizer.population_size 32 -_minimizer.proposal_moves stretch +_minimizer.proposal_moves de _minimizer.parallel_workers 0 _minimizer.initialization_method ball _minimizer.random_seed 42 diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index 247d57f4e..3380bed32 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -289,15 +289,15 @@ Mark `[x]` as each step lands. `src/easydiffraction/analysis/categories/minimizer/emcee.py`. `EmceeMinimizer(BayesianMinimizerBase)` declares: - `type_info` with `tag=MinimizerTypeEnum.EMCEE` and a description. - - `_engine_metadata: ClassVar[dict[str, str]] = {'optimizer_name': 'emcee', 'method_name': 'stretch'}` + - `_engine_metadata: ClassVar[dict[str, str]] = {'optimizer_name': 'emcee', 'method_name': 'de'}` (matching the `BumpsDreamMinimizer` precedent for the `_restore_fit_results_from_projection` lookup). - `_native_key_map` override mapping the verbose names to emcee's native kwargs (see §"Decisions already made" point 3). - Class-level defaults for emcee-specific values: - `sampling_steps=5000`, `burn_in_steps=1000`, `thinning_interval=5`, + `sampling_steps=5000`, `burn_in_steps=1000`, `thinning_interval=1`, `population_size=32`, `parallel_workers=0`, - `proposal_moves='stretch'`. + `proposal_moves='de'`. - `__init__` constructs descriptors via the inherited helpers (`_sampling_steps_descriptor(default)`, etc. from `BayesianMinimizerBase`) and adds a new `proposal_moves` descriptor @@ -412,7 +412,7 @@ Mark `[x]` as each step lands. ```python class EmceeMinimizer(MinimizerBase): name = MinimizerTypeEnum.EMCEE - method = 'stretch' + method = 'de' # Set by Fitter.fit before this fit() call: _sidecar_path: Path | None = None diff --git a/src/easydiffraction/analysis/categories/minimizer/emcee.py b/src/easydiffraction/analysis/categories/minimizer/emcee.py index e71d8b1c7..651c5be7b 100644 --- a/src/easydiffraction/analysis/categories/minimizer/emcee.py +++ b/src/easydiffraction/analysis/categories/minimizer/emcee.py @@ -18,11 +18,11 @@ DEFAULT_SAMPLING_STEPS = 5000 DEFAULT_BURN_IN_STEPS = 1000 -DEFAULT_THINNING_INTERVAL = 5 +DEFAULT_THINNING_INTERVAL = 1 DEFAULT_POPULATION_SIZE = 32 DEFAULT_PARALLEL_WORKERS = 0 DEFAULT_INITIALIZATION_METHOD = InitializationMethodEnum.BALL -DEFAULT_PROPOSAL_MOVES = 'stretch' +DEFAULT_PROPOSAL_MOVES = 'de' SUPPORTED_PROPOSAL_MOVES = ('stretch', 'de', 'de_snooker', 'walk') @@ -32,7 +32,7 @@ class EmceeMinimizer(BayesianMinimizerBase): _engine_metadata: ClassVar[dict[str, str]] = { 'optimizer_name': 'emcee', - 'method_name': 'stretch', + 'method_name': 'de', } _expected_descriptor_names: ClassVar[tuple[str, ...]] = ( *BayesianMinimizerBase._expected_descriptor_names, diff --git a/src/easydiffraction/analysis/minimizers/emcee.py b/src/easydiffraction/analysis/minimizers/emcee.py index 018006b05..17c1c9333 100644 --- a/src/easydiffraction/analysis/minimizers/emcee.py +++ b/src/easydiffraction/analysis/minimizers/emcee.py @@ -28,14 +28,14 @@ from easydiffraction.core.metadata import TypeInfo from easydiffraction.utils.enums import VerbosityEnum -DEFAULT_METHOD = 'stretch' +DEFAULT_METHOD = 'de' DEFAULT_NSTEPS = 5000 DEFAULT_NBURN = 1000 -DEFAULT_THIN = 5 +DEFAULT_THIN = 1 DEFAULT_NWALKERS = 32 DEFAULT_PARALLEL_WORKERS = 0 DEFAULT_INITIALIZATION_METHOD = InitializationMethodEnum.BALL -DEFAULT_PROPOSAL_MOVES = 'stretch' +DEFAULT_PROPOSAL_MOVES = 'de' MAX_RANDOM_SEED = int(np.iinfo(np.uint32).max) EMCEE_CHAIN_GROUP = 'emcee_chain' EMCEE_FAILURES = (ArithmeticError, RuntimeError, TypeError, ValueError) diff --git a/tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py b/tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py index a90e34349..1b38d65aa 100644 --- a/tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py +++ b/tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py @@ -16,3 +16,22 @@ def test_emcee_minimizer_category_defaults_to_max_parallel_workers(): assert DEFAULT_PARALLEL_WORKERS == 0 assert minimizer.parallel_workers.value == 0 assert minimizer._native_kwargs()['parallel_workers'] == 0 + + +def test_emcee_minimizer_category_defaults_to_de_without_thinning(): + from easydiffraction.analysis.categories.minimizer.emcee import ( + DEFAULT_PROPOSAL_MOVES, + ) + from easydiffraction.analysis.categories.minimizer.emcee import ( + DEFAULT_THINNING_INTERVAL, + ) + from easydiffraction.analysis.categories.minimizer.emcee import EmceeMinimizer + + minimizer = EmceeMinimizer() + + assert DEFAULT_PROPOSAL_MOVES == 'de' + assert DEFAULT_THINNING_INTERVAL == 1 + assert minimizer.proposal_moves.value == 'de' + assert minimizer.thinning_interval.value == 1 + assert minimizer._native_kwargs()['proposal_moves'] == 'de' + assert minimizer._native_kwargs()['thin'] == 1 diff --git a/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py b/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py index 12cb2b5c2..e7a6c37e7 100644 --- a/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py +++ b/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py @@ -61,6 +61,19 @@ def test_emcee_minimizer_defaults_to_max_parallel_workers(): assert minimizer.parallel_workers == 0 +def test_emcee_minimizer_defaults_to_de_without_thinning(): + from easydiffraction.analysis.minimizers.emcee import DEFAULT_PROPOSAL_MOVES + from easydiffraction.analysis.minimizers.emcee import DEFAULT_THIN + from easydiffraction.analysis.minimizers.emcee import EmceeMinimizer + + minimizer = EmceeMinimizer() + + assert DEFAULT_PROPOSAL_MOVES == 'de' + assert DEFAULT_THIN == 1 + assert minimizer.proposal_moves == 'de' + assert minimizer.thin == 1 + + def test_emcee_pool_context_uses_fork_worker_for_unpicklable_objective(monkeypatch): from easydiffraction.analysis.minimizers.emcee import EmceeMinimizer from easydiffraction.analysis.minimizers.emcee import _emcee_log_prob_worker From 8cdb047132e75690d55cf70ebae01eb59674d96e Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 16:14:18 +0200 Subject: [PATCH 32/65] Introduce emcee minimizer tutorials --- docs/docs/tutorials/ed-21.py | 8 +- docs/docs/tutorials/ed-24.py | 2 +- docs/docs/tutorials/ed-25.py | 230 ++++++++++++++++++++++++++--------- docs/docs/tutorials/ed-26.py | 118 ++++++++++++++++++ docs/docs/tutorials/ed-5.py | 3 + docs/mkdocs.yml | 9 +- 6 files changed, 301 insertions(+), 69 deletions(-) create mode 100644 docs/docs/tutorials/ed-26.py diff --git a/docs/docs/tutorials/ed-21.py b/docs/docs/tutorials/ed-21.py index 563992ee1..087fa6718 100644 --- a/docs/docs/tutorials/ed-21.py +++ b/docs/docs/tutorials/ed-21.py @@ -1,5 +1,5 @@ # %% [markdown] -# # Bayesian Analysis: LBCO, HRPT +# # Bayesian Analysis (`bumps-dream`): LBCO, HRPT # # This tutorial demonstrates a practical two-stage workflow for powder # diffraction analysis with EasyDiffraction. @@ -41,7 +41,7 @@ project = ed.Project() # %% -project.save_as('projects/lbco_hrpt_bayesian') +project.save_as('projects/lbco_hrpt_bumps-dream') # %% [markdown] # ## Step 2: Build the Structural Model @@ -297,8 +297,8 @@ project.analysis.minimizer.type = 'bumps (dream)' # %% -project.analysis.minimizer.sampling_steps = 100 # lower than the default 3000 -project.analysis.minimizer.burn_in_steps = 20 # lower than the default 600 +project.analysis.minimizer.sampling_steps = 1000 # lower than the default 3000 +project.analysis.minimizer.burn_in_steps = 200 # lower than the default 600 # %% project.analysis.fit() diff --git a/docs/docs/tutorials/ed-24.py b/docs/docs/tutorials/ed-24.py index 01f3daff9..e1efcecf5 100644 --- a/docs/docs/tutorials/ed-24.py +++ b/docs/docs/tutorials/ed-24.py @@ -1,5 +1,5 @@ # %% [markdown] -# # Load Saved Bayesian Project: LBCO, HRPT +# # Bayesian Analysis Display (`bumps-dream`): LBCO, HRPT # # This tutorial shows how to reopen the Bayesian project created in # `ed-21.py` and inspect the saved fit results without rerunning DREAM. diff --git a/docs/docs/tutorials/ed-25.py b/docs/docs/tutorials/ed-25.py index 34965d2b4..d1f5a57c7 100644 --- a/docs/docs/tutorials/ed-25.py +++ b/docs/docs/tutorials/ed-25.py @@ -1,18 +1,25 @@ # %% [markdown] -# # Bayesian Analysis with emcee: LBCO, HRPT +# # Bayesian Analysis (`emcee`): LBCO, HRPT # -# This tutorial demonstrates how to run Bayesian sampling with the -# emcee minimizer and then resume the same chain from the saved project. +# This tutorial demonstrates a practical two-stage workflow for powder +# diffraction analysis with EasyDiffraction. # -# The workflow uses the same La0.5Ba0.5CoO3 powder diffraction example -# as the DREAM Bayesian tutorial: +# In the first stage, we run a fast local refinement to obtain a sensible +# point estimate and parameter uncertainties. In the second stage, we use +# these refined values to define fit bounds and then sample the posterior +# distribution with emcee. # -# - run a short local refinement, -# - derive finite fit bounds for the sampled parameters, -# - switch to emcee and sample the posterior, -# - save the project with the emcee chain, -# - resume the chain with additional steps, -# - inspect posterior plots after each sampling stage. +# The example uses constant-wavelength neutron powder diffraction data +# for La0.5Ba0.5CoO3 measured on HRPT at PSI. +# +# The goal is not only to obtain a good fit, but also to answer Bayesian +# questions such as: +# +# - Which parameter values are most probable? +# - How broad are the credible intervals? +# - Which parameters are strongly correlated? +# - How much uncertainty propagates into the calculated diffraction +# pattern? # %% [markdown] # ## Import Library @@ -21,10 +28,14 @@ import easydiffraction as ed # %% [markdown] -# ## Create a Project Container +# ## Step 1: Create a Project Container # -# The project is saved before sampling because emcee stores its chain in -# the project's analysis sidecar file. +# The project object keeps structures, experiments, fit settings, and +# plotting utilities together in a single place. We will build the full +# workflow inside this object. +# +# Save the project to a directory early on so that you can easily reload +# it later if needed. # %% project = ed.Project() @@ -33,9 +44,11 @@ project.save_as('projects/lbco_hrpt_emcee') # %% [markdown] -# ## Build the Structural Model +# ## Step 2: Build the Structural Model # -# Define a compact cubic perovskite model for La0.5Ba0.5CoO3. +# We define a simple cubic perovskite model for LBCO. La and Ba share the +# same crystallographic site with equal occupancy, while Co and O occupy +# the remaining ideal perovskite positions. # %% project.structures.create(name='lbco') @@ -50,6 +63,11 @@ # %% structure.cell.length_a = 3.88 +# %% [markdown] +# The atom-site definitions below form the starting structural model. The +# parameters are intentionally reasonable rather than fully optimized, +# because the refinement step will improve them. + # %% structure.atom_sites.create( label='La', @@ -95,14 +113,24 @@ ) # %% [markdown] -# ## Define the Diffraction Experiment +# ## Step 3: Define the Diffraction Experiment # -# Download the HRPT powder pattern, create a neutron powder experiment, -# and set the key instrument, peak-profile, and background values. +# Next we download the measured powder pattern, create a neutron powder +# experiment, and configure the instrument, profile, background, and +# excluded regions. + +# %% [markdown] +# Download the measured data from the repository. Alternatively, you +# could use your own data file by providing the path to it instead of +# downloading from the repository. # %% data_path = ed.download_data(id=3, destination='data') +# %% [markdown] +# Create the experiment object and specify the sample form, beam mode, +# and radiation probe. + # %% project.experiments.add_from_data_path( name='hrpt', @@ -115,9 +143,18 @@ # %% experiment = project.experiments['hrpt'] +# %% [markdown] +# Link the structural phase to the experiment. + # %% experiment.linked_phases.create(id='lbco', scale=9.1351) +# %% [markdown] +# Set instrument and peak profile parameters. +# +# These values provide the initial instrument description for the local +# refinement. Later, a subset of them will be refined. + # %% experiment.instrument.setup_wavelength = 1.494 experiment.instrument.calib_twotheta_offset = 0.0 @@ -128,6 +165,12 @@ experiment.peak.broad_gauss_w = 0.1204 experiment.peak.broad_lorentz_y = 0.0844 +# %% [markdown] +# Add background points and excluded regions. +# +# The line-segment background is defined by a few anchor points. We also +# exclude regions that are not intended to contribute to the fit. + # %% experiment.background.create(id='1', x=10, y=168.5585) experiment.background.create(id='2', x=30, y=164.3357) @@ -139,10 +182,18 @@ experiment.excluded_regions.create(id='2', start=100, end=180) # %% [markdown] -# ## Run a Local Refinement First +# ## Step 4: Run an Initial Local Refinement +# +# Before Bayesian sampling, it is useful to run a deterministic fit. This +# gives us: +# +# - a good point estimate near the best-fit region, +# - uncertainties from the local optimizer, +# - a quick check that the model and experiment are configured +# sensibly. # -# The local fit provides starting values and uncertainties that are used -# to build finite bounds for emcee. +# In this tutorial we refine only a small set of parameters that are easy +# to interpret in the later Bayesian stage. # %% structure.cell.length_a.free = True @@ -153,8 +204,13 @@ experiment.peak.broad_gauss_v.free = True experiment.instrument.calib_twotheta_offset.free = True +# %% [markdown] +# We keep LMFIT Levenberg-Marquardt minimizer as a fast local optimizer. +# Its main purpose here is to provide a stable starting point and +# uncertainty estimates for the Bayesian run. + # %% -project.analysis.minimizer.type = 'bumps (lm)' +project.analysis.minimizer.show_supported() # %% project.analysis.fit() @@ -162,81 +218,135 @@ # %% project.display.fit.results() +# %% [markdown] +# The correlation plot shows how strongly the fitted parameters move +# together in the local refinement. The measured-vs-calculated plots show +# how well the refined model reproduces the data globally and in a zoomed +# region. + # %% -for param in project.free_parameters: - param.set_fit_bounds_from_uncertainty() +project.display.fit.correlations() # %% -project.display.parameters.free() +project.display.pattern(expt_name='hrpt') # %% [markdown] -# ## Run emcee Sampling +# ## Step 5: Prepare for Bayesian Sampling +# +# Bayesian samplers require finite bounds for the free parameters. Instead of +# setting them manually, we derive them from the uncertainties estimated +# in the local refinement. # -# The sampling settings are intentionally small for tutorial runtime. -# Use more steps and inspect convergence diagnostics for production -# analysis. +# The helper method `set_fit_bounds_from_uncertainty` centers the bounds +# on the current parameter value and expands them by a chosen multiple of +# the reported uncertainty. +# +# The default `multiplier` is 4. If the local refinement is very tight, +# or if you expect a broader posterior, increase it explicitly. +# +# Show unset fit bounds before setting them from the local refinement uncertainties. # %% -project.analysis.minimizer.type = 'emcee' +project.display.parameters.free() -# %% -project.analysis.minimizer.sampling_steps = 1000 -project.analysis.minimizer.burn_in_steps = 200 -project.analysis.minimizer.thinning_interval = 10 -project.analysis.minimizer.population_size = 32 -project.analysis.minimizer.initialization_method = 'ball' -project.analysis.minimizer.random_seed = 12345 +# %% [markdown] +# Set fit bounds for all free parameters using the default multiplier of +# 4. In this tutorial that means the posterior pair plot will later +# refer to a `±4 × uncertainty` region in its title. To use a different +# region, pass another value, for example `multiplier=6`. # %% -project.analysis.fit() +for param in project.free_parameters: + param.set_fit_bounds_from_uncertainty() + +# %% [markdown] +# Displaying the free parameters again is a convenient way to confirm +# that the fit bounds have been assigned as expected before launching the +# sampler. # %% -project.display.fit.results() +project.display.parameters.free() # %% [markdown] -# ## Inspect the Posterior +# ## Step 6: Configure and Run emcee +# +# We now switch from the local minimizer to the Bayesian emcee sampler. +# +# The settings below are intentionally small so the tutorial runs +# quickly. For production analysis you would usually increase the number +# of steps and often the burn-in as well. emcee also lets you tune how +# walkers are initialized, how many walkers are used, and which proposal +# move drives the ensemble. # -# The posterior distribution plot shows the sampled marginal -# distributions after the first emcee run. +# The default emcee proposal is the stretch move. This tutorial uses the +# differential-evolution move instead, because it mixes better for the +# strongly correlated LBCO/HRPT parameters. The walker count is kept +# below the default to keep runtime close to the DREAM tutorial while +# retaining good convergence diagnostics for this five-parameter example. # %% -project.display.posterior.distribution() +project.analysis.minimizer.show_supported() -# %% [markdown] -# The posterior predictive plot propagates the sampled parameter -# uncertainty into the calculated diffraction pattern. +# %% +project.analysis.minimizer.type = 'emcee' # %% -project.display.posterior.predictive(expt_name='hrpt') +project.analysis.minimizer.sampling_steps = 10000 # lower than the default 5000 +project.analysis.minimizer.burn_in_steps = 2000 # lower than the default 1000 + +# %% +project.analysis.fit(resume=False) # %% [markdown] -# ## Save the Sampled Project +# ## Step 7: Inspect Bayesian Results # -# Saving persists both the analysis state and the emcee chain sidecar so -# the same chain can be resumed later. +# The fit-results display now includes sampler settings, convergence +# diagnostics, committed parameter values, and posterior summary +# statistics. # %% -project.save() +project.display.fit.results() # %% [markdown] -# ## Resume emcee Sampling +# The correlation and posterior-pair plots are complementary: # -# Resume from the saved backend and append 500 more emcee steps to the -# existing chain. +# - `plot_param_correlations` summarizes pairwise structure in a compact +# matrix. +# - `plot_posterior_pairs` shows marginal densities on the diagonal and +# posterior contours off-diagonal. In this tutorial its title also +# reminds you that the display region follows the `±4 × uncertainty` +# bounds defined above, while numeric subplot ranges are omitted to +# keep the grid readable. # %% -project.analysis.fit(resume=True, extra_steps=500) +project.display.fit.correlations() # %% -project.display.fit.results() +project.display.posterior.pairs() # %% [markdown] -# ## Inspect the Resumed Posterior -# -# After resume, the posterior plots use the extended chain. +# The one-dimensional posterior distributions below make it easier to +# inspect individual parameters in isolation, including asymmetry or +# multimodality. # %% project.display.posterior.distribution() +# %% [markdown] +# Finally, the posterior predictive plot propagates the sampled parameter +# uncertainty into the calculated diffraction pattern. Comparing this to +# the zoomed measured-vs-calculated view helps assess whether the sampled +# model family explains the data in the region of interest. + # %% project.display.posterior.predictive(expt_name='hrpt') + +# %% [markdown] +# A final zoomed measured-vs-calculated plot is useful for checking how +# the posterior-supported model behaves in a narrow region of the pattern +# after the Bayesian run. + +# %% +project.display.posterior.predictive(expt_name='hrpt', x_min=92, x_max=93) + +# %% diff --git a/docs/docs/tutorials/ed-26.py b/docs/docs/tutorials/ed-26.py new file mode 100644 index 000000000..29a39fc3f --- /dev/null +++ b/docs/docs/tutorials/ed-26.py @@ -0,0 +1,118 @@ +# %% [markdown] +# # Bayesian Analysis Resume (`emcee`): LBCO, HRPT +# +# This tutorial shows how to reopen the Bayesian project created previously, +# inspect the saved fit results and then run more sampling steps to extend the existing chain. +# Resuming only works with EMCEE because the current BUMPS-DREAM implementation does not support +# saving and resuming its state. +# +# This workflow is useful when: +# - the initial sampling run has not yet converged and more steps are needed, +# - the initial sampling run has converged but more steps are desired for better posterior resolution, +# - the initial sampling run has converged but the posterior plots have not yet been inspected and the user wants to see the plots before deciding whether to run more steps. +# +# The workflow uses the same La0.5Ba0.5CoO3 powder diffraction example +# as the DREAM Bayesian tutorial: +# +# - run a short local refinement, +# - derive finite fit bounds for the sampled parameters, +# - switch to emcee and sample the posterior, +# - save the project with the emcee chain, +# - resume the chain with additional steps, +# - inspect posterior plots after each sampling stage. + +# %% [markdown] +# ## Import Library + +# %% +import easydiffraction as ed + +# %% [markdown] +# ## Download Saved Project +# +# The returned path points directly to the saved project directory with +# the completed Bayesian fit and persisted posterior samples and plot +# caches. + +# %% +project_dir = ed.download_data(id=35, destination='projects') + +# %% [markdown] +# ## Load the Saved Bayesian Project +# +# Loading restores the persisted fit state, posterior samples, and plot +# caches. No new fit is launched in this tutorial. + +# %% +project = ed.Project.load(project_dir) + +# %% [markdown] +# ## Review the Saved Fit Summary +# +# The fit summary reports the committed point estimate, sampler +# settings, convergence diagnostics, and posterior parameter summaries +# from the saved Bayesian run. + +# %% +project.display.fit.results() + +# %% [markdown] +# ## Show Correlations +# +# The correlation matrix is restored from the saved project state. + +# %% +project.display.fit.correlations() + +# %% [markdown] +# ## Inspect Posterior Densities and Pair Structure +# +# The pair plot and one-dimensional posterior distributions now load +# from the persisted caches generated when the Bayesian fit was saved. + +# %% +project.display.posterior.pairs() + +# %% +project.display.posterior.distribution() + +# %% [markdown] +# ## Plot Posterior Predictive Checks +# +# The posterior predictive view reuses the cached predictive summary +# stored in the project rather than recalculating it on first display. +# It overlays the 95% credible interval propagated from the posterior +# samples. + +# %% +project.display.posterior.predictive(expt_name='hrpt') + +# %% [markdown] +# A zoomed view is useful for checking the propagated uncertainty in a +# narrow region of the diffraction pattern. + +# %% +project.display.posterior.predictive(expt_name='hrpt', x_min=92, x_max=93) + +# %% [markdown] +# ## Resume emcee Sampling +# +# Resume from the saved backend and append 100 more emcee steps to the +# existing chain. We use only 100 steps here to keep the tutorial fast, but in practice you would typically run more steps to ensure convergence and better posterior resolution. + +# %% +project.analysis.fit(resume=True, extra_steps=100) + +# %% +project.display.fit.results() + +# %% [markdown] +# ## Inspect the Resumed Posterior +# +# After resume, the posterior plots use the extended chain. + +# %% +project.display.posterior.distribution() + +# %% +project.display.posterior.predictive(expt_name='hrpt', x_min=92, x_max=93) diff --git a/docs/docs/tutorials/ed-5.py b/docs/docs/tutorials/ed-5.py index 7c307b1bd..cbd235664 100644 --- a/docs/docs/tutorials/ed-5.py +++ b/docs/docs/tutorials/ed-5.py @@ -179,6 +179,9 @@ # %% project = Project() +# %% +project.save_as('projects/cosio_d20') + # %% [markdown] # #### Add Structure diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 72985eec5..15e33ceb9 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -199,7 +199,6 @@ nav: - Load Project: - LBCO Single: tutorials/ed-18.ipynb - Co2SiO4 Sequential: tutorials/ed-23.ipynb - - LBCO Bayesian: tutorials/ed-24.ipynb - Powder Diffraction: - Co2SiO4 pd-neut-cwl: tutorials/ed-5.ipynb - HS pd-neut-cwl: tutorials/ed-6.ipynb @@ -220,9 +219,11 @@ nav: - LBCO+Si McStas: tutorials/ed-9.ipynb - BEER McStas: tutorials/ed-20.ipynb - Bayesian Analysis: - - LBCO Bayesian: tutorials/ed-21.ipynb - - LBCO emcee Resume: tutorials/ed-25.ipynb - - Tb2TiO7 Bayesian: tutorials/ed-22.ipynb + - LBCO pd bumps-dream: tutorials/ed-21.ipynb + - LBCO pd bumps-dream Display: tutorials/ed-24.ipynb + - LBCO pd emcee: tutorials/ed-25.ipynb + - LBCO pd emcee Resume: tutorials/ed-26.ipynb + - Tb2TiO7 sg bumps-dream: tutorials/ed-22.ipynb - Workshops & Schools: - DMSC Summer School: tutorials/ed-13.ipynb - Command-Line: From e2fda13ef3f556835ab4e85c1bd20ebc0ab3fe25 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 20:47:11 +0200 Subject: [PATCH 33/65] Format switch warnings as bullet lists --- src/easydiffraction/analysis/analysis.py | 234 ++++++++++++++---- .../datablocks/experiment/item/base.py | 11 +- src/easydiffraction/utils/utils.py | 134 +++++++++- 3 files changed, 312 insertions(+), 67 deletions(-) diff --git a/src/easydiffraction/analysis/analysis.py b/src/easydiffraction/analysis/analysis.py index f746bec86..cca6fd244 100644 --- a/src/easydiffraction/analysis/analysis.py +++ b/src/easydiffraction/analysis/analysis.py @@ -29,6 +29,8 @@ from easydiffraction.analysis.enums import FitCorrelationSourceEnum from easydiffraction.analysis.enums import FitModeEnum from easydiffraction.analysis.enums import FitResultKindEnum +from easydiffraction.analysis.fit_helpers.bayesian import ESS_BULK_CONVERGENCE_THRESHOLD +from easydiffraction.analysis.fit_helpers.bayesian import R_HAT_CONVERGENCE_THRESHOLD from easydiffraction.analysis.fit_helpers.bayesian import BayesianFitResults from easydiffraction.analysis.fit_helpers.bayesian import PosteriorPredictiveSummary from easydiffraction.analysis.fit_helpers.bayesian import PosteriorSamples @@ -52,6 +54,7 @@ from easydiffraction.utils.logging import log from easydiffraction.utils.utils import _help_method_rows from easydiffraction.utils.utils import _help_property_rows +from easydiffraction.utils.utils import format_bulleted_warning from easydiffraction.utils.utils import render_cif from easydiffraction.utils.utils import render_object_help from easydiffraction.utils.utils import render_table @@ -671,10 +674,142 @@ def _restored_predictive_summaries(self) -> dict[str, PosteriorPredictiveSummary experiment_name, x_axis_name, include_draws=True, - ) - ] = summary + ) + ] = summary return restored_predictive + @staticmethod + def _finite_float(value: object) -> float | None: + """Return a finite float or ``None``.""" + if value is None: + return None + try: + numeric_value = float(value) + except (TypeError, ValueError): + return None + return numeric_value if np.isfinite(numeric_value) else None + + @classmethod + def _restored_bayesian_converged( + cls, + *, + max_r_hat: object, + min_ess_bulk: object, + ) -> bool: + """Return restored convergence status.""" + r_hat = cls._finite_float(max_r_hat) + ess_bulk = cls._finite_float(min_ess_bulk) + if r_hat is None or ess_bulk is None: + return False + return ( + r_hat <= R_HAT_CONVERGENCE_THRESHOLD + and ess_bulk >= ESS_BULK_CONVERGENCE_THRESHOLD + ) + + def _restored_bayesian_convergence_diagnostics( + self, + *, + sample_shape: tuple[int, int, int], + n_parameters: int, + ) -> dict[str, object]: + """Return restored convergence diagnostics.""" + max_r_hat = self.fit_result.gelman_rubin_max.value + min_ess_bulk = self.fit_result.effective_sample_size_min.value + diagnostics: dict[str, object] = { + 'converged': self._restored_bayesian_converged( + max_r_hat=max_r_hat, + min_ess_bulk=min_ess_bulk, + ), + 'max_r_hat': max_r_hat, + 'min_ess_bulk': min_ess_bulk, + 'n_draws': int(sample_shape[0]), + 'n_chains': int(sample_shape[1]), + 'n_parameters': int(n_parameters), + } + + acceptance_rate_mean = self.fit_result.acceptance_rate_mean.value + if acceptance_rate_mean is not None: + diagnostics['acceptance_rate_mean'] = acceptance_rate_mean + return diagnostics + + def _restored_bayesian_reduced_chi_square( + self, + value: object, + *, + restored_parameters: list[Parameter], + ) -> float | None: + """Return restored Bayesian reduced chi-square.""" + persisted_value = self._finite_float(value) + if persisted_value is not None: + return persisted_value + + best_log_posterior = self._finite_float(self.fit_result.best_log_posterior.value) + if best_log_posterior is None: + return None + + n_data_points = self._fit_data_point_count(self.project.experiments) + degrees_of_freedom = n_data_points - len(restored_parameters) + if degrees_of_freedom <= 0: + return None + return -2.0 * best_log_posterior / degrees_of_freedom + + def _restore_bayesian_fit_results_from_projection( + self, + *, + restored_parameters: list[Parameter], + fitting_time: float | None, + reduced_chi_square: float | None, + ) -> BayesianFitResults: + """Rebuild a Bayesian runtime result from saved state.""" + posterior_samples = self._restored_posterior_samples() + sample_shape = ( + np.asarray(posterior_samples.parameter_samples).shape + if posterior_samples is not None + else (0, 0, 0) + ) + posterior_summaries = self._restored_posterior_summaries() + n_parameters = int(sample_shape[2]) or len(posterior_summaries) + sampler_settings = self.minimizer._native_kwargs() + resolved_random_seed = self._restored_bayesian_random_seed(sampler_settings) + sampler_name = ( + 'dream' + if self.minimizer.type == MinimizerTypeEnum.BUMPS_DREAM.value + else str(self.minimizer.type) + ) + restored_results = BayesianFitResults( + success=bool(self.fit_result.success.value), + parameters=restored_parameters, + reduced_chi_square=self._restored_bayesian_reduced_chi_square( + reduced_chi_square, + restored_parameters=restored_parameters, + ), + starting_parameters=list(restored_parameters), + fitting_time=fitting_time, + sampler_name=sampler_name, + point_estimate_name=self.fit_result.point_estimate_name.value or 'best_sample', + posterior_samples=posterior_samples, + posterior_parameter_summaries=posterior_summaries, + posterior_predictive=self._restored_predictive_summaries(), + credible_interval_levels=( + float(self.fit_result.credible_interval_inner.value), + float(self.fit_result.credible_interval_outer.value), + ), + sampler_settings=self._restored_bayesian_sampler_settings( + sampler_settings, + random_seed=resolved_random_seed, + n_parameters=n_parameters, + ), + convergence_diagnostics=self._restored_bayesian_convergence_diagnostics( + sample_shape=sample_shape, + n_parameters=n_parameters, + ), + sampler_completed=bool(self.fit_result.sampler_completed.value), + best_log_posterior=self.fit_result.best_log_posterior.value, + ) + restored_results.message = self.fit_result.message.value or '' + restored_results.iterations = _int_or_none(self.fit_result.iterations.value) or 0 + return restored_results + def _restore_fit_results_from_projection(self) -> object | None: """Rebuild a runtime fit-result object from saved state.""" if not self._has_persisted_fit_state(): @@ -708,51 +843,11 @@ def _restore_fit_results_from_projection(self) -> object | None: reduced_chi_square = self.fit_result.reduced_chi_square.value if self.fit_result.result_kind.value == FitResultKindEnum.BAYESIAN.value: - posterior_samples = self._restored_posterior_samples() - sample_shape = ( - np.asarray(posterior_samples.parameter_samples).shape - if posterior_samples is not None - else (0, 0, 0) - ) - sampler_settings = self.minimizer._native_kwargs() - resolved_random_seed = self._restored_bayesian_random_seed(sampler_settings) - sampler_name = ( - 'dream' - if self.minimizer.type == MinimizerTypeEnum.BUMPS_DREAM.value - else str(self.minimizer.type) - ) - restored_results = BayesianFitResults( - success=bool(self.fit_result.success.value), - parameters=restored_parameters, - reduced_chi_square=reduced_chi_square, - starting_parameters=list(restored_parameters), + restored_results = self._restore_bayesian_fit_results_from_projection( + restored_parameters=restored_parameters, fitting_time=fitting_time, - sampler_name=sampler_name, - point_estimate_name=self.fit_result.point_estimate_name.value or 'best_sample', - posterior_samples=posterior_samples, - posterior_parameter_summaries=self._restored_posterior_summaries(), - posterior_predictive=self._restored_predictive_summaries(), - credible_interval_levels=( - float(self.fit_result.credible_interval_inner.value), - float(self.fit_result.credible_interval_outer.value), - ), - sampler_settings=self._restored_bayesian_sampler_settings( - sampler_settings, - random_seed=resolved_random_seed, - ), - convergence_diagnostics={ - 'converged': False, - 'max_r_hat': self.fit_result.gelman_rubin_max.value, - 'min_ess_bulk': self.fit_result.effective_sample_size_min.value, - 'n_draws': int(sample_shape[0]), - 'n_chains': int(sample_shape[1]), - 'n_parameters': int(sample_shape[2]), - }, - sampler_completed=bool(self.fit_result.sampler_completed.value), - best_log_posterior=self.fit_result.best_log_posterior.value, + reduced_chi_square=reduced_chi_square, ) - restored_results.message = self.fit_result.message.value or '' - restored_results.iterations = _int_or_none(self.fit_result.iterations.value) or 0 self.fit_results = restored_results return restored_results @@ -788,10 +883,11 @@ def _restored_bayesian_sampler_settings( sampler_settings: dict[str, object], *, random_seed: object | None = None, + n_parameters: int = 0, ) -> dict[str, object]: """Return display settings for restored Bayesian results.""" if self.minimizer.type == MinimizerTypeEnum.EMCEE.value: - return { + restored_settings = { 'steps': self._int_sampler_setting(sampler_settings, 'nsteps'), 'burn': self._int_sampler_setting(sampler_settings, 'nburn'), 'thin': self._int_sampler_setting(sampler_settings, 'thin'), @@ -801,8 +897,13 @@ def _restored_bayesian_sampler_settings( 'proposal_moves': str(sampler_settings.get('proposal_moves', '')), 'random_seed': random_seed, } + restored_settings['samples'] = self._sampler_sample_count( + restored_settings, + n_parameters=n_parameters, + ) + return restored_settings - return { + restored_settings = { 'steps': self._int_sampler_setting(sampler_settings, 'steps'), 'burn': self._int_sampler_setting(sampler_settings, 'burn'), 'thin': self._int_sampler_setting(sampler_settings, 'thin'), @@ -811,6 +912,11 @@ def _restored_bayesian_sampler_settings( 'init': str(sampler_settings.get('init', '')), 'random_seed': random_seed, } + restored_settings['samples'] = self._sampler_sample_count( + restored_settings, + n_parameters=n_parameters, + ) + return restored_settings def _restored_bayesian_random_seed( self, @@ -831,6 +937,17 @@ def _int_sampler_setting( value = sampler_settings.get(key, 0) return 0 if value is None else int(value) + @staticmethod + def _sampler_sample_count( + sampler_settings: dict[str, object], + *, + n_parameters: int, + ) -> int: + """Return restored total sampled scalar count.""" + steps = int(sampler_settings.get('steps') or 0) + population = int(sampler_settings.get('pop') or 0) + return max(0, steps) * max(0, population) * max(0, int(n_parameters)) + def help(self) -> None: """Print a summary of analysis properties and methods.""" cls = type(self) @@ -1260,8 +1377,8 @@ def _minimizer_swap_diff( on ``new_minimizer`` (a value the user previously customised is no longer applicable). ``added`` lists settings introduced by the new minimizer with their default value. ``changed`` lists - settings shared by both whose default value differs, in the - ``'{name}={old!r}->{new!r}'`` form. + settings shared by both whose default value differs, in a + ``'{name}: {old!r} -> {new!r}'`` form. """ old_values = old_minimizer._descriptor_values(old_minimizer._setting_descriptor_names) new_values = new_minimizer._descriptor_values(new_minimizer._setting_descriptor_names) @@ -1270,7 +1387,7 @@ def _minimizer_swap_diff( removed = sorted(old_keys - new_keys) added = sorted(f'{name}={new_values[name]!r}' for name in (new_keys - old_keys)) changed = sorted( - f'{name}={old_values[name]!r}->{new_values[name]!r}' + f'{name}: {old_values[name]!r} -> {new_values[name]!r}' for name in (old_keys & new_keys) if old_values[name] != new_values[name] ) @@ -1293,14 +1410,25 @@ def _warn_about_minimizer_swap_defaults( """ removed, added, changed = cls._minimizer_swap_diff(old_minimizer, new_minimizer) if removed: - log.warning(f'Switching minimizer type removes these settings: {", ".join(removed)}.') + log.warning( + format_bulleted_warning( + 'Switching minimizer type removes these settings:', + removed, + ) + ) if added: log.warning( - f'Switching minimizer type adds these settings with defaults: {", ".join(added)}.' + format_bulleted_warning( + 'Switching minimizer type adds these settings with defaults:', + added, + ) ) if changed: log.warning( - f'Switching minimizer type changes these default values: {", ".join(changed)}.' + format_bulleted_warning( + 'Switching minimizer type changes these default values:', + changed, + ) ) def _sync_engine_from_minimizer_category(self) -> None: diff --git a/src/easydiffraction/datablocks/experiment/item/base.py b/src/easydiffraction/datablocks/experiment/item/base.py index b0120d0de..f6500b2db 100644 --- a/src/easydiffraction/datablocks/experiment/item/base.py +++ b/src/easydiffraction/datablocks/experiment/item/base.py @@ -30,6 +30,7 @@ from easydiffraction.io.cif.serialize import experiment_to_cif from easydiffraction.utils.logging import console from easydiffraction.utils.logging import log +from easydiffraction.utils.utils import format_bulleted_warning from easydiffraction.utils.utils import render_cif if TYPE_CHECKING: @@ -618,8 +619,16 @@ def _replace_peak_profile( return if self._peak is not None and announce: + old_profile = PeakFactory._local_alias_for(self._peak.type, **context) + new_profile = PeakFactory._local_alias_for(canonical_type, **context) log.warning( - 'Switching peak profile type discards existing peak parameters.', + format_bulleted_warning( + 'Switching peak profile type changes profile:', + [ + f'{old_profile} -> {new_profile}', + 'existing peak parameters will be discarded.', + ], + ) ) old_peak = self._peak diff --git a/src/easydiffraction/utils/utils.py b/src/easydiffraction/utils/utils.py index e07b15755..d27682b49 100644 --- a/src/easydiffraction/utils/utils.py +++ b/src/easydiffraction/utils/utils.py @@ -28,12 +28,108 @@ pooch.get_logger().setLevel('WARNING') # Suppress pooch info messages + +def display_path(path: pathlib.Path | str) -> str: + """ + Format a filesystem path for user-facing display. + + Returns the path relative to the current working directory so + messages stay compact and avoid forced line breaks. Paths + outside the cwd subtree use ``..`` segments to walk up to a + common ancestor (e.g. ``../sibling/data.cif``) rather than + falling back to an absolute path. The absolute path is only + used when no relative form is possible — on Windows that + happens when the path is on a different drive from the cwd. + + Parameters + ---------- + path : pathlib.Path | str + Filesystem path to format. + + Returns + ------- + str + Display string suitable for inline use in console messages. + """ + resolved = pathlib.Path(path).resolve() + cwd = pathlib.Path.cwd().resolve() + try: + return str(resolved.relative_to(cwd, walk_up=True)) + except ValueError: + return str(resolved) + + +def print_metrics_table(rows: list[list[str]]) -> None: + """ + Render a two-column ``Metric | Value`` table. + + Used for fit-results, settings, and similar summary blocks where + each row is one labelled scalar. Skips rendering entirely when + ``rows`` is empty. + + Parameters + ---------- + rows : list[list[str]] + Each inner list is ``[label, value_string]``. + """ + if not rows: + return + render_table( + columns_headers=['Metric', 'Value'], + columns_alignment=['left', 'right'], + columns_data=rows, + ) + + +def print_table_footnote(entries: list[tuple[str, str]]) -> None: + """ + Print a glossary block below a fit-results-style table. + + Each entry renders as a left-aligned ``• header = description`` + bullet line. The block uses :meth:`ConsolePrinter.small` so it + shows as dim, smaller supplementary text — in Jupyter the font + size matches the table-cell text. + + Parameters + ---------- + entries : list[tuple[str, str]] + Each tuple is `(column header, one-line description)`. + """ + if not entries: + return + width = max(len(name) for name, _ in entries) + 4 + lines = [f' • {name:<{width}} = {description}' for name, description in entries] + console.small(*lines) + + +def format_bulleted_warning(header: str, items: list[str]) -> str: + """ + Format a warning as a header followed by indented bullets. + + Parameters + ---------- + header : str + First warning line. Use a trailing colon when bullets follow. + items : list[str] + Bullet line bodies. + + Returns + ------- + str + Multiline warning text. + """ + if not items: + return header + bullet_lines = [f' • {item}' for item in items] + return '\n'.join([header, *bullet_lines]) + + _DATA_REPO = 'easyscience/diffraction' _DATA_ROOT = 'data' # commit SHA preferred -_DATA_INDEX_REF = 'dbe92a87e0106c4742eee0ff9a8e32bdb8b483cb' +_DATA_INDEX_REF = '83657ee120fc6a30fda231649692930eaa038758' # macOS: sha256sum index.json -_DATA_INDEX_HASH = 'sha256:9e7bbaf2cb650f4126572e85157c63bc76f201408856fe4af566bee55dcdfbb4' +_DATA_INDEX_HASH = 'sha256:e7685d7c81c3b3559a7f630178f4d1b7f441fb1ed14388c10ab9f6aeb93927a7' def _build_data_url(path: str) -> str: @@ -241,8 +337,8 @@ def download_data( existing_project_dir = _existing_project_dir(extraction_dir) if existing_project_dir is not None: console.print( - f"✅ Data #{id} already extracted at '{existing_project_dir}'. " - 'Keeping existing project.' + f"✅ Data #{id} already extracted at '{display_path(existing_project_dir)}'. " + 'Keeping existing.' ) return str(existing_project_dir) @@ -250,14 +346,18 @@ def download_data( if is_project_archive and not overwrite: project_dir = extract_project_from_zip(file_path, destination=extraction_dir) file_path.unlink() - console.print(f"✅ Data #{id} extracted to '{project_dir}'") + console.print(f"✅ Data #{id} extracted to '{display_path(project_dir)}'") return str(project_dir) if not overwrite: console.print( - f"✅ Data #{id} already present at '{file_path}'. Keeping existing file." + f"✅ Data #{id} already present at '{display_path(file_path)}'. " + 'Keeping existing.' ) return str(file_path) - log.debug(f"Data #{id} already present at '{file_path}', but will be overwritten.") + log.debug( + f"Data #{id} already present at '{display_path(file_path)}', " + 'but will be overwritten.' + ) file_path.unlink() known_hash = _normalize_known_hash(record.get('hash')) @@ -276,10 +376,12 @@ def download_data( if is_project_archive: project_dir = extract_project_from_zip(file_path, destination=extraction_dir) file_path.unlink() - console.print(f"✅ Data #{id} downloaded and extracted to\n'{project_dir}'") + console.print( + f"✅ Data #{id} downloaded and extracted to '{display_path(project_dir)}'" + ) return str(project_dir) - console.print(f"✅ Data #{id} downloaded to:\n'{file_path}'") + console.print(f"✅ Data #{id} downloaded to '{display_path(file_path)}'") return str(file_path) @@ -550,17 +652,21 @@ def download_tutorial( if file_path.exists(): if not overwrite: console.print( - f"✅ Tutorial #{id} already present at '{file_path}'. Keeping existing file." + f"✅ Tutorial #{id} already present at '{display_path(file_path)}'. " + 'Keeping existing.' ) return str(file_path) - log.debug(f"Tutorial #{id} already present at '{file_path}', but will be overwritten.") + log.debug( + f"Tutorial #{id} already present at '{display_path(file_path)}', " + 'but will be overwritten.' + ) file_path.unlink() # Download the notebook with _safe_urlopen(url) as resp: file_path.write_bytes(resp.read()) - console.print(f"✅ Tutorial #{id} downloaded to:\n'{file_path}'") + console.print(f"✅ Tutorial #{id} downloaded to '{display_path(file_path)}'") return str(file_path) @@ -606,7 +712,9 @@ def download_all_tutorials( except (OSError, ValueError) as e: log.warning(f'Failed to download tutorial #{tutorial_id}: {e}') - console.print(f'✅ Downloaded {len(downloaded_paths)} tutorials to "{destination}/"') + console.print( + f"✅ Downloaded {len(downloaded_paths)} tutorials to '{display_path(destination)}'" + ) return downloaded_paths From d4ad498ece439bed49fff096d97602f236ceadd8 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 21:00:43 +0200 Subject: [PATCH 34/65] Improve fitting output and peak warnings --- .../categories/fit_result/bayesian.py | 6 +- .../analysis/fit_helpers/bayesian.py | 203 ++++++++---------- .../analysis/fit_helpers/reporting.py | 104 ++++----- src/easydiffraction/analysis/fitting.py | 1 + .../analysis/minimizers/emcee.py | 35 ++- src/easydiffraction/analysis/sequential.py | 3 +- .../datablocks/experiment/item/base.py | 78 +++++-- src/easydiffraction/project/display.py | 2 +- src/easydiffraction/project/project.py | 4 +- src/easydiffraction/utils/logging.py | 74 +++++++ src/easydiffraction/utils/utils.py | 2 +- .../fitting/test_bayesian_helper_support.py | 94 ++++---- .../categories/fit_result/test_bayesian.py | 13 ++ .../minimizer/test_bayesian_base.py | 23 ++ .../analysis/fit_helpers/test_bayesian.py | 32 +-- .../analysis/fit_helpers/test_reporting.py | 20 +- .../analysis/minimizers/test_emcee.py | 39 ++++ .../easydiffraction/analysis/test_analysis.py | 91 +++++++- .../datablocks/experiment/item/test_base.py | 106 +++++++++ .../unit/easydiffraction/utils/test_utils.py | 8 + 20 files changed, 674 insertions(+), 264 deletions(-) diff --git a/src/easydiffraction/analysis/categories/fit_result/bayesian.py b/src/easydiffraction/analysis/categories/fit_result/bayesian.py index 85410ec33..2cd2c2c58 100644 --- a/src/easydiffraction/analysis/categories/fit_result/bayesian.py +++ b/src/easydiffraction/analysis/categories/fit_result/bayesian.py @@ -41,6 +41,7 @@ class BayesianFitResult(FitResultBase): 'acceptance_rate_mean', 'resolved_random_seed', ) + _omitted_result_descriptor_names: ClassVar[tuple[str, ...]] = ('iterations',) _expected_descriptor_names: ClassVar[tuple[str, ...]] = _result_descriptor_names def __init__(self) -> None: @@ -234,6 +235,9 @@ def _set_best_log_posterior(self, value: float | None) -> None: def _cif_parameters(self) -> list[object]: """Return Bayesian fit-result descriptors for CIF output.""" + omitted_descriptor_ids = { + id(getattr(self, name)) for name in self._omitted_result_descriptor_names + } optional_descriptor_ids = { id(getattr(self, name)) for name in self._optional_result_descriptor_names @@ -242,5 +246,5 @@ def _cif_parameters(self) -> list[object]: return [ descriptor for descriptor in self.parameters - if id(descriptor) not in optional_descriptor_ids + if id(descriptor) not in omitted_descriptor_ids | optional_descriptor_ids ] diff --git a/src/easydiffraction/analysis/fit_helpers/bayesian.py b/src/easydiffraction/analysis/fit_helpers/bayesian.py index b248a92fc..67da76d9a 100644 --- a/src/easydiffraction/analysis/fit_helpers/bayesian.py +++ b/src/easydiffraction/analysis/fit_helpers/bayesian.py @@ -9,18 +9,16 @@ import arviz as az import numpy as np -from rich.text import Text - from easydiffraction.analysis.fit_helpers.metrics import calculate_r_factor from easydiffraction.analysis.fit_helpers.metrics import calculate_r_factor_squared from easydiffraction.analysis.fit_helpers.metrics import calculate_rb_factor from easydiffraction.analysis.fit_helpers.metrics import calculate_weighted_r_factor from easydiffraction.analysis.fit_helpers.reporting import FitResults from easydiffraction.analysis.fit_helpers.reporting import _build_parameter_row -from easydiffraction.analysis.fit_helpers.reporting import _format_optional_float from easydiffraction.core.posterior import PosteriorParameterSummary from easydiffraction.utils.logging import console -from easydiffraction.utils.logging import log +from easydiffraction.utils.utils import print_metrics_table +from easydiffraction.utils.utils import print_table_footnote from easydiffraction.utils.utils import render_table R_HAT_CONVERGENCE_THRESHOLD = 1.01 @@ -303,60 +301,82 @@ def display_results( f_calc=f_calc, ) - self._display_summary_header() - _print_fit_quality_metrics(metrics) + console.print('📋 Bayesian fit results:') + print_metrics_table(self._build_fit_results_rows(metrics)) console.print('📈 Committed parameters:') _render_committed_parameter_table(self.parameters) + print_table_footnote(_COMMITTED_PARAMETERS_FOOTNOTE) - console.print('📊 Posterior parameter summaries:') + console.print('📊 Posterior distribution:') _render_posterior_summary_table( parameters=self.parameters, posterior_parameter_summaries=self.posterior_parameter_summaries, ) + print_table_footnote(_POSTERIOR_DISTRIBUTION_FOOTNOTE) self._print_table_notes() - def _print_table_notes(self) -> None: - """ - Print parameter and posterior-diagnostic notes below tables. - """ - super()._print_table_notes() - for note in _posterior_table_notes(self.posterior_parameter_summaries): - log.warning(note) - - def _display_summary_header(self) -> None: - """Render the high-level Bayesian fit summary.""" - status_icon, overall_status = _format_bayesian_overall_status( + def _build_fit_results_rows(self, metrics: dict[str, float | None]) -> list[list[str]]: + """Return the rows for the 'Bayesian fit results' table.""" + overall_status = _bayesian_overall_status( success=self.success, sampler_completed=self.sampler_completed, convergence_diagnostics=self.convergence_diagnostics, ) - fitting_time = _format_optional_float(self.fitting_time, suffix=' seconds') - goodness_of_fit = _format_optional_float(self.reduced_chi_square) - console.paragraph('Bayesian fit results') - console.print(f'{status_icon} Overall status: {overall_status}') + rows: list[list[str]] = [] + sampler_label = self.minimizer_type or self.sampler_name + if sampler_label: + rows.append(['🧪 Sampler', str(sampler_label)]) + rows.append(['✅ Overall status', overall_status]) if self.message: - console.print(f'💬 Sampler status: {self.message}') - console.print(f'🧪 Sampler: {self.sampler_name}') - console.print( - f'🎯 Committed point estimate: {_format_point_estimate_name(self.point_estimate_name)}' - ) - sampler_completed = 'yes' if self.sampler_completed else 'no' - console.print(f'🔁 Sampler completed: {sampler_completed}') - console.print(f'⏱️ Fitting time: {fitting_time}') - console.print(f'📏 Goodness-of-fit (reduced χ²): {goodness_of_fit}') + rows.append(['💬 Engine message', self.message]) + if self.fitting_time is not None: + rows.append(['⏱️ Fitting time (seconds)', f'{self.fitting_time:.2f}']) + if self.reduced_chi_square is not None: + rows.append(['📏 Goodness-of-fit (reduced χ²)', f'{self.reduced_chi_square:.2f}']) + rf = metrics.get('rf') + rf2 = metrics.get('rf2') + wr = metrics.get('wr') + br = metrics.get('br') + if rf is not None: + rows.append(['📏 R-factor (Rf, %)', f'{rf:.2f}']) + if rf2 is not None: + rows.append(['📏 R-factor squared (Rf², %)', f'{rf2:.2f}']) + if wr is not None: + rows.append(['📏 Weighted R-factor (wR, %)', f'{wr:.2f}']) + if br is not None: + rows.append(['📏 Bragg R-factor (BR, %)', f'{br:.2f}']) if self.best_log_posterior is not None: - console.print(f'📉 Best log-posterior: {self.best_log_posterior:.2f}') + rows.append(['📉 Best log-posterior', f'{self.best_log_posterior:.2f}']) + + diagnostics = self.convergence_diagnostics or {} + converged = diagnostics.get('converged') + if converged is not None: + rows.append(['📊 Convergence status', 'passed' if converged else 'failed']) + max_r_hat = diagnostics.get('max_r_hat') + if max_r_hat is not None: + rows.append(['📊 Max r-hat', f'{max_r_hat:.3f}']) + min_ess_bulk = diagnostics.get('min_ess_bulk') + if min_ess_bulk is not None: + rows.append(['📊 Min ess bulk', f'{min_ess_bulk:.1f}']) + n_draws = diagnostics.get('n_draws') + if n_draws is not None: + rows.append(['📊 Draws per chain', str(n_draws)]) + n_chains = diagnostics.get('n_chains') + if n_chains is not None: + rows.append(['📊 Chains', str(n_chains)]) + return rows - sampler_settings = _format_sampler_settings(self.sampler_settings) - if sampler_settings is not None: - console.print(Text(f'⚙️ Sampler settings: {sampler_settings}')) - - convergence_summary = _format_convergence_summary(self.convergence_diagnostics) - if convergence_summary is not None: - console.print(Text.from_markup(f'📊 Convergence: {convergence_summary}')) + def _print_table_notes(self) -> None: + """ + Print parameter and posterior-diagnostic notes below tables. + """ + super()._print_table_notes() + notes = _posterior_table_notes(self.posterior_parameter_summaries) + if notes: + console.small(*notes) def compute_convergence_diagnostics(posterior_samples: PosteriorSamples) -> dict[str, object]: @@ -518,18 +538,6 @@ def _maybe_scalar(value: object) -> float | None: return scalar -def _format_sampler_settings(sampler_settings: dict[str, object]) -> str | None: - if not sampler_settings: - return None - - parts = [ - f'{key}={sampler_settings[key]}' - for key in ('steps', 'burn', 'thin', 'pop', 'init', 'samples') - if key in sampler_settings - ] - return ', '.join(parts) if parts else None - - def _calculate_fit_quality_metrics( *, y_obs: list[float] | None, @@ -555,69 +563,41 @@ def _calculate_fit_quality_metrics( return metrics -def _print_fit_quality_metrics(metrics: dict[str, float | None]) -> None: - """Render any available fit-quality metrics.""" - metric_labels = ( - ('📏 R-factor (Rf)', metrics['rf']), - ('📏 R-factor squared (Rf²)', metrics['rf2']), - ('📏 Weighted R-factor (wR)', metrics['wr']), - ('📏 Bragg R-factor (BR)', metrics['br']), - ) - for label, value in metric_labels: - if value is not None: - console.print(f'{label}: {value:.2f}%') - - -def _format_point_estimate_name(point_estimate_name: str) -> str: - """Return a user-facing label for the committed point estimate.""" - normalized_name = point_estimate_name.strip().lower().replace('_', ' ') - if normalized_name in {'best sample', 'map'}: - return 'Best posterior sample' - return point_estimate_name.replace('_', ' ').title() - - -def _format_bayesian_overall_status( +def _bayesian_overall_status( *, success: bool, sampler_completed: bool, convergence_diagnostics: dict[str, object], -) -> tuple[str, str]: - """Return icon and text for Bayesian run status.""" - if not success: - return '❌', 'failed' +) -> str: + """ + Return ``'success'`` or ``'failed'`` for the Bayesian run. - converged = convergence_diagnostics.get('converged') + Bayesian success requires both the sampler to have completed and + the convergence diagnostics to have passed. Anything else is + rendered as ``failed`` in the overall row; the per-metric + convergence rows below carry the detail. + """ + if not success or not sampler_completed: + return 'failed' + converged = convergence_diagnostics.get('converged') if convergence_diagnostics else None if converged is False: - return '⚠️', 'completed with warnings' - if sampler_completed: - return '✅', 'completed' - return '✅', 'posterior available' - - -def _format_convergence_summary(convergence_diagnostics: dict[str, object]) -> str | None: - if not convergence_diagnostics: - return None - - parts: list[str] = [] - converged = convergence_diagnostics.get('converged') - if converged is not None: - status = 'passed' if converged else '[red]failed[/red]' - parts.append(f'status={status}') - - max_r_hat = _maybe_scalar(convergence_diagnostics.get('max_r_hat')) - if max_r_hat is not None: - parts.append(f'max_r_hat={_format_r_hat(max_r_hat)}') + return 'failed' + return 'success' - min_ess_bulk = _maybe_scalar(convergence_diagnostics.get('min_ess_bulk')) - if min_ess_bulk is not None: - parts.append(f'min_ess_bulk={_format_ess_bulk(min_ess_bulk)}') - n_draws = convergence_diagnostics.get('n_draws') - n_chains = convergence_diagnostics.get('n_chains') - if n_draws is not None and n_chains is not None: - parts.append(f'draws={n_draws}, chains={n_chains}') +_COMMITTED_PARAMETERS_FOOTNOTE: list[tuple[str, str]] = [ + ('start', 'parameter value before sampling'), + ('value', 'estimate written back to the project (best posterior sample)'), + ('s.u.', 'standard uncertainty (1σ), the posterior standard deviation'), + ('change', 'relative change from start, in %; ↑ = increase, ↓ = decrease'), +] - return ', '.join(parts) if parts else None +_POSTERIOR_DISTRIBUTION_FOOTNOTE: list[tuple[str, str]] = [ + ('median', '50th percentile of the marginal posterior'), + ('95% CI', '95% credible interval (2.5%–97.5%, asymmetric)'), + ('r-hat', 'Gelman–Rubin diagnostic R̂ (good convergence: r-hat ≤ 1.01)'), + ('ess bulk', 'bulk effective sample size (typically ≥ 400)'), +] def _render_committed_parameter_table(parameters: list[object]) -> None: @@ -628,8 +608,8 @@ def _render_committed_parameter_table(parameters: list[object]) -> None: 'parameter', 'units', 'start', - 'best posterior sample', - 'uncertainty', + 'value', + 's.u.', 'change', ] alignments = [ @@ -668,7 +648,7 @@ def _render_posterior_summary_table( 'parameter', 'units', 'median', - '95% interval', + '95% CI', 'r-hat', 'ess bulk', ] @@ -763,12 +743,13 @@ def _posterior_table_notes( notes: list[str] = [] if has_failed_r_hat: notes.append( - f'[red]r-hat > {R_HAT_CONVERGENCE_THRESHOLD:.2f}[/red]: ' - 'Consider longer sampling, better initialization, or reparameterization.' + f'⚠️ [red]r-hat > {R_HAT_CONVERGENCE_THRESHOLD:.2f}[/red]: ' + 'Consider longer sampling, better initialization, or ' + 'reparameterization.' ) if has_failed_ess_bulk: notes.append( - f'[red]ess bulk < {ESS_BULK_CONVERGENCE_THRESHOLD:.0f}[/red]: ' + f'⚠️ [red]ess bulk < {ESS_BULK_CONVERGENCE_THRESHOLD:.0f}[/red]: ' 'Consider longer sampling or reparameterization.' ) return notes diff --git a/src/easydiffraction/analysis/fit_helpers/reporting.py b/src/easydiffraction/analysis/fit_helpers/reporting.py index aefad27b6..56ef0e526 100644 --- a/src/easydiffraction/analysis/fit_helpers/reporting.py +++ b/src/easydiffraction/analysis/fit_helpers/reporting.py @@ -7,7 +7,8 @@ from easydiffraction.analysis.fit_helpers.metrics import calculate_rb_factor from easydiffraction.analysis.fit_helpers.metrics import calculate_weighted_r_factor from easydiffraction.utils.logging import console -from easydiffraction.utils.logging import log +from easydiffraction.utils.utils import print_metrics_table +from easydiffraction.utils.utils import print_table_footnote from easydiffraction.utils.utils import render_table @@ -65,6 +66,7 @@ def __init__( starting_parameters if starting_parameters is not None else [] ) self.fitting_time: float | None = fitting_time + self.minimizer_type: str | None = None if 'redchi' in kwargs and self.reduced_chi_square is None: self.reduced_chi_square = kwargs.get('redchi') @@ -96,7 +98,6 @@ def display_results( f_calc : list[float] | None, default=None Calculated structure-factor magnitudes for Bragg R. """ - status_icon = '✅' if self.success else '❌' rf = rf2 = wr = br = None if y_obs is not None and y_calc is not None: rf = calculate_r_factor(y_obs, y_calc) * 100 @@ -106,21 +107,10 @@ def display_results( if f_obs is not None and f_calc is not None: br = calculate_rb_factor(f_obs, f_calc) * 100 - console.paragraph('Fit results') - console.print(f'{status_icon} Success: {self.success}') - fitting_time = _format_optional_float(self.fitting_time, suffix=' seconds') - goodness_of_fit = _format_optional_float(self.reduced_chi_square) - console.print(f'⏱️ Fitting time: {fitting_time}') - console.print(f'📏 Goodness-of-fit (reduced χ²): {goodness_of_fit}') - if rf is not None: - console.print(f'📏 R-factor (Rf): {rf:.2f}%') - if rf2 is not None: - console.print(f'📏 R-factor squared (Rf²): {rf2:.2f}%') - if wr is not None: - console.print(f'📏 Weighted R-factor (wR): {wr:.2f}%') - if br is not None: - console.print(f'📏 Bragg R-factor (BR): {br:.2f}%') - console.print('📈 Fitted parameters:') + console.print('📋 Least-squares fit results:') + print_metrics_table(self._build_fit_results_rows(rf=rf, rf2=rf2, wr=wr, br=br)) + + console.print('📈 Refined parameters:') headers = [ 'datablock', @@ -129,8 +119,8 @@ def display_results( 'parameter', 'units', 'start', - 'fitted', - 'uncertainty', + 'value', + 's.u.', 'change', ] alignments = [ @@ -153,23 +143,61 @@ def display_results( columns_data=rows, ) + print_table_footnote(_REFINED_PARAMETERS_FOOTNOTE) self._print_table_notes() + def _build_fit_results_rows( + self, + *, + rf: float | None, + rf2: float | None, + wr: float | None, + br: float | None, + ) -> list[list[str]]: + """Return the rows for the 'Least-squares fit results' table.""" + rows: list[list[str]] = [] + if self.minimizer_type is not None: + rows.append(['🧪 Minimizer', str(self.minimizer_type)]) + rows.append(['✅ Overall status', 'success' if self.success else 'failed']) + if self.fitting_time is not None: + rows.append(['⏱️ Fitting time (seconds)', f'{self.fitting_time:.2f}']) + if self.iterations: + rows.append(['🔁 Iterations', str(self.iterations)]) + if self.reduced_chi_square is not None: + rows.append(['📏 Goodness-of-fit (reduced χ²)', f'{self.reduced_chi_square:.2f}']) + if rf is not None: + rows.append(['📏 R-factor (Rf, %)', f'{rf:.2f}']) + if rf2 is not None: + rows.append(['📏 R-factor squared (Rf², %)', f'{rf2:.2f}']) + if wr is not None: + rows.append(['📏 Weighted R-factor (wR, %)', f'{wr:.2f}']) + if br is not None: + rows.append(['📏 Bragg R-factor (BR, %)', f'{br:.2f}']) + return rows + def _print_table_notes(self) -> None: - """Print color-coded notes below the fitted parameters table.""" + """Print color-coded warnings below the refined parameters table.""" notes: list[str] = [] if any(getattr(p, '_outside_physical_limits', False) for p in self.parameters): notes.append( - '[red]Red fitted value:[/red] outside expected physical limits (consider ' - 'adding constraints)' + '⚠️ [red]Red value:[/red] outside expected physical limits ' + '(consider adding constraints)' ) if any(_is_uncertainty_large(p) for p in self.parameters): notes.append( - '[red]Red uncertainty:[/red] exceeds the fitted value (consider adding ' - 'constraints)' + '⚠️ [red]Red s.u.:[/red] exceeds the refined value ' + '(consider adding constraints)' ) - for note in notes: - log.warning(note) + if notes: + console.small(*notes) + + +_REFINED_PARAMETERS_FOOTNOTE: list[tuple[str, str]] = [ + ('start', 'parameter value before refinement'), + ('value', 'refined value from least-squares minimization'), + ('s.u.', 'standard uncertainty (1σ), from the covariance matrix'), + ('change', 'relative change from start, in %; ↑ = increase, ↓ = decrease'), +] def _is_uncertainty_large(param: object) -> bool: @@ -252,27 +280,3 @@ def _compute_relative_change(param: object) -> str: return f'{abs(change):.2f} % {arrow}' -def _format_optional_float( - value: float | None, - *, - suffix: str = '', -) -> str: - """ - Format an optional float for console output. - - Parameters - ---------- - value : float | None - Value to format. - suffix : str, default='' - Optional suffix appended to formatted numeric values. - - Returns - ------- - str - ``'N/A'`` when the value is ``None``; otherwise a formatted - string with two decimal places. - """ - if value is None: - return 'N/A' - return f'{value:.2f}{suffix}' diff --git a/src/easydiffraction/analysis/fitting.py b/src/easydiffraction/analysis/fitting.py index fd47dc82c..90587853d 100644 --- a/src/easydiffraction/analysis/fitting.py +++ b/src/easydiffraction/analysis/fitting.py @@ -127,6 +127,7 @@ def _postprocess_fit_results( self.results.message = _resolve_fit_result_message(self.results) self.results.iterations = _resolve_fit_result_iterations(self.results) self.results.chi_square = _resolve_fit_result_chi_square(self.results) + self.results.minimizer_type = self.selection if analysis is None: return diff --git a/src/easydiffraction/analysis/minimizers/emcee.py b/src/easydiffraction/analysis/minimizers/emcee.py index 17c1c9333..d44002826 100644 --- a/src/easydiffraction/analysis/minimizers/emcee.py +++ b/src/easydiffraction/analysis/minimizers/emcee.py @@ -20,6 +20,7 @@ from easydiffraction.analysis.fit_helpers.bayesian import compute_convergence_diagnostics from easydiffraction.analysis.fit_helpers.bayesian import standard_deviations_from_summaries from easydiffraction.analysis.fit_helpers.bayesian import summarize_posterior_parameters +from easydiffraction.analysis.fit_helpers.metrics import calculate_reduced_chi_square from easydiffraction.analysis.fit_helpers.tracking import SamplerProgressUpdate from easydiffraction.analysis.minimizers.base import MinimizerBase from easydiffraction.analysis.minimizers.enums import InitializationMethodEnum @@ -674,6 +675,12 @@ def _run_solver( starting_values=kwargs['starting_values'], starting_uncertainties=kwargs['starting_uncertainties'], ) + if getattr(result, 'success', False): + result.reduced_chi_square = self._best_sample_reduced_chi_square( + objective_function=objective_function, + parameter_names=parameter_names, + best_sample_values=np.asarray(result.x, dtype=float), + ) self.tracker.start_sampler_post_processing() return result @@ -1094,6 +1101,7 @@ def _build_success_result( # noqa: PLR0914 self._track_sampler_completion( total_steps=total_steps, best_log_posterior=best_log_posterior, + reduced_chi_square=None, ) return OptimizeResult( @@ -1115,6 +1123,26 @@ def _build_success_result( # noqa: PLR0914 starting_uncertainties=list(starting_uncertainties), ) + @staticmethod + def _best_sample_reduced_chi_square( + *, + objective_function: Callable[[dict[str, object]], object], + parameter_names: list[str], + best_sample_values: np.ndarray, + ) -> float | None: + """Evaluate reduced chi-square at the committed sample.""" + engine_params = { + name: float(value) + for name, value in zip(parameter_names, best_sample_values, strict=True) + } + try: + residuals = np.asarray(objective_function(engine_params), dtype=float) + except Exception: # noqa: BLE001 - calculator failures leave chi-square unknown. + return None + if residuals.size == 0 or not np.all(np.isfinite(residuals)): + return None + return calculate_reduced_chi_square(residuals, len(parameter_names)) + def _convergence_diagnostics( self, *, @@ -1157,9 +1185,12 @@ def _track_sampler_completion( *, total_steps: int, best_log_posterior: float, + reduced_chi_square: float | None, ) -> None: """Record one final sampler progress row.""" - reduced_chi2 = self.tracker.best_chi2 + reduced_chi2 = reduced_chi_square + if reduced_chi2 is None: + reduced_chi2 = self.tracker.best_chi2 if reduced_chi2 is None: reduced_chi2 = np.nan self.tracker.track_sampler_progress( @@ -1228,7 +1259,7 @@ def _build_fit_results( fit_results = BayesianFitResults( success=success, parameters=parameters, - reduced_chi_square=self.tracker.best_chi2, + reduced_chi_square=getattr(raw_result, 'reduced_chi_square', self.tracker.best_chi2), engine_result=getattr(raw_result, 'raw_state', raw_result), starting_parameters=parameters, fitting_time=self.tracker.fitting_time, diff --git a/src/easydiffraction/analysis/sequential.py b/src/easydiffraction/analysis/sequential.py index ce9fe54b2..a6b338051 100644 --- a/src/easydiffraction/analysis/sequential.py +++ b/src/easydiffraction/analysis/sequential.py @@ -26,6 +26,7 @@ from easydiffraction.utils.logging import console from easydiffraction.utils.logging import log from easydiffraction.utils.utils import build_table_renderable +from easydiffraction.utils.utils import display_path # ------------------------------------------------------------------ # Template dataclass (picklable for ProcessPoolExecutor) @@ -893,7 +894,7 @@ def _print_sequential_completion( return console.print(f'✅ Sequential fitting complete: {processed_count} files processed.') - console.print(f'📄 Results saved to:\n{csv_path}') + console.print(f"📄 Results saved to '{display_path(csv_path)}'") def _prepare_sequential_run( diff --git a/src/easydiffraction/datablocks/experiment/item/base.py b/src/easydiffraction/datablocks/experiment/item/base.py index f6500b2db..ee1de7fda 100644 --- a/src/easydiffraction/datablocks/experiment/item/base.py +++ b/src/easydiffraction/datablocks/experiment/item/base.py @@ -618,21 +618,12 @@ def _replace_peak_profile( log.warning(msg) return - if self._peak is not None and announce: - old_profile = PeakFactory._local_alias_for(self._peak.type, **context) - new_profile = PeakFactory._local_alias_for(canonical_type, **context) - log.warning( - format_bulleted_warning( - 'Switching peak profile type changes profile:', - [ - f'{old_profile} -> {new_profile}', - 'existing peak parameters will be discarded.', - ], - ) - ) - old_peak = self._peak - self._peak = PeakFactory.create(canonical_type) + new_peak = PeakFactory.create(canonical_type) + if old_peak is not None and announce: + self._warn_about_peak_profile_swap(old_peak, new_peak) + + self._peak = new_peak if old_peak is not None: old_peak._parent = None self._peak._parent = self @@ -682,3 +673,62 @@ def _restore_switchable_types(self, block: object) -> None: peak_type = read_cif_str(block, '_peak.type') if peak_type is not None: self._set_peak_profile_type(peak_type) + + @staticmethod + def _peak_parameter_values(peak: object) -> dict[str, object]: + """Return peak parameter values excluding the selector type.""" + return { + parameter.name: parameter.value + for parameter in getattr(peak, 'parameters', []) + if parameter.name != 'type' + } + + @classmethod + def _peak_profile_swap_diff( + cls, + old_peak: object, + new_peak: object, + ) -> tuple[list[str], list[str], list[str]]: + """Return removed, added, and reset peak-setting rows.""" + old_values = cls._peak_parameter_values(old_peak) + new_values = cls._peak_parameter_values(new_peak) + old_keys = set(old_values) + new_keys = set(new_values) + removed = sorted(old_keys - new_keys) + added = sorted(f'{name}={new_values[name]!r}' for name in (new_keys - old_keys)) + reset = sorted( + f'{name}: {old_values[name]!r} -> {new_values[name]!r}' + for name in (old_keys & new_keys) + if old_values[name] != new_values[name] + ) + return removed, added, reset + + @classmethod + def _warn_about_peak_profile_swap( + cls, + old_peak: object, + new_peak: object, + ) -> None: + """Warn about peak settings changed by a profile swap.""" + removed, added, reset = cls._peak_profile_swap_diff(old_peak, new_peak) + if removed: + log.warning( + format_bulleted_warning( + 'Switching peak profile type removes these settings:', + removed, + ) + ) + if added: + log.warning( + format_bulleted_warning( + 'Switching peak profile type adds these settings with defaults:', + added, + ) + ) + if reset: + log.warning( + format_bulleted_warning( + 'Switching peak profile type resets these settings to defaults:', + reset, + ) + ) diff --git a/src/easydiffraction/project/display.py b/src/easydiffraction/project/display.py index d6067d1ec..fb5983d8c 100644 --- a/src/easydiffraction/project/display.py +++ b/src/easydiffraction/project/display.py @@ -103,7 +103,7 @@ def _show_settings_used(self) -> None: if not rows: return - console.paragraph('Settings used') + console.print('⚙️ Settings used:') render_table( columns_headers=['Name', 'Value', 'Description'], columns_alignment=['left', 'right', 'left'], diff --git a/src/easydiffraction/project/project.py b/src/easydiffraction/project/project.py index a9172be35..3f64ab20b 100644 --- a/src/easydiffraction/project/project.py +++ b/src/easydiffraction/project/project.py @@ -30,6 +30,7 @@ from easydiffraction.utils.environment import resolve_artifact_path from easydiffraction.utils.logging import console from easydiffraction.utils.logging import log +from easydiffraction.utils.utils import display_path if TYPE_CHECKING: from collections.abc import Callable @@ -463,8 +464,7 @@ def save(self) -> None: log.error('Project path not specified. Use save_as() to define the path first.') return - console.paragraph(f"Saving project 📦 '{self.name}' to") - console.print(self.info.path.resolve()) + console.paragraph(f"Saving project 📦 '{self.name}' to '{display_path(self.info.path)}'") # Apply constraints so dependent parameters are flagged # before serialization (user-constrained params are written diff --git a/src/easydiffraction/utils/logging.py b/src/easydiffraction/utils/logging.py index 3c4da8ee5..2a6f851c5 100644 --- a/src/easydiffraction/utils/logging.py +++ b/src/easydiffraction/utils/logging.py @@ -24,6 +24,7 @@ if TYPE_CHECKING: # pragma: no cover from types import TracebackType +import html import re import sys from pathlib import Path @@ -641,6 +642,37 @@ def critical(cls, *messages: str, exc_type: type[BaseException] = RuntimeError) # ====================================================================== +_RICH_RED_MARKUP_PATTERN = re.compile(r'\[red\](.*?)\[/red\]') +_RICH_DIM_MARKUP_PATTERN = re.compile(r'\[/?dim\]') + + +def _rich_markup_to_inline_html(markup: str) -> str: + """ + Translate a narrow subset of Rich markup to inline HTML. + + Handles ``[red]…[/red]`` (→ a red ````) and silently + strips ``[dim]`` / ``[/dim]`` tags (the surrounding container + already conveys dimness via CSS opacity). Other Rich markup + passes through unescaped — callers must only emit markup from + this allow-list when calling ``ConsolePrinter.small`` in + Jupyter. + + Parameters + ---------- + markup : str + Rich markup string. + + Returns + ------- + str + HTML-escaped string with the allow-listed Rich tags + translated to inline ```` styles. + """ + escaped = html.escape(markup) + without_dim = _RICH_DIM_MARKUP_PATTERN.sub('', escaped) + return _RICH_RED_MARKUP_PATTERN.sub(r'\1', without_dim) + + class ConsolePrinter: """Printer utility for the shared console with left padding.""" @@ -706,6 +738,48 @@ def section(cls, title: str) -> None: formatted = f'\n{formatted}' cls._console.print(formatted) + @classmethod + def small(cls, *lines: str) -> None: + """ + Print one or more lines as dim, smaller supplementary text. + + Intended for table footnote glossaries and inline warning + notes that should read as subordinate to the table or + block they sit beneath. In Jupyter the lines render inside a + single ````-style HTML element so the font size + matches Jupyter's ``.dataframe`` table-cell text. In a + terminal the lines render with Rich's ``dim`` style. Rich + ``[red]…[/red]`` markup inside ``lines`` is preserved in + both renderers. + + Parameters + ---------- + *lines : str + Pre-formatted display lines. Each may contain Rich + ``[red]…[/red]`` markup; other Rich markup is rendered + in the terminal and stripped in the HTML output. + """ + if not lines: + return + if in_jupyter(): + try: + from IPython.display import HTML # noqa: PLC0415 + from IPython.display import display # noqa: PLC0415 + + body = '
'.join(_rich_markup_to_inline_html(line) for line in lines) + display( + HTML( + '
' + f'{body}
' + ) + ) + return + except ImportError: # pragma: no cover + pass + for line in lines: + cls._console.print(f'[dim]{line}[/dim]') + @classmethod def chapter(cls, title: str) -> None: """Format a chapter header in bold magenta, uppercase.""" diff --git a/src/easydiffraction/utils/utils.py b/src/easydiffraction/utils/utils.py index d27682b49..7834df1dd 100644 --- a/src/easydiffraction/utils/utils.py +++ b/src/easydiffraction/utils/utils.py @@ -120,7 +120,7 @@ def format_bulleted_warning(header: str, items: list[str]) -> str: """ if not items: return header - bullet_lines = [f' • {item}' for item in items] + bullet_lines = [f'• {item}' for item in items] return '\n'.join([header, *bullet_lines]) diff --git a/tests/integration/fitting/test_bayesian_helper_support.py b/tests/integration/fitting/test_bayesian_helper_support.py index b67d5db02..45a89be89 100644 --- a/tests/integration/fitting/test_bayesian_helper_support.py +++ b/tests/integration/fitting/test_bayesian_helper_support.py @@ -217,12 +217,9 @@ def test_standard_deviations_from_summaries_returns_float_array(): def test_bayesian_format_helpers_cover_edge_cases(): + from easydiffraction.analysis.fit_helpers.bayesian import _bayesian_overall_status from easydiffraction.analysis.fit_helpers.bayesian import _calculate_fit_quality_metrics from easydiffraction.analysis.fit_helpers.bayesian import _dataset_to_scalar_dict - from easydiffraction.analysis.fit_helpers.bayesian import _format_bayesian_overall_status - from easydiffraction.analysis.fit_helpers.bayesian import _format_convergence_summary - from easydiffraction.analysis.fit_helpers.bayesian import _format_point_estimate_name - from easydiffraction.analysis.fit_helpers.bayesian import _format_sampler_settings from easydiffraction.analysis.fit_helpers.bayesian import _maybe_scalar dataset = type( @@ -235,44 +232,29 @@ def test_bayesian_format_helpers_cover_edge_cases(): assert _maybe_scalar(float('inf')) is None assert _maybe_scalar(3.0) == pytest.approx(3.0) assert _dataset_to_scalar_dict(dataset) == {'a': None, 'b': 3.0} - assert _format_sampler_settings({}) is None - assert ( - _format_sampler_settings({'steps': 10, 'burn': 2, 'samples': 40}) - == 'steps=10, burn=2, samples=40' - ) - assert _format_point_estimate_name('map') == 'Best posterior sample' - assert _format_point_estimate_name('best_sample') == 'Best posterior sample' - assert _format_bayesian_overall_status( + + # Two-state overall-status helper: 'success' only when sampler + # completed AND convergence passed. + assert _bayesian_overall_status( success=False, sampler_completed=False, convergence_diagnostics={}, - ) == ('❌', 'failed') - assert _format_bayesian_overall_status( + ) == 'failed' + assert _bayesian_overall_status( success=True, sampler_completed=False, convergence_diagnostics={'converged': False}, - ) == ('⚠️', 'completed with warnings') - assert _format_bayesian_overall_status( + ) == 'failed' + assert _bayesian_overall_status( success=True, sampler_completed=True, convergence_diagnostics={'converged': True}, - ) == ('✅', 'completed') - assert _format_bayesian_overall_status( + ) == 'success' + assert _bayesian_overall_status( success=True, sampler_completed=False, convergence_diagnostics={}, - ) == ('✅', 'posterior available') - assert _format_convergence_summary({}) is None - assert _format_convergence_summary({ - 'converged': False, - 'max_r_hat': 1.02, - 'min_ess_bulk': 200.0, - 'n_draws': 30, - 'n_chains': 8, - }) == ( - 'status=[red]failed[/red], max_r_hat=[red]1.020[/red], ' - 'min_ess_bulk=[red]200.0[/red], draws=30, chains=8' - ) + ) == 'failed' metrics = _calculate_fit_quality_metrics( y_obs=[10.0, 20.0], @@ -339,22 +321,24 @@ def test_bayesian_fit_results_display_results_prints_sampler_and_convergence(cap out = _unstyled_output(capsys.readouterr().out) assert 'Bayesian fit results' in out - assert 'Overall status: completed with warnings' in out - assert 'Sampler status: DREAM sampling completed' in out - assert 'Sampler: dream' in out - assert 'Sampler completed: yes' in out - assert 'steps=200' in out - assert 'init=lhs' in out - assert 'random_seed=1313900679' not in out - assert 'status=failed' in out - assert 'max_r_hat=1.107' in out - assert 'min_ess_bulk=125.9' in out - assert 'Posterior parameter summaries:' in out + assert 'Overall status' in out + assert 'failed' in out # convergence failed → overall failed + assert 'DREAM sampling completed' in out # engine message + assert 'Sampler' in out + assert 'Convergence status' in out + assert 'Max r-hat' in out + assert '1.107' in out + assert 'Min ess bulk' in out + assert '125.9' in out + assert 'Posterior distribution:' in out assert 'Success: True' not in out + assert 'Sampler completed' not in out # dropped — redundant with Overall status + assert 'Sampler settings' not in out # dropped — covered by Settings used table + assert 'Committed point estimate' not in out # dropped — covered by footnote assert 'datablock' in out assert 'category' in out assert 'entry' in out - assert '95% interval' in out + assert '95% CI' in out assert '68% interval' not in out assert 'std' not in out @@ -424,8 +408,8 @@ def fake_render_table(*, columns_headers, columns_alignment, columns_data): 'parameter', 'units', 'start', - 'best posterior sample', - 'uncertainty', + 'value', + 's.u.', 'change', ] assert captured['columns_alignment'] == [ @@ -491,7 +475,7 @@ def fake_render_table(*, columns_headers, columns_alignment, columns_data): 'parameter', 'units', 'median', - '95% interval', + '95% CI', 'r-hat', 'ess bulk', ] @@ -605,14 +589,16 @@ def test_fitresults_display_results_prints_and_table(capsys): ) out = _unstyled_output(capsys.readouterr().out) - assert 'Fit results' in out - assert 'Success: True' in out + assert 'Least-squares fit results:' in out + assert 'Overall status' in out + assert 'success' in out assert 'reduced χ²' in out - assert 'R-factor (Rf)' in out - assert 'R-factor squared (Rf²)' in out - assert 'Weighted R-factor (wR)' in out - assert 'Bragg R-factor (BR)' in out - assert 'Fitted parameters:' in out + assert 'R-factor (Rf' in out + assert 'R-factor squared (Rf²' in out + assert 'Weighted R-factor (wR' in out + assert 'Bragg R-factor (BR' in out + assert 'Refined parameters:' in out + assert 'Success: True' not in out # replaced by Overall status row assert any(char in out for char in ('╒', '┌', '+', '─')) @@ -640,8 +626,8 @@ def fake_render_table(*, columns_headers, columns_alignment, columns_data): 'parameter', 'units', 'start', - 'fitted', - 'uncertainty', + 'value', + 's.u.', 'change', ] assert captured['columns_alignment'] == [ diff --git a/tests/unit/easydiffraction/analysis/categories/fit_result/test_bayesian.py b/tests/unit/easydiffraction/analysis/categories/fit_result/test_bayesian.py index c9e5e560d..f636170dd 100644 --- a/tests/unit/easydiffraction/analysis/categories/fit_result/test_bayesian.py +++ b/tests/unit/easydiffraction/analysis/categories/fit_result/test_bayesian.py @@ -66,6 +66,19 @@ def test_bayesian_fit_result_omits_optional_unknown_outputs(): assert '_fit_result.resolved_random_seed' not in cif_text +def test_bayesian_fit_result_omits_redundant_iterations(): + from easydiffraction.analysis.categories.fit_result.bayesian import ( + BayesianFitResult, + ) + + fit_result = BayesianFitResult() + fit_result._set_iterations(100) + + cif_text = fit_result.as_cif + + assert '_fit_result.iterations' not in cif_text + + def test_bayesian_fit_result_keeps_optional_outputs_when_populated(): from easydiffraction.analysis.categories.fit_result.bayesian import ( BayesianFitResult, diff --git a/tests/unit/easydiffraction/analysis/categories/minimizer/test_bayesian_base.py b/tests/unit/easydiffraction/analysis/categories/minimizer/test_bayesian_base.py index 576ac976a..bfffd94b2 100644 --- a/tests/unit/easydiffraction/analysis/categories/minimizer/test_bayesian_base.py +++ b/tests/unit/easydiffraction/analysis/categories/minimizer/test_bayesian_base.py @@ -43,6 +43,29 @@ def test_bayesian_minimizer_rejects_unsupported_initialization_method(): minimizer.initialization_method = 'ball' +def test_bayesian_minimizer_keeps_unset_random_seed_in_cif(): + from easydiffraction.analysis.categories.minimizer.bumps_dream import ( + BumpsDreamMinimizer, + ) + + cif_text = BumpsDreamMinimizer().as_cif + + assert '_minimizer.random_seed ?' in cif_text + + +def test_bayesian_minimizer_keeps_configured_random_seed_in_cif(): + from easydiffraction.analysis.categories.minimizer.bumps_dream import ( + BumpsDreamMinimizer, + ) + + minimizer = BumpsDreamMinimizer() + minimizer.random_seed = 123 + + cif_text = minimizer.as_cif + + assert '_minimizer.random_seed 123' in cif_text + + def test_bayesian_minimizer_reads_cif_unknown_values_as_defaults(): from easydiffraction.analysis.categories.minimizer.bumps_dream import ( BumpsDreamMinimizer, diff --git a/tests/unit/easydiffraction/analysis/fit_helpers/test_bayesian.py b/tests/unit/easydiffraction/analysis/fit_helpers/test_bayesian.py index 554131130..960214bbd 100644 --- a/tests/unit/easydiffraction/analysis/fit_helpers/test_bayesian.py +++ b/tests/unit/easydiffraction/analysis/fit_helpers/test_bayesian.py @@ -208,22 +208,24 @@ def test_bayesian_fit_results_display_results_prints_sampler_and_convergence(cap out = capsys.readouterr().out assert 'Bayesian fit results' in out - assert 'Overall status: completed with warnings' in out - assert 'Sampler status: DREAM sampling completed' in out - assert 'Sampler: dream' in out - assert 'Sampler completed: yes' in out - assert 'steps=200' in out - assert 'init=lhs' in out - assert 'random_seed=1313900679' not in out - assert 'status=failed' in out - assert 'max_r_hat=1.107' in out - assert 'min_ess_bulk=125.9' in out - assert 'Posterior parameter summaries:' in out + assert 'Overall status' in out + assert 'failed' in out # convergence failed → overall failed + assert 'DREAM sampling completed' in out # engine message + assert 'Sampler' in out + assert 'Convergence status' in out + assert 'Max r-hat' in out + assert '1.107' in out + assert 'Min ess bulk' in out + assert '125.9' in out + assert 'Posterior distribution:' in out assert 'Success: True' not in out + assert 'Sampler completed' not in out # dropped — redundant with Overall status + assert 'Sampler settings' not in out # dropped — covered by Settings used table + assert 'Committed point estimate' not in out # dropped — covered by footnote assert 'datablock' in out assert 'category' in out assert 'entry' in out - assert '95% interval' in out + assert '95% CI' in out assert '68% interval' not in out assert 'std' not in out @@ -285,8 +287,8 @@ def fake_render_table(*, columns_headers, columns_alignment, columns_data): 'parameter', 'units', 'start', - 'best posterior sample', - 'uncertainty', + 'value', + 's.u.', 'change', ] assert captured['columns_alignment'] == [ @@ -352,7 +354,7 @@ def fake_render_table(*, columns_headers, columns_alignment, columns_data): 'parameter', 'units', 'median', - '95% interval', + '95% CI', 'r-hat', 'ess bulk', ] diff --git a/tests/unit/easydiffraction/analysis/fit_helpers/test_reporting.py b/tests/unit/easydiffraction/analysis/fit_helpers/test_reporting.py index 9758f70f8..ab21c0d79 100644 --- a/tests/unit/easydiffraction/analysis/fit_helpers/test_reporting.py +++ b/tests/unit/easydiffraction/analysis/fit_helpers/test_reporting.py @@ -49,14 +49,16 @@ def __init__(self, start, value, uncertainty, name='p', units='u'): # Assert: key lines printed and a table rendered out = capsys.readouterr().out - assert 'Fit results' in out - assert 'Success: True' in out + assert 'Least-squares fit results:' in out + assert 'Overall status' in out + assert 'success' in out assert 'reduced χ²' in out - assert 'R-factor (Rf)' in out - assert 'R-factor squared (Rf²)' in out - assert 'Weighted R-factor (wR)' in out - assert 'Bragg R-factor (BR)' in out - assert 'Fitted parameters:' in out + assert 'R-factor (Rf' in out + assert 'R-factor squared (Rf²' in out + assert 'Weighted R-factor (wR' in out + assert 'Bragg R-factor (BR' in out + assert 'Refined parameters:' in out + assert 'Success: True' not in out # replaced by Overall status row # Table border: accept common border glyphs from Rich assert any(ch in out for ch in ('╒', '┌', '+', '─')) @@ -97,8 +99,8 @@ def fake_render_table(*, columns_headers, columns_alignment, columns_data): 'parameter', 'units', 'start', - 'fitted', - 'uncertainty', + 'value', + 's.u.', 'change', ] assert captured['columns_alignment'] == [ diff --git a/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py b/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py index e7a6c37e7..a89f0f262 100644 --- a/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py +++ b/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py @@ -74,6 +74,45 @@ def test_emcee_minimizer_defaults_to_de_without_thinning(): assert minimizer.thin == 1 +def test_emcee_best_sample_reduced_chi_square_uses_objective_residuals(): + from easydiffraction.analysis.minimizers.emcee import EmceeMinimizer + + reduced_chi_square = EmceeMinimizer._best_sample_reduced_chi_square( + objective_function=lambda params: np.asarray( + [params['a'] - 1.0, params['b'] - 2.0, 2.0, 4.0], + dtype=float, + ), + parameter_names=['a', 'b'], + best_sample_values=np.asarray([2.0, 4.0], dtype=float), + ) + + assert reduced_chi_square == pytest.approx(12.5) + + +def test_emcee_build_fit_results_prefers_raw_reduced_chi_square(): + from easydiffraction.analysis.minimizers.emcee import EmceeMinimizer + + minimizer = EmceeMinimizer() + raw_result = SimpleNamespace( + reduced_chi_square=1.25, + raw_state=object(), + sampler_settings={}, + convergence_diagnostics={}, + posterior_parameter_summaries=[], + sampler_completed=True, + best_log_posterior=-10.0, + message='emcee sampling completed', + ) + + fit_results = minimizer._build_fit_results( + parameters=[], + raw_result=raw_result, + success=True, + ) + + assert fit_results.reduced_chi_square == 1.25 + + def test_emcee_pool_context_uses_fork_worker_for_unpicklable_objective(monkeypatch): from easydiffraction.analysis.minimizers.emcee import EmceeMinimizer from easydiffraction.analysis.minimizers.emcee import _emcee_log_prob_worker diff --git a/tests/unit/easydiffraction/analysis/test_analysis.py b/tests/unit/easydiffraction/analysis/test_analysis.py index bc022506a..0b83fdd38 100644 --- a/tests/unit/easydiffraction/analysis/test_analysis.py +++ b/tests/unit/easydiffraction/analysis/test_analysis.py @@ -100,10 +100,22 @@ def test_minimizer_selector_swap_warns_for_different_defaults(monkeypatch): # Inter-family swap should split warnings into "removed"/"added" # lines rather than emitting "" sentinels per # finding F3. - assert any('removes these settings' in w and 'max_iterations' in w for w in warnings) - assert any( - 'adds these settings with defaults' in w and 'sampling_steps' in w for w in warnings + removed_warning = next(w for w in warnings if 'removes these settings' in w) + added_warning = next(w for w in warnings if 'adds these settings' in w) + assert removed_warning == ( + 'Switching minimizer type removes these settings:\n' + '• max_iterations' ) + assert added_warning.splitlines() == [ + 'Switching minimizer type adds these settings with defaults:', + '• burn_in_steps=600', + "• initialization_method='latin_hypercube'", + '• parallel_workers=0', + '• population_size=4', + '• random_seed=None', + '• sampling_steps=3000', + '• thinning_interval=1', + ] assert not any('' in w for w in warnings) @@ -143,6 +155,79 @@ def test_store_posterior_projection_persists_resolved_random_seed(): assert analysis.fit_result.resolved_random_seed.value == 12345 +def test_restored_bayesian_diagnostics_reconstruct_passed_status(): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.categories.fit_result.bayesian import BayesianFitResult + + analysis = Analysis(project=_make_project_with_names([])) + analysis._fit_result._parent = None + analysis._fit_result = BayesianFitResult() + analysis._fit_result._parent = analysis + analysis.fit_result._set_gelman_rubin_max(1.002) + analysis.fit_result._set_effective_sample_size_min(8810.5) + analysis.fit_result._set_acceptance_rate_mean(0.3) + + diagnostics = analysis._restored_bayesian_convergence_diagnostics( + sample_shape=(10001, 16, 5), + n_parameters=5, + ) + + assert diagnostics['converged'] is True + assert diagnostics['max_r_hat'] == 1.002 + assert diagnostics['min_ess_bulk'] == 8810.5 + assert diagnostics['acceptance_rate_mean'] == 0.3 + assert diagnostics['n_draws'] == 10001 + assert diagnostics['n_chains'] == 16 + assert diagnostics['n_parameters'] == 5 + + +def test_restored_bayesian_sampler_settings_reconstruct_sample_count(): + from easydiffraction.analysis.analysis import Analysis + + analysis = Analysis(project=_make_project_with_names([])) + analysis.minimizer.type = 'emcee' + + settings = analysis._restored_bayesian_sampler_settings( + { + 'nsteps': 10000, + 'nburn': 2000, + 'thin': 1, + 'nwalkers': 16, + 'parallel_workers': 0, + 'initialization_method': 'ball', + 'proposal_moves': 'de', + }, + random_seed=123, + n_parameters=5, + ) + + assert settings['steps'] == 10000 + assert settings['burn'] == 2000 + assert settings['thin'] == 1 + assert settings['pop'] == 16 + assert settings['samples'] == 800000 + assert settings['random_seed'] == 123 + + +def test_restored_bayesian_reduced_chi_square_recovers_from_log_posterior(monkeypatch): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.categories.fit_result.bayesian import BayesianFitResult + + analysis = Analysis(project=_make_project_with_names([])) + analysis._fit_result._parent = None + analysis._fit_result = BayesianFitResult() + analysis._fit_result._parent = analysis + analysis.fit_result._set_best_log_posterior(-50.0) + monkeypatch.setattr(analysis, '_fit_data_point_count', lambda experiments: 102) + + reduced_chi_square = analysis._restored_bayesian_reduced_chi_square( + float('nan'), + restored_parameters=[object(), object()], + ) + + assert reduced_chi_square == 1.0 + + def test_fit_interrupt_cleans_state_and_prints_message(monkeypatch, capsys): from easydiffraction.analysis import analysis as analysis_mod from easydiffraction.analysis.analysis import Analysis diff --git a/tests/unit/easydiffraction/datablocks/experiment/item/test_base.py b/tests/unit/easydiffraction/datablocks/experiment/item/test_base.py index 185a21f30..f2e808c5f 100644 --- a/tests/unit/easydiffraction/datablocks/experiment/item/test_base.py +++ b/tests/unit/easydiffraction/datablocks/experiment/item/test_base.py @@ -40,6 +40,112 @@ def _load_ascii_data_to_experiment(self, data_path: str) -> int: assert ex.peak.type == 'cwl-pseudo-voigt' +def test_pd_experiment_peak_profile_switch_warning_lists_added_settings(monkeypatch): + from easydiffraction.datablocks.experiment.categories.experiment_type import ExperimentType + from easydiffraction.datablocks.experiment.item import base as item_base + from easydiffraction.datablocks.experiment.item.base import PdExperimentBase + from easydiffraction.datablocks.experiment.item.enums import BeamModeEnum + from easydiffraction.datablocks.experiment.item.enums import RadiationProbeEnum + from easydiffraction.datablocks.experiment.item.enums import SampleFormEnum + from easydiffraction.datablocks.experiment.item.enums import ScatteringTypeEnum + + class ConcretePd(PdExperimentBase): + def _load_ascii_data_to_experiment(self, data_path: str) -> int: + return 0 + + et = ExperimentType() + et._set_sample_form(SampleFormEnum.POWDER.value) + et._set_beam_mode(BeamModeEnum.CONSTANT_WAVELENGTH.value) + et._set_radiation_probe(RadiationProbeEnum.NEUTRON.value) + et._set_scattering_type(ScatteringTypeEnum.BRAGG.value) + + warnings: list[str] = [] + monkeypatch.setattr(item_base.log, 'warning', warnings.append) + ex = ConcretePd(name='ex1', type=et) + + ex.peak.type = 'pseudo-voigt + empirical asymmetry' + + assert warnings == [ + ( + 'Switching peak profile type adds these settings with defaults:\n' + '• asym_empir_1=0.0\n' + '• asym_empir_2=0.0\n' + '• asym_empir_3=0.0\n' + '• asym_empir_4=0.0' + ) + ] + + +def test_pd_experiment_peak_profile_switch_warning_lists_reset_settings(monkeypatch): + from easydiffraction.datablocks.experiment.categories.experiment_type import ExperimentType + from easydiffraction.datablocks.experiment.item import base as item_base + from easydiffraction.datablocks.experiment.item.base import PdExperimentBase + from easydiffraction.datablocks.experiment.item.enums import BeamModeEnum + from easydiffraction.datablocks.experiment.item.enums import RadiationProbeEnum + from easydiffraction.datablocks.experiment.item.enums import SampleFormEnum + from easydiffraction.datablocks.experiment.item.enums import ScatteringTypeEnum + + class ConcretePd(PdExperimentBase): + def _load_ascii_data_to_experiment(self, data_path: str) -> int: + return 0 + + et = ExperimentType() + et._set_sample_form(SampleFormEnum.POWDER.value) + et._set_beam_mode(BeamModeEnum.CONSTANT_WAVELENGTH.value) + et._set_radiation_probe(RadiationProbeEnum.NEUTRON.value) + et._set_scattering_type(ScatteringTypeEnum.BRAGG.value) + + warnings: list[str] = [] + monkeypatch.setattr(item_base.log, 'warning', warnings.append) + ex = ConcretePd(name='ex1', type=et) + ex.peak.broad_gauss_u = 0.05 + + ex.peak.type = 'pseudo-voigt + empirical asymmetry' + + assert warnings[1] == ( + 'Switching peak profile type resets these settings to defaults:\n' + '• broad_gauss_u: 0.05 -> 0.01' + ) + + +def test_pd_experiment_peak_profile_switch_warning_lists_removed_settings(monkeypatch): + from easydiffraction.datablocks.experiment.categories.experiment_type import ExperimentType + from easydiffraction.datablocks.experiment.item import base as item_base + from easydiffraction.datablocks.experiment.item.base import PdExperimentBase + from easydiffraction.datablocks.experiment.item.enums import BeamModeEnum + from easydiffraction.datablocks.experiment.item.enums import RadiationProbeEnum + from easydiffraction.datablocks.experiment.item.enums import SampleFormEnum + from easydiffraction.datablocks.experiment.item.enums import ScatteringTypeEnum + + class ConcretePd(PdExperimentBase): + def _load_ascii_data_to_experiment(self, data_path: str) -> int: + return 0 + + et = ExperimentType() + et._set_sample_form(SampleFormEnum.POWDER.value) + et._set_beam_mode(BeamModeEnum.CONSTANT_WAVELENGTH.value) + et._set_radiation_probe(RadiationProbeEnum.NEUTRON.value) + et._set_scattering_type(ScatteringTypeEnum.BRAGG.value) + + warnings: list[str] = [] + monkeypatch.setattr(item_base.log, 'warning', warnings.append) + ex = ConcretePd(name='ex1', type=et) + ex.peak.type = 'pseudo-voigt + empirical asymmetry' + warnings.clear() + + ex.peak.type = 'pseudo-voigt' + + assert warnings == [ + ( + 'Switching peak profile type removes these settings:\n' + '• asym_empir_1\n' + '• asym_empir_2\n' + '• asym_empir_3\n' + '• asym_empir_4' + ) + ] + + def test_pd_experiment_set_peak_profile_type_silent(capsys): """_set_peak_profile_type switches the peak type without console output.""" from easydiffraction.datablocks.experiment.categories.experiment_type import ExperimentType diff --git a/tests/unit/easydiffraction/utils/test_utils.py b/tests/unit/easydiffraction/utils/test_utils.py index 3fb165574..1d6a8b23e 100644 --- a/tests/unit/easydiffraction/utils/test_utils.py +++ b/tests/unit/easydiffraction/utils/test_utils.py @@ -13,6 +13,14 @@ def test_module_import(): assert expected_module_name == actual_module_name +def test_format_bulleted_warning(): + import easydiffraction.utils.utils as MUT + + warning = MUT.format_bulleted_warning('Header:', ['first', 'second']) + + assert warning == 'Header:\n• first\n• second' + + def test_twotheta_to_d_scalar_and_array(): import easydiffraction.utils.utils as MUT From f5c014e56d664a1782bc2b59829c141c2e80f9d2 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 21:00:55 +0200 Subject: [PATCH 35/65] Add ADR on fit results display naming --- .../accepted/fit-results-display-naming.md | 345 ++++++++++++++++++ docs/dev/adrs/index.md | 1 + 2 files changed, 346 insertions(+) create mode 100644 docs/dev/adrs/accepted/fit-results-display-naming.md diff --git a/docs/dev/adrs/accepted/fit-results-display-naming.md b/docs/dev/adrs/accepted/fit-results-display-naming.md new file mode 100644 index 000000000..5947565ef --- /dev/null +++ b/docs/dev/adrs/accepted/fit-results-display-naming.md @@ -0,0 +1,345 @@ +# ADR: Fit Results Display Naming Convention + +## Status + +Accepted. + +## Date + +2026-05-25 + +## Group + +User-facing API. + +## Context + +`project.display.fit.results()` and `project.display.posterior.*` (see +[`display-ux.md`](display-ux.md)) currently emit fit-result tables with +inconsistent and sometimes long column headers across the two fitting +modes: + +- **LSQ:** `📈 Fitted parameters:` with columns + `start | fitted | uncertainty | change`. +- **Bayesian:** two tables. + - `📈 Committed parameters:` with columns + `start | best posterior sample | uncertainty | change`. + - `📊 Posterior parameter summaries:` with columns + `median | 95% interval | r-hat | ess bulk`. + +Three problems: + +1. `best posterior sample` (21 chars) is too wide for HTML / markdown + layouts and forces the other columns into narrow space. +2. `uncertainty` is the column header in both LSQ and Bayesian + committed tables but the underlying quantities differ + (covariance-derived σ vs posterior SD). The display layer does not + annotate the difference. +3. LSQ's `fitted` and Bayesian's `best posterior sample` are + conceptually parallel (the value committed back to the project) but + the headers do not signal that parallelism, complicating + side-by-side reading. + +Two conventions guide the cross-method naming choice: + +- **IUCr CIF** prefers the `_su` suffix (standard uncertainty); `_esd` + (estimated standard deviation) is deprecated. +- **GUM** (Guide to the Expression of Uncertainty in Measurement) + treats Bayesian posterior SD and frequentist standard uncertainty as + the same physical quantity — 1σ of the inferred distribution of the + measurand. + +Both converge on `s.u.` as the appropriate cross-method label. + +[`display-ux.md`](display-ux.md) defines facade method names but not +column headers or footnotes; +[`iucr-cif-tag-alignment.md`](../suggestions/iucr-cif-tag-alignment.md) +defines persisted CIF tag names but not display labels; +[`analysis-cif-fit-state.md`](analysis-cif-fit-state.md) defines +Python and CIF attribute names but not user-visible labels. Display +naming for fit-results tables is a real gap. + +## Decision + +### 1. Short headers paired with a footnote glossary + +Every fit-results table emits a glossary block immediately below the +table that expands the short column headers into one-line +descriptions. The footnote disambiguates per fitting mode so the +column header itself can stay short. + +### 2. Cross-method consistency where the physical quantity is the same + +Same column header where the underlying physical quantity matches: + +- `start` — initial parameter value, both modes. +- `value` — refined / committed value, both modes. +- `s.u.` — 1σ standard uncertainty, both modes (covariance for LSQ, + posterior SD for Bayesian; same physical meaning per GUM). +- `change` — `value − start`, both modes. + +Different headers only for Bayesian-only quantities (no LSQ analogue): +`median`, `95% CI`, `r-hat`, `ess bulk`. + +### 3. Canonical column layouts and titles + +**LSQ — `📈 Refined parameters:`** + +``` +| datablock | category | entry | parameter | units | start | value | s.u. | change | +``` + +Footnote: + +``` +start = parameter value before refinement +value = refined value from least-squares minimization +s.u. = standard uncertainty (1σ), from the covariance matrix +change = relative change from start, in %; ↑ = increase, ↓ = decrease +``` + +**Bayesian — `📈 Committed parameters:`** (title unchanged) + +``` +| datablock | category | entry | parameter | units | start | value | s.u. | change | +``` + +Footnote: + +``` +start = parameter value before sampling +value = estimate written back to the project (best posterior sample) +s.u. = standard uncertainty (1σ), the posterior standard deviation +change = relative change from start, in %; ↑ = increase, ↓ = decrease +``` + +**Bayesian — `📊 Posterior distribution:`** + +``` +| datablock | category | entry | parameter | units | median | 95% CI | r-hat | ess bulk | +``` + +Footnote: + +``` +median = 50th percentile of the marginal posterior +95% CI = 95% credible interval (2.5%–97.5%, asymmetric) +r-hat = Gelman–Rubin diagnostic (good convergence: r-hat ≤ 1.01) +ess bulk = bulk effective sample size (typically ≥ 400) +``` + +### 4. Title changes from the current implementation + +- `📈 Fitted parameters:` → `📈 Refined parameters:` (IUCr-style + "refinement" wording, also matches the cross-method `value` column). +- `📈 Committed parameters:` stays unchanged — the duality of + "committed values" vs "posterior distribution" is meaningful and + worth preserving on the Bayesian side. +- `📊 Posterior parameter summaries:` → `📊 Posterior distribution:` + (shorter and explicit about what the second table shows). + +### 5. Chart legend convention + +Chart legends use the full footnote-form name where the chart has +horizontal space. Where the plot title already signals context (e.g. +"Posterior distribution of "), legends may shorten to the +table-header form: + +- Posterior distribution plots: `estimate`, `median`, + `95% credible interval`. +- Measured-vs-calculated plots: `measured`, `calculated`. + +Existing chart legends that describe plot **type** (e.g. +`Marginal density`, `Posterior contours`, `Posterior samples`) are +not parameter-value labels and are out of scope for this ADR. + +### 6. Internal attribute names unchanged + +`Parameter.value`, `Parameter.uncertainty`, +`Parameter.posterior_uncertainty`, and every persisted CIF tag stay +as they are. This ADR governs **display strings only**, not the +Python or CIF API. + +## Addendum (2026-05-25): Fit-results table replaces emoji-line summary + +The original ADR specified two parameter-level tables for Bayesian +fits (`Committed parameters`, `Posterior distribution`) and one for +LSQ (`Refined parameters`), each below an emoji-line summary block +(`✅ Success: True`, `📏 Goodness-of-fit (reduced χ²): 1.29`, …). In +practice the emoji-line block grew long, mixed multi-value lines +(`📊 Convergence: status=passed, max_r_hat=1.004, …`) with +single-value lines (`📏 R-factor (Rf): 5.65%`), and split related +information across visually-different formats. + +The block is now rendered as **one additional 2-column table** per +fit method, sitting directly above the parameter tables: + +- LSQ: `📋 Least-squares fit results:` — title. +- Bayesian: `📋 Bayesian fit results:` — title. + +Column layout: `Metric | Value`, left/right alignment. Each row +carries one emoji-prefixed metric name in the first column and one +scalar value in the second. The previous `console.paragraph('Fit +results')` / `console.paragraph('Bayesian fit results')` section +header is dropped — the table title now signals the section. + +Canonical row order (top-to-bottom): + +1. `🧪 Minimizer` / `🧪 Sampler` — the minimizer.type string + (e.g. `lmfit (leastsq)`, `bumps (dream)`). +2. `✅ Overall status` — single shared value vocabulary: + `success` / `failed`. For LSQ this mirrors `FitResults.success`. + For Bayesian this is `success` only when the sampler completed + *and* convergence passed, else `failed`. Per-metric convergence + detail goes in rows 12–16 below. +3. `💬 Engine message` *(Bayesian, optional)* — the engine's + free-form status message, e.g. `DREAM sampling completed`. +4. `⏱️ Fitting time (seconds)` — `fitting_time`. +5. `🔁 Iterations` *(LSQ, optional)* — shown only when + `FitResults.iterations > 0`. +6. `📏 Goodness-of-fit (reduced χ²)` — `reduced_chi_square`. +7–10. `📏 R-factor (Rf, %)`, `📏 R-factor squared (Rf², %)`, + `📏 Weighted R-factor (wR, %)`, `📏 Bragg R-factor (BR, %)` — + each row when the corresponding inputs are available. Units + appear in the metric name, so the value cell holds a bare + number. (R-factors come immediately after goodness-of-fit and + before `Best log-posterior` — both methods agree on this + order.) +11. `📉 Best log-posterior` *(Bayesian, optional)* — shown when + `best_log_posterior is not None`. +12–16. *(Bayesian only)* Convergence rows derived from + `convergence_diagnostics`: + - `📊 Convergence status` — `passed` / `failed`. + - `📊 Max r-hat` — formatted to 3 decimals. + - `📊 Min ess bulk` — formatted to 1 decimal. + - `📊 Draws per chain`. + - `📊 Chains`. + +The shared-vocabulary `success` / `failed` for `Overall status` is +intentional cross-method consistency: a reader scanning LSQ and +Bayesian outputs side-by-side sees the same status word in the +same row position regardless of method. Bayesian-specific nuance +(sampler completed but convergence flagged, etc.) is exposed in +the convergence rows below. + +**Rows dropped relative to the previous emoji-line summary:** + +- `🎯 Committed point estimate: Best posterior sample` — already + documented by the `Committed parameters` table footnote + (`value = estimate written back to the project (best posterior + sample)`). +- `🔁 Sampler completed: yes` — redundant with `Overall status`. +- `⚙️ Sampler settings: steps=…, burn=…, …` — already in the + `Settings used` table above the fit-results table. +- The derived `samples = n_draws × n_chains` count — derived from + the `Draws per chain` and `Chains` rows immediately below. + +**Table-title icons.** The four fit-output tables now carry a +distinguishing icon in their title so the four blocks are visually +separable when scrolling: + +| Table | Title prefix | +| --- | --- | +| Minimizer settings | `⚙️ Settings used:` | +| Fit-method summary | `📋 Least-squares fit results:` / `📋 Bayesian fit results:` | +| Committed values | `📈 Refined parameters:` / `📈 Committed parameters:` | +| Posterior summary (Bayesian only) | `📊 Posterior distribution:` | + +The icons are also the same emoji used inside the rows of the +corresponding fit-results-summary table (📏 for goodness-of-fit +metrics, 📊 for convergence diagnostics), so the visual language +is internally consistent. + +**Internal-implementation note.** Helper +`print_metrics_table(rows)` in +`easydiffraction.utils.utils` renders the new 2-column table from +a list of `[label, value]` rows. Both +`reporting.FitResults.display_results()` and +`bayesian.BayesianFitResults.display_results()` build their rows +via a `_build_fit_results_rows()` instance method and feed +`print_metrics_table()`. The shared signature keeps the two +methods structurally parallel. + +## Consequences + +### Positive + +- Tables fit standard HTML / markdown width without truncating the + formerly 21-character `best posterior sample` column. +- Users can compare LSQ and Bayesian results column-by-column + (`start`, `value`, `s.u.`, `change` line up identically). +- IUCr / GUM-aligned terminology. +- The inline footnote glossary gives non-programmer users a + discoverability path without having to leave the table to read + external docs. +- Setting the convention in an ADR keeps future fit-result tables (a + new sampler, an alternative refinement strategy) on the same + naming. + +### Trade-offs + +- Existing tutorials, tests, and integration outputs that pin the + literal strings `Fitted parameters`, `Posterior parameter summaries`, + `fitted`, `best posterior sample`, `uncertainty`, `95% interval` + need updating in the implementation PR. +- `s.u.` is unfamiliar to readers who do not know GUM or IUCr CIF. + The footnote covers this; the compactness win at the column header + is the main argument. + +### ADRs related to this ADR + +None directly amended. This ADR complements: + +- [`display-ux.md`](display-ux.md) — defines facade method names; this + ADR fills in the column-header layer underneath. +- [`iucr-cif-tag-alignment.md`](../suggestions/iucr-cif-tag-alignment.md) + — defines persisted CIF tag names; this ADR is the matching + display-time label layer. +- [`analysis-cif-fit-state.md`](analysis-cif-fit-state.md) — defines + Python / CIF attribute names (e.g. `Parameter.uncertainty`, + `posterior_uncertainty`); display headers map to those without + renaming them. + +## Alternatives Considered + +### A. Keep `uncertainty` as the column header for both modes + +Pros: zero changes. Cons: ambiguous in Bayesian context (users may +confuse it with the 95% credible interval below); inconsistent with +the IUCr CIF `_su` convention; reinforces the wider `best posterior +sample` problem because it does not solve the layout issue. + +### B. `posterior SD` for Bayesian, `uncertainty` for LSQ + +Pros: explicit on the Bayesian side. Cons: different column headers +for the same physical quantity (1σ width), breaking the +column-by-column comparison; longer (10 chars vs 4 for `s.u.`). + +### C. Different headers for the committed-value column (`refined` vs +`estimate` vs `value`) + +Three different headers for "the value committed to the project". +Pros: each method-accurate. Cons: breaks the cross-method consistency +goal; readers seeing `refined` next to `estimate` in side-by-side +tables wonder what the semantic difference is even though the +underlying quantity is the same. Decision: use neutral `value` +everywhere, let the footnote disambiguate. + +### D. Single Bayesian table covering both committed values and +posterior summary + +Pros: one table to read. Cons: nine value columns plus identity +columns exceed standard HTML width and truncate. The two-table split +is forced by layout and meaningfully preserves the "what did I +commit" vs "what does the posterior look like" duality. + +## Deferred Work + +- The `acceptance rate` column in the posterior distribution table. + Not displayed by default today; a future ADR can decide whether it + joins the canonical layout or stays in a verbose mode. +- Inline footnote text vs Markdown link to a docs-site glossary. + Inline is the initial form; promotion to a glossary page is a + future ADR if footnote lengths grow. +- Localisation. All display strings are English; non-English UIs are + out of scope. diff --git a/docs/dev/adrs/index.md b/docs/dev/adrs/index.md index fcdf7bbfa..c235042db 100644 --- a/docs/dev/adrs/index.md +++ b/docs/dev/adrs/index.md @@ -43,6 +43,7 @@ folders. | Quality | Accepted | Test Strategy | Defines layered unit, functional, integration, script, and notebook testing. | [`test-strategy.md`](accepted/test-strategy.md) | | Structure model | Accepted | Type-Neutral ADP Parameters | Keeps ADP parameter object identities stable across B/U and iso/ani switches. | [`type-neutral-adp-parameters.md`](accepted/type-neutral-adp-parameters.md) | | User-facing API | Accepted | Display UX Facade | Defines `project.display` and `project.rendering` responsibilities and display method names. | [`display-ux.md`](accepted/display-ux.md) | +| User-facing API | Accepted | Fit Results Display Naming | Short, IUCr/GUM-aligned column headers (`s.u.`, `value`, `95% CI`) with a footnote glossary on every fit table. | [`fit-results-display-naming.md`](accepted/fit-results-display-naming.md) | | User-facing API | Accepted | Selector Families | Distinguishes backend selectors, switchable-category selectors, and active-sibling selectors. | [`selector-families.md`](accepted/selector-families.md) | | User-facing API | Accepted | String Paths and Live Descriptors | Separates persisted field selectors from references to live model parameters. | [`string-paths-and-live-descriptors.md`](accepted/string-paths-and-live-descriptors.md) | | User-facing API | Accepted | Switchable Category API | Places multi-type category selectors on the owner and omits public selectors for fixed or single-type categories. | [`switchable-category-api.md`](accepted/switchable-category-api.md) | From 910f90089fed0f579ad5634a9c8c3f9d6a4e09c9 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 21:26:08 +0200 Subject: [PATCH 36/65] Replace arviz with custom diagnostics --- .../minimizer-category-consolidation.md | 13 + pixi.lock | 223 ++++++++---------- pyproject.toml | 1 - .../analysis/fit_helpers/_diagnostics.py | 172 ++++++++++++++ .../analysis/fit_helpers/bayesian.py | 67 ++---- .../analysis/fit_helpers/reporting.py | 11 +- src/easydiffraction/display/plotting.py | 29 +-- src/easydiffraction/utils/utils.py | 14 +- .../fitting/test_bayesian_helper_support.py | 36 +-- .../analysis/fit_helpers/test__diagnostics.py | 111 +++++++++ .../analysis/fit_helpers/test_bayesian.py | 37 +-- .../analysis/fit_helpers/test_reporting.py | 9 +- .../unit/easydiffraction/utils/test_utils.py | 73 ++++++ 13 files changed, 539 insertions(+), 257 deletions(-) create mode 100644 src/easydiffraction/analysis/fit_helpers/_diagnostics.py create mode 100644 tests/unit/easydiffraction/analysis/fit_helpers/test__diagnostics.py diff --git a/docs/dev/adrs/accepted/minimizer-category-consolidation.md b/docs/dev/adrs/accepted/minimizer-category-consolidation.md index 189b564cd..f99a169a8 100644 --- a/docs/dev/adrs/accepted/minimizer-category-consolidation.md +++ b/docs/dev/adrs/accepted/minimizer-category-consolidation.md @@ -231,6 +231,19 @@ Verbose CIF tags are user-facing. The canonical MCMC abbreviation field so it appears in `help()` output but does not become a Python attribute or a CIF tag. +**Implementation note (2026-05-25).** The per-parameter R̂ and bulk +ESS values feeding `gelman_rubin_max` and +`effective_sample_size_min` are computed by an in-tree helper at +[`analysis/fit_helpers/_diagnostics.py`](../../../../src/easydiffraction/analysis/fit_helpers/_diagnostics.py) +— pure NumPy + SciPy implementations of split R̂ and +rank-normalized bulk ESS (Vehtari, Gelman, Simpson, Carpenter and +Bürkner 2019; Geyer 1992). The earlier `arviz` dependency, which +the library only used to call `az.rhat()` and `az.ess(method='bulk')`, +has been removed; the diagnostics' public surface +(`gelman_rubin_max`, `effective_sample_size_min`, +`r_hat_by_parameter`, `ess_bulk_by_parameter`) and the convergence +thresholds (R̂ ≤ 1.01, ESS ≥ 400) are unchanged. + ### 6. Unified `initialization_method` enum A single `(str, Enum)` `InitializationMethodEnum` with members: diff --git a/pixi.lock b/pixi.lock index 5a9141de8..d69e22de0 100644 --- a/pixi.lock +++ b/pixi.lock @@ -96,7 +96,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/httpx-0.28.1-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/hyperframe-6.1.0-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/idna-3.15-pyhcf101f3_0.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-8.8.0-pyhcf101f3_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-9.0.0-pyhcf101f3_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipykernel-7.2.0-pyha191276_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython-9.13.0-pyh53cf698_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython_pygments_lexers-1.1.1-pyhd8ed1ab_0.conda @@ -178,7 +178,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/0b/2e/7eea398450457ecb54e18e9d10110993fa65561c4f3add5e8eccd2b9cd41/scipy-1.17.1-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/0b/f9/f15c95d6b200167cb22c5eca5eecfa9d28a8ee3f74095f1cd2345c71f2f9/pydoclint-0.8.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/0d/12/bbce9472f489cb5c4c23b0d13e5c59c37c1aab11b7ac637dfe6bbdccebe7/copier-9.15.1-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/0d/1f/d398de1612f7a611e22d743280339c9af4903675635e41be3370091c704b/arviz_stats-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/0d/fe/6bea5c9162869c5beba5d9c8abbed835ec85bf1ec1fba05a3822325c45f3/build-1.5.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/10/bd/c038d7cc38edc1aa5bf91ab8068b63d4308c66c4c8bb3cbba7dfbc049f9c/pyparsing-3.3.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/10/cb/f2ad4230dc2eb1a74edf38f1a38b9b52277f75bef262d8908e60d957e13c/blinker-1.9.0-py3-none-any.whl @@ -186,7 +185,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/11/8d/d2532ad2a603ca2b93ad9f5135732124e57811d0168155852f37fbce2458/pillow-12.2.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/12/c9/6869a1dcf4aaf309b9543ec070be3ec3adebee7c9bec9af8c230494134b9/interrogate-1.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/13/95/cf3f7fe4910cf0365fa8ea0c731f4b8a624d97cd76ea777913ac8d0868e2/mkdocs_jupyter-0.26.3-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/19/d4/225027a913621a879b429a043674aa35220e6ce67785acad4f7bd0c4ff33/xarray_einstats-0.10.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/1c/59/964ecb8008722d27d8a835baea81f56a91cea8e097b3be992bc6ccde6367/versioningit-3.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/1d/77/928ea2e70641ca177a11140062cc5840d421795f2e82749d408d0cce900a/narwhals-2.21.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/1e/77/dc8c558f7593132cf8fefec57c4f60c83b16941c574ac5f619abb3ae7933/dill-0.4.1-py3-none-any.whl @@ -202,7 +200,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2c/01/bc663630c510822c95c47a66af9fa7a443c295b47d5f041e5e6ae62ef659/mkdocs_material-9.7.6-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2f/40/dc34d1a8d5f1e51fc64640b62b191684da52ca469da9cd74e84936ffa4a6/msgpack-1.1.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - - pypi: https://files.pythonhosted.org/packages/2f/c8/005d1de3af80f54411703d1263a0b9d31276411ec9f273d9432c59b17299/arviz_plots-1.1.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/31/26/ef168b184a25701f9995e8fb7e503fafd7a99c1c77cda1bc1a26ea2ed486/sqlalchemy-2.0.50-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/5d/f7e914f7d9325abff4057cee62c0fa70263683189f774473cbfb534cd13b/matplotlib-3.10.9-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl @@ -222,7 +220,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/46/b4/0887c88ddfaba1d7140ea335144eb904af97550786ee58bdb295ff10d255/crysfml-0.6.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl - pypi: https://files.pythonhosted.org/packages/52/59/0782e51887ac6b07ffd1570e0364cf901ebc36345fea669969d2084baebb/simple_websocket-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/57/da/6fbf010c8ebb347679d0d100b22fe9ba5e13fd04046c5df7280d2f0bf706/sqlalchemy-2.0.50.tar.gz - pypi: https://files.pythonhosted.org/packages/58/3b/1cdec6772bdbaf7b25dab360c59f03cadf05492dd724c6540af905389b07/pandas-3.0.3-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/58/e0/f1871f520c359e4e3a2eb7437c9e7e792bb6c356414e8617937561167caf/pycifrw-5.0.1.tar.gz - pypi: https://files.pythonhosted.org/packages/5b/29/74eeb4d3f3ae61ca096b018ad486b3b3c74b17bec09ab4edab721cbefec3/typeguard-4.5.2-py3-none-any.whl @@ -230,8 +227,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/5b/69/93b34728cc386efdde0c342f8c680b9187dea7beb7adaf6b58a0713be101/mpld3-0.5.12-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/5f/fc/a7bf5b6e4e617b45f90f2d9d2a68519c249c81dd4fc2658c7a2a61c4f4b7/aiohappyeyeballs-2.6.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/61/2b/e260d50e64690d2a9e405d52ccd18a63c286c5088937dd0107cb23eb3195/diffpy_utils-3.7.2-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/64/a8/c09fbe44b12fa919c5bfe0afb71e60d1231a7dc93405e54c30496c57c9d3/arviz-1.1.0-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/69/d1/705e6c19b437a4105bf3b9ae7945fcfc3ad2abb73d14bae0a3f2d58b305b/arviz_base-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/6e/94/be70f8ee9c45f2f62b39a1f0e9303bc20e138a8f3b8e50ffd89498e177e1/mkdocstrings-1.0.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/72/b9/313e8f2f2e9517ae050a692ae7b3e4b3f17cc5e6dfea0db51fe14e586580/jinja2_ansible_filters-1.3.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/72/dc/0decaf5da92a7a969374474025787102d811d42aed1d32191fa338620e15/python_socketio-5.16.2-py3-none-any.whl @@ -250,7 +245,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/88/39/799be3f2f0f38cc727ee3b4f1445fe6d5e4133064ec2e4115069418a5bb6/cloudpickle-3.1.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/88/b2/d0896bdcdc8d28a7fc5717c305f1a861c26e18c05047949fb371034d98bd/nodeenv-1.10.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/89/1d/8eff589b45bb8190a9d12c49cfad0f176a5cbd1534908a6b5125e2886239/pydantic_core-2.46.4-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl - - pypi: https://files.pythonhosted.org/packages/8a/a1/8d812e53a5da1687abb10445275d41a8b13adb781bbf7196ddbcf8d88505/lazy_loader-0.5-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8e/5a/7fd1b784a87e96e0078f49a0a13a98b4c5f644ba5597a4a3b70a2ba3e613/py3dmol-2.5.5-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8f/5e/f1e1dd319e35e962a4e00b33150a8868b6329cc1d19fd533436ba5488f09/uncertainties-3.2.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/90/96/04b8e52da071d28f5e21a805b19cb9390aa17a47462ac87f5e2696b9566d/paginate-0.5.7-py2.py3-none-any.whl @@ -301,7 +295,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/d7/27/a58ddaf8c588a3ef080db9d0b7e0b97215cee3a45df74f3a94dbbf5c893a/pycodestyle-2.14.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/d7/51/ec641c26e6dca1b25a7d2035ba6ecb7c884ef1a100a9e42fbe4ce4405139/coverage-7.14.0-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl - pypi: https://files.pythonhosted.org/packages/db/3c/33bac158f8ab7f89b2e59426d5fe2e4f63f7ed25df84c036890172b412b5/cfgv-3.5.0-py2.py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/dc/83/6d810a8a9ebc9c307989b418840c20e46907c74d707beb67ab566773e6fc/xarray-2026.4.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl @@ -356,7 +349,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/httpx-0.28.1-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/hyperframe-6.1.0-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/idna-3.15-pyhcf101f3_0.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-8.8.0-pyhcf101f3_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-9.0.0-pyhcf101f3_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipykernel-7.2.0-pyh5552912_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython-9.13.0-pyh53cf698_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython_pygments_lexers-1.1.1-pyhd8ed1ab_0.conda @@ -489,14 +482,12 @@ environments: - pypi: https://files.pythonhosted.org/packages/06/41/4e70dea1d0311016c0b0b1c53a24a266f9f8a34c6bc1af0f17cfca20aa1d/gemmi-0.7.5-cp314-cp314-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/0b/f9/f15c95d6b200167cb22c5eca5eecfa9d28a8ee3f74095f1cd2345c71f2f9/pydoclint-0.8.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/0d/12/bbce9472f489cb5c4c23b0d13e5c59c37c1aab11b7ac637dfe6bbdccebe7/copier-9.15.1-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/0d/1f/d398de1612f7a611e22d743280339c9af4903675635e41be3370091c704b/arviz_stats-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/0d/fe/6bea5c9162869c5beba5d9c8abbed835ec85bf1ec1fba05a3822325c45f3/build-1.5.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/10/bd/c038d7cc38edc1aa5bf91ab8068b63d4308c66c4c8bb3cbba7dfbc049f9c/pyparsing-3.3.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/10/cb/f2ad4230dc2eb1a74edf38f1a38b9b52277f75bef262d8908e60d957e13c/blinker-1.9.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/11/8c/c9138d881c79aa0ea9ed83cbd58d5ca75624378b38cee225dcf5c42cc91f/griffelib-2.0.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/12/c9/6869a1dcf4aaf309b9543ec070be3ec3adebee7c9bec9af8c230494134b9/interrogate-1.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/13/95/cf3f7fe4910cf0365fa8ea0c731f4b8a624d97cd76ea777913ac8d0868e2/mkdocs_jupyter-0.26.3-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/19/d4/225027a913621a879b429a043674aa35220e6ce67785acad4f7bd0c4ff33/xarray_einstats-0.10.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/1b/9f/38ff9e57a2eade7bf9dfee5eae17f39fc0e998658050279cbb14d97d36d9/msgpack-1.1.2-cp314-cp314-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/1c/59/964ecb8008722d27d8a835baea81f56a91cea8e097b3be992bc6ccde6367/versioningit-3.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/1d/77/928ea2e70641ca177a11140062cc5840d421795f2e82749d408d0cce900a/narwhals-2.21.2-py3-none-any.whl @@ -512,7 +503,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/2a/2d/d4bf65e47cea8ff2c794a600c4fd1273a7902f268757c531e0ee9f18aa58/pooch-1.9.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/01/bc663630c510822c95c47a66af9fa7a443c295b47d5f041e5e6ae62ef659/mkdocs_material-9.7.6-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/2f/c8/005d1de3af80f54411703d1263a0b9d31276411ec9f273d9432c59b17299/arviz_plots-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/b2/986d1220f6ee931e338d272bc1f3ec02cfe5f9b5fad84e95afdad57f1ebc/format_docstring-0.2.7-py3-none-any.whl @@ -540,9 +530,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/60/61/23f27c172f022e04025b7dc2367f4d63c1a398120607ec896228649a6f48/numpy-2.4.6-cp314-cp314-macosx_14_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/61/2b/e260d50e64690d2a9e405d52ccd18a63c286c5088937dd0107cb23eb3195/diffpy_utils-3.7.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/63/b1/4260d67d6bd85e58a66b72d54ce15d5de789b6f3870cc6bedf8ff9667401/propcache-0.5.2-cp314-cp314-macosx_11_0_arm64.whl - - pypi: https://files.pythonhosted.org/packages/64/a8/c09fbe44b12fa919c5bfe0afb71e60d1231a7dc93405e54c30496c57c9d3/arviz-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/68/10/bf2d6738d72748b961a3751ab89522d58c54efc36a8e1a12161216cd45cf/pandas-3.0.3-cp314-cp314-macosx_11_0_arm64.whl - - pypi: https://files.pythonhosted.org/packages/69/d1/705e6c19b437a4105bf3b9ae7945fcfc3ad2abb73d14bae0a3f2d58b305b/arviz_base-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/6a/b7/9366ed44ced9b7ef357ab48c94205280276db9d7f064aa3012a97227e966/h5py-3.16.0-cp314-cp314-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/6e/94/be70f8ee9c45f2f62b39a1f0e9303bc20e138a8f3b8e50ffd89498e177e1/mkdocstrings-1.0.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/72/b9/313e8f2f2e9517ae050a692ae7b3e4b3f17cc5e6dfea0db51fe14e586580/jinja2_ansible_filters-1.3.2-py3-none-any.whl @@ -560,7 +548,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/88/29/744136411e785c4b0b744d5413e56555265939ab3a104c6a4b719dad33fd/mkdocs_get_deps-0.2.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/88/39/799be3f2f0f38cc727ee3b4f1445fe6d5e4133064ec2e4115069418a5bb6/cloudpickle-3.1.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/88/b2/d0896bdcdc8d28a7fc5717c305f1a861c26e18c05047949fb371034d98bd/nodeenv-1.10.0-py2.py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/8a/a1/8d812e53a5da1687abb10445275d41a8b13adb781bbf7196ddbcf8d88505/lazy_loader-0.5-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8e/5a/7fd1b784a87e96e0078f49a0a13a98b4c5f644ba5597a4a3b70a2ba3e613/py3dmol-2.5.5-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8f/5e/f1e1dd319e35e962a4e00b33150a8868b6329cc1d19fd533436ba5488f09/uncertainties-3.2.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/90/96/04b8e52da071d28f5e21a805b19cb9390aa17a47462ac87f5e2696b9566d/paginate-0.5.7-py2.py3-none-any.whl @@ -612,7 +599,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/db/3c/33bac158f8ab7f89b2e59426d5fe2e4f63f7ed25df84c036890172b412b5/cfgv-3.5.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/db/7b/8624a203326675d7746a254083a187398090a179335b2e4a20e2ddc46e83/scipy-1.17.1-cp314-cp314-macosx_14_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/db/7d/7f3d619e951c88ed75c6037b246ddcf2d322812ee8ea189be89511721d54/watchdog-6.0.0.tar.gz - - pypi: https://files.pythonhosted.org/packages/dc/83/6d810a8a9ebc9c307989b418840c20e46907c74d707beb67ab566773e6fc/xarray-2026.4.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/df/32/10ac51b4be7cdecd7e93d069251c86dfbf70b7adbd7c67b48ccea6c49e1c/sqlalchemy-2.0.50-cp314-cp314-macosx_11_0_arm64.whl @@ -666,7 +652,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/httpx-0.28.1-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/hyperframe-6.1.0-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/idna-3.15-pyhcf101f3_0.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-8.8.0-pyhcf101f3_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-9.0.0-pyhcf101f3_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipykernel-7.2.0-pyh6dadd2b_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython-9.13.0-pyhe2676ad_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython_pygments_lexers-1.1.1-pyhd8ed1ab_0.conda @@ -792,14 +778,12 @@ environments: - pypi: https://files.pythonhosted.org/packages/0b/f9/f15c95d6b200167cb22c5eca5eecfa9d28a8ee3f74095f1cd2345c71f2f9/pydoclint-0.8.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/0c/53/b50773ecf1d1e4a5858ee13011e30317ba02639ae4a1411a34967951fc9b/crysfml-0.6.2-cp314-cp314-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/0d/12/bbce9472f489cb5c4c23b0d13e5c59c37c1aab11b7ac637dfe6bbdccebe7/copier-9.15.1-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/0d/1f/d398de1612f7a611e22d743280339c9af4903675635e41be3370091c704b/arviz_stats-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/0d/fe/6bea5c9162869c5beba5d9c8abbed835ec85bf1ec1fba05a3822325c45f3/build-1.5.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/10/bd/c038d7cc38edc1aa5bf91ab8068b63d4308c66c4c8bb3cbba7dfbc049f9c/pyparsing-3.3.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/10/cb/f2ad4230dc2eb1a74edf38f1a38b9b52277f75bef262d8908e60d957e13c/blinker-1.9.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/11/8c/c9138d881c79aa0ea9ed83cbd58d5ca75624378b38cee225dcf5c42cc91f/griffelib-2.0.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/12/c9/6869a1dcf4aaf309b9543ec070be3ec3adebee7c9bec9af8c230494134b9/interrogate-1.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/13/95/cf3f7fe4910cf0365fa8ea0c731f4b8a624d97cd76ea777913ac8d0868e2/mkdocs_jupyter-0.26.3-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/19/d4/225027a913621a879b429a043674aa35220e6ce67785acad4f7bd0c4ff33/xarray_einstats-0.10.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/1c/59/964ecb8008722d27d8a835baea81f56a91cea8e097b3be992bc6ccde6367/versioningit-3.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/1d/77/928ea2e70641ca177a11140062cc5840d421795f2e82749d408d0cce900a/narwhals-2.21.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/1e/77/dc8c558f7593132cf8fefec57c4f60c83b16941c574ac5f619abb3ae7933/dill-0.4.1-py3-none-any.whl @@ -813,7 +797,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/2a/2d/d4bf65e47cea8ff2c794a600c4fd1273a7902f268757c531e0ee9f18aa58/pooch-1.9.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/01/bc663630c510822c95c47a66af9fa7a443c295b47d5f041e5e6ae62ef659/mkdocs_material-9.7.6-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/2f/c8/005d1de3af80f54411703d1263a0b9d31276411ec9f273d9432c59b17299/arviz_plots-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/b2/986d1220f6ee931e338d272bc1f3ec02cfe5f9b5fad84e95afdad57f1ebc/format_docstring-0.2.7-py3-none-any.whl @@ -840,8 +823,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/5f/fc/a7bf5b6e4e617b45f90f2d9d2a68519c249c81dd4fc2658c7a2a61c4f4b7/aiohappyeyeballs-2.6.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/61/2b/e260d50e64690d2a9e405d52ccd18a63c286c5088937dd0107cb23eb3195/diffpy_utils-3.7.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/61/d2/45c9defbaa1ea297035d9d4cce9e8f80daafbf19319c6007f157c6256ea9/propcache-0.5.2-cp314-cp314-win_amd64.whl - - pypi: https://files.pythonhosted.org/packages/64/a8/c09fbe44b12fa919c5bfe0afb71e60d1231a7dc93405e54c30496c57c9d3/arviz-1.1.0-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/69/d1/705e6c19b437a4105bf3b9ae7945fcfc3ad2abb73d14bae0a3f2d58b305b/arviz_base-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/6e/94/be70f8ee9c45f2f62b39a1f0e9303bc20e138a8f3b8e50ffd89498e177e1/mkdocstrings-1.0.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/6e/f1/abd09c2ae91228c5f3998dbd7f41353def9eac64253de3c8105efa2082f7/msgpack-1.1.2-cp314-cp314-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/72/b9/313e8f2f2e9517ae050a692ae7b3e4b3f17cc5e6dfea0db51fe14e586580/jinja2_ansible_filters-1.3.2-py3-none-any.whl @@ -862,7 +843,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/88/29/744136411e785c4b0b744d5413e56555265939ab3a104c6a4b719dad33fd/mkdocs_get_deps-0.2.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/88/39/799be3f2f0f38cc727ee3b4f1445fe6d5e4133064ec2e4115069418a5bb6/cloudpickle-3.1.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/88/b2/d0896bdcdc8d28a7fc5717c305f1a861c26e18c05047949fb371034d98bd/nodeenv-1.10.0-py2.py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/8a/a1/8d812e53a5da1687abb10445275d41a8b13adb781bbf7196ddbcf8d88505/lazy_loader-0.5-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8a/bd/e11a108317485075e68af9d23039619b86b28130c3b50d227d42edece64b/greenlet-3.5.1-cp314-cp314-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/8e/5a/7fd1b784a87e96e0078f49a0a13a98b4c5f644ba5597a4a3b70a2ba3e613/py3dmol-2.5.5-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8f/5e/f1e1dd319e35e962a4e00b33150a8868b6329cc1d19fd533436ba5488f09/uncertainties-3.2.3-py3-none-any.whl @@ -914,7 +894,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/d7/27/a58ddaf8c588a3ef080db9d0b7e0b97215cee3a45df74f3a94dbbf5c893a/pycodestyle-2.14.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/db/3c/33bac158f8ab7f89b2e59426d5fe2e4f63f7ed25df84c036890172b412b5/cfgv-3.5.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/db/d9/c495884c6e548fce18a8f40568ff120bc3a4b7b99813081c8ac0c936fa64/watchdog-6.0.0-py3-none-win_amd64.whl - - pypi: https://files.pythonhosted.org/packages/dc/83/6d810a8a9ebc9c307989b418840c20e46907c74d707beb67ab566773e6fc/xarray-2026.4.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/df/ac/46de6dda46478f7942f839e094970be2d4a861e005c4b3bf07c92e291a09/numpy-2.4.6-cp314-cp314-win_amd64.whl @@ -1030,7 +1009,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/httpx-0.28.1-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/hyperframe-6.1.0-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/idna-3.15-pyhcf101f3_0.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-8.8.0-pyhcf101f3_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-9.0.0-pyhcf101f3_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipykernel-7.2.0-pyha191276_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython-9.13.0-pyh53cf698_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython_pygments_lexers-1.1.1-pyhd8ed1ab_0.conda @@ -1111,7 +1090,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/04/11/432f32f8097b03e3cd5fe57e88efb685d964e2e5178a48ed61e841f7fdce/pyyaml_env_tag-1.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/0b/f9/f15c95d6b200167cb22c5eca5eecfa9d28a8ee3f74095f1cd2345c71f2f9/pydoclint-0.8.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/0d/12/bbce9472f489cb5c4c23b0d13e5c59c37c1aab11b7ac637dfe6bbdccebe7/copier-9.15.1-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/0d/1f/d398de1612f7a611e22d743280339c9af4903675635e41be3370091c704b/arviz_stats-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/0d/fe/6bea5c9162869c5beba5d9c8abbed835ec85bf1ec1fba05a3822325c45f3/build-1.5.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/10/bd/c038d7cc38edc1aa5bf91ab8068b63d4308c66c4c8bb3cbba7dfbc049f9c/pyparsing-3.3.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/10/cb/f2ad4230dc2eb1a74edf38f1a38b9b52277f75bef262d8908e60d957e13c/blinker-1.9.0-py3-none-any.whl @@ -1119,7 +1097,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/12/c9/6869a1dcf4aaf309b9543ec070be3ec3adebee7c9bec9af8c230494134b9/interrogate-1.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/13/95/cf3f7fe4910cf0365fa8ea0c731f4b8a624d97cd76ea777913ac8d0868e2/mkdocs_jupyter-0.26.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/14/15/5574111ae50dd6e879456888c0eadd4c5a869959775854e18e18a6b345f3/propcache-0.5.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - - pypi: https://files.pythonhosted.org/packages/19/d4/225027a913621a879b429a043674aa35220e6ce67785acad4f7bd0c4ff33/xarray_einstats-0.10.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/1c/59/964ecb8008722d27d8a835baea81f56a91cea8e097b3be992bc6ccde6367/versioningit-3.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/1d/77/928ea2e70641ca177a11140062cc5840d421795f2e82749d408d0cce900a/narwhals-2.21.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/1e/77/dc8c558f7593132cf8fefec57c4f60c83b16941c574ac5f619abb3ae7933/dill-0.4.1-py3-none-any.whl @@ -1133,7 +1110,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/2a/2d/d4bf65e47cea8ff2c794a600c4fd1273a7902f268757c531e0ee9f18aa58/pooch-1.9.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/01/bc663630c510822c95c47a66af9fa7a443c295b47d5f041e5e6ae62ef659/mkdocs_material-9.7.6-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/2f/c8/005d1de3af80f54411703d1263a0b9d31276411ec9f273d9432c59b17299/arviz_plots-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/91/d024616abdba99e83120e07a20658976f6a343646710760c4a51df126029/matplotlib-3.10.9-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl @@ -1154,17 +1130,14 @@ environments: - pypi: https://files.pythonhosted.org/packages/52/59/0782e51887ac6b07ffd1570e0364cf901ebc36345fea669969d2084baebb/simple_websocket-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/57/d8/8d44036d7eb7b6a8ec4c5494ea0c8c8b94fbc0ed3991c1a7adf230df03bf/aiohttp-3.13.5-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - - pypi: https://files.pythonhosted.org/packages/57/da/6fbf010c8ebb347679d0d100b22fe9ba5e13fd04046c5df7280d2f0bf706/sqlalchemy-2.0.50.tar.gz - pypi: https://files.pythonhosted.org/packages/5b/29/74eeb4d3f3ae61ca096b018ad486b3b3c74b17bec09ab4edab721cbefec3/typeguard-4.5.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/5b/54/662a4743aa81d9582ee9339d4ffa3c8fd40a4965e033d77b9da9774d3960/mkdocs_material_extensions-1.3.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/5b/69/93b34728cc386efdde0c342f8c680b9187dea7beb7adaf6b58a0713be101/mpld3-0.5.12-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/5f/97/2aab507d3d00ca626e8e57c1eac6a79e4e5fbcc63eb99733ff55d1717f65/pydantic_core-2.46.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl - pypi: https://files.pythonhosted.org/packages/5f/fc/a7bf5b6e4e617b45f90f2d9d2a68519c249c81dd4fc2658c7a2a61c4f4b7/aiohappyeyeballs-2.6.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/61/2b/e260d50e64690d2a9e405d52ccd18a63c286c5088937dd0107cb23eb3195/diffpy_utils-3.7.2-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/64/a8/c09fbe44b12fa919c5bfe0afb71e60d1231a7dc93405e54c30496c57c9d3/arviz-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/65/92/a5100f7185a800a5d29f8d14041f61475b9de465ffcc0f3b9fba606e4505/msgpack-1.1.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/65/b6/09b01cdbc15224e2850365192d17b7bdebb8bdbd8780ed221fcdf0d9a515/pandas-3.0.3-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl - - pypi: https://files.pythonhosted.org/packages/69/d1/705e6c19b437a4105bf3b9ae7945fcfc3ad2abb73d14bae0a3f2d58b305b/arviz_base-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/69/ff/6699e7b71e60d3049eb2bdcbc95ee3f35707b2b0e48f32e9e63d3ce30c08/coverage-7.14.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl - pypi: https://files.pythonhosted.org/packages/6a/bd/d91c5e39f490a49df14320f4e8c80161cfcce09f1e2cde1edd16a551abb3/frozenlist-1.8.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl - pypi: https://files.pythonhosted.org/packages/6e/94/be70f8ee9c45f2f62b39a1f0e9303bc20e138a8f3b8e50ffd89498e177e1/mkdocstrings-1.0.4-py3-none-any.whl @@ -1185,7 +1158,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/88/29/744136411e785c4b0b744d5413e56555265939ab3a104c6a4b719dad33fd/mkdocs_get_deps-0.2.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/88/39/799be3f2f0f38cc727ee3b4f1445fe6d5e4133064ec2e4115069418a5bb6/cloudpickle-3.1.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/88/b2/d0896bdcdc8d28a7fc5717c305f1a861c26e18c05047949fb371034d98bd/nodeenv-1.10.0-py2.py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/8a/a1/8d812e53a5da1687abb10445275d41a8b13adb781bbf7196ddbcf8d88505/lazy_loader-0.5-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/8a/2b/514fce8a7df81cf5bad7ff7865de7ac0c5776a38cc043475c4703eb7fe8b/sqlalchemy-2.0.50-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/8e/5a/7fd1b784a87e96e0078f49a0a13a98b4c5f644ba5597a4a3b70a2ba3e613/py3dmol-2.5.5-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8f/5e/f1e1dd319e35e962a4e00b33150a8868b6329cc1d19fd533436ba5488f09/uncertainties-3.2.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/90/96/04b8e52da071d28f5e21a805b19cb9390aa17a47462ac87f5e2696b9566d/paginate-0.5.7-py2.py3-none-any.whl @@ -1237,7 +1210,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/d5/08/c2409cb01d5368dcfedcbaffa7d044cc8957d57a9d0855244a5eb4709d30/funcy-2.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/d7/27/a58ddaf8c588a3ef080db9d0b7e0b97215cee3a45df74f3a94dbbf5c893a/pycodestyle-2.14.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/db/3c/33bac158f8ab7f89b2e59426d5fe2e4f63f7ed25df84c036890172b412b5/cfgv-3.5.0-py2.py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/dc/83/6d810a8a9ebc9c307989b418840c20e46907c74d707beb67ab566773e6fc/xarray-2026.4.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl @@ -1289,7 +1261,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/httpx-0.28.1-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/hyperframe-6.1.0-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/idna-3.15-pyhcf101f3_0.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-8.8.0-pyhcf101f3_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-9.0.0-pyhcf101f3_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipykernel-7.2.0-pyh5552912_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython-9.13.0-pyh53cf698_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython_pygments_lexers-1.1.1-pyhd8ed1ab_0.conda @@ -1422,7 +1394,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/08/ef/b3c6b9b5be2f82416d73fe2ed2e96e2793cd80e7510bd6a17ca79cdd88ec/fonttools-4.63.0-cp312-cp312-macosx_10_13_universal2.whl - pypi: https://files.pythonhosted.org/packages/0b/f9/f15c95d6b200167cb22c5eca5eecfa9d28a8ee3f74095f1cd2345c71f2f9/pydoclint-0.8.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/0d/12/bbce9472f489cb5c4c23b0d13e5c59c37c1aab11b7ac637dfe6bbdccebe7/copier-9.15.1-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/0d/1f/d398de1612f7a611e22d743280339c9af4903675635e41be3370091c704b/arviz_stats-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/0d/fe/6bea5c9162869c5beba5d9c8abbed835ec85bf1ec1fba05a3822325c45f3/build-1.5.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/10/bd/c038d7cc38edc1aa5bf91ab8068b63d4308c66c4c8bb3cbba7dfbc049f9c/pyparsing-3.3.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/10/cb/f2ad4230dc2eb1a74edf38f1a38b9b52277f75bef262d8908e60d957e13c/blinker-1.9.0-py3-none-any.whl @@ -1431,7 +1402,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/13/95/cf3f7fe4910cf0365fa8ea0c731f4b8a624d97cd76ea777913ac8d0868e2/mkdocs_jupyter-0.26.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/15/1d/9f9e30d76300b0150afaa8b37fab9a0194d44fd4f6b1e5038aca4a1440ed/crysfml-0.6.2-cp312-cp312-macosx_14_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/19/95/6195171e385007300f0f5574592e467c568becce2d937a0b6804f218bc49/pydantic_core-2.46.4-cp312-cp312-macosx_11_0_arm64.whl - - pypi: https://files.pythonhosted.org/packages/19/d4/225027a913621a879b429a043674aa35220e6ce67785acad4f7bd0c4ff33/xarray_einstats-0.10.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/1c/59/964ecb8008722d27d8a835baea81f56a91cea8e097b3be992bc6ccde6367/versioningit-3.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/1d/77/928ea2e70641ca177a11140062cc5840d421795f2e82749d408d0cce900a/narwhals-2.21.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/1e/77/dc8c558f7593132cf8fefec57c4f60c83b16941c574ac5f619abb3ae7933/dill-0.4.1-py3-none-any.whl @@ -1448,7 +1418,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/2c/01/bc663630c510822c95c47a66af9fa7a443c295b47d5f041e5e6ae62ef659/mkdocs_material-9.7.6-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/7d/49777a3e20b55863d4794384a38acd460c04157b0a00f8602b0d508b8431/propcache-0.5.2-cp312-cp312-macosx_11_0_arm64.whl - - pypi: https://files.pythonhosted.org/packages/2f/c8/005d1de3af80f54411703d1263a0b9d31276411ec9f273d9432c59b17299/arviz_plots-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/a0/37fb236da6040e337381dd656cafb97d09eacb998c5db3057547f5ffddd9/pycifrw-5.0.1-cp312-cp312-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl @@ -1475,8 +1444,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/5b/7e/8f322f5e600812e6f9a31b75d242631068ca8f4ef0582dd3ae6e72daecc8/watchdog-6.0.0-cp312-cp312-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/5f/fc/a7bf5b6e4e617b45f90f2d9d2a68519c249c81dd4fc2658c7a2a61c4f4b7/aiohappyeyeballs-2.6.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/61/2b/e260d50e64690d2a9e405d52ccd18a63c286c5088937dd0107cb23eb3195/diffpy_utils-3.7.2-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/64/a8/c09fbe44b12fa919c5bfe0afb71e60d1231a7dc93405e54c30496c57c9d3/arviz-1.1.0-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/69/d1/705e6c19b437a4105bf3b9ae7945fcfc3ad2abb73d14bae0a3f2d58b305b/arviz_base-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/6e/94/be70f8ee9c45f2f62b39a1f0e9303bc20e138a8f3b8e50ffd89498e177e1/mkdocstrings-1.0.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/72/b9/313e8f2f2e9517ae050a692ae7b3e4b3f17cc5e6dfea0db51fe14e586580/jinja2_ansible_filters-1.3.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/72/dc/0decaf5da92a7a969374474025787102d811d42aed1d32191fa338620e15/python_socketio-5.16.2-py3-none-any.whl @@ -1493,7 +1460,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/88/29/744136411e785c4b0b744d5413e56555265939ab3a104c6a4b719dad33fd/mkdocs_get_deps-0.2.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/88/39/799be3f2f0f38cc727ee3b4f1445fe6d5e4133064ec2e4115069418a5bb6/cloudpickle-3.1.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/88/b2/d0896bdcdc8d28a7fc5717c305f1a861c26e18c05047949fb371034d98bd/nodeenv-1.10.0-py2.py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/8a/a1/8d812e53a5da1687abb10445275d41a8b13adb781bbf7196ddbcf8d88505/lazy_loader-0.5-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8e/5a/7fd1b784a87e96e0078f49a0a13a98b4c5f644ba5597a4a3b70a2ba3e613/py3dmol-2.5.5-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8f/5e/f1e1dd319e35e962a4e00b33150a8868b6329cc1d19fd533436ba5488f09/uncertainties-3.2.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/90/96/04b8e52da071d28f5e21a805b19cb9390aa17a47462ac87f5e2696b9566d/paginate-0.5.7-py2.py3-none-any.whl @@ -1549,7 +1515,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/d7/27/a58ddaf8c588a3ef080db9d0b7e0b97215cee3a45df74f3a94dbbf5c893a/pycodestyle-2.14.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/d8/95/0a351b9289c2b5cbde0bacd4a83ebc44023e835490a727b2a3bd60ddc0f4/pillow-12.2.0-cp312-cp312-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/db/3c/33bac158f8ab7f89b2e59426d5fe2e4f63f7ed25df84c036890172b412b5/cfgv-3.5.0-py2.py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/dc/83/6d810a8a9ebc9c307989b418840c20e46907c74d707beb67ab566773e6fc/xarray-2026.4.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl @@ -1598,7 +1563,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/httpx-0.28.1-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/hyperframe-6.1.0-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/idna-3.15-pyhcf101f3_0.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-8.8.0-pyhcf101f3_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-9.0.0-pyhcf101f3_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipykernel-7.2.0-pyh6dadd2b_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython-9.13.0-pyhe2676ad_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython_pygments_lexers-1.1.1-pyhd8ed1ab_0.conda @@ -1725,14 +1690,12 @@ environments: - pypi: https://files.pythonhosted.org/packages/04/a1/4571fc46e7702de8d0c2dc54ad1b2f8e29328dea3ee90831181f7353d93c/matplotlib-3.10.9-cp312-cp312-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/0b/f9/f15c95d6b200167cb22c5eca5eecfa9d28a8ee3f74095f1cd2345c71f2f9/pydoclint-0.8.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/0d/12/bbce9472f489cb5c4c23b0d13e5c59c37c1aab11b7ac637dfe6bbdccebe7/copier-9.15.1-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/0d/1f/d398de1612f7a611e22d743280339c9af4903675635e41be3370091c704b/arviz_stats-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/0d/fe/6bea5c9162869c5beba5d9c8abbed835ec85bf1ec1fba05a3822325c45f3/build-1.5.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/10/bd/c038d7cc38edc1aa5bf91ab8068b63d4308c66c4c8bb3cbba7dfbc049f9c/pyparsing-3.3.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/10/cb/f2ad4230dc2eb1a74edf38f1a38b9b52277f75bef262d8908e60d957e13c/blinker-1.9.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/11/8c/c9138d881c79aa0ea9ed83cbd58d5ca75624378b38cee225dcf5c42cc91f/griffelib-2.0.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/12/c9/6869a1dcf4aaf309b9543ec070be3ec3adebee7c9bec9af8c230494134b9/interrogate-1.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/13/95/cf3f7fe4910cf0365fa8ea0c731f4b8a624d97cd76ea777913ac8d0868e2/mkdocs_jupyter-0.26.3-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/19/d4/225027a913621a879b429a043674aa35220e6ce67785acad4f7bd0c4ff33/xarray_einstats-0.10.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/19/e8/6026ed58a64563186a9ee3f29f41261fd1828f527dd93d33b60feca63352/contourpy-1.3.3-cp312-cp312-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/1a/c7/78200c18404ded028758b28b588aa1f4f3acd851271a74156a2a3db9eadf/crysfml-0.6.2-cp312-cp312-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/1c/59/964ecb8008722d27d8a835baea81f56a91cea8e097b3be992bc6ccde6367/versioningit-3.3.0-py3-none-any.whl @@ -1748,7 +1711,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/2a/2d/d4bf65e47cea8ff2c794a600c4fd1273a7902f268757c531e0ee9f18aa58/pooch-1.9.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/01/bc663630c510822c95c47a66af9fa7a443c295b47d5f041e5e6ae62ef659/mkdocs_material-9.7.6-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/2f/c8/005d1de3af80f54411703d1263a0b9d31276411ec9f273d9432c59b17299/arviz_plots-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/30/f5/310d104ddf41eb5a70f4c268d22508dfb0c3c8e86fec152be34d0d2ed819/greenlet-3.5.1-cp312-cp312-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/60/fca69c534602a7ced04280c952a246ad1edde2a6ca3a164f65d32ac41fe7/chardet-7.4.3-cp312-cp312-win_amd64.whl @@ -1774,8 +1736,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/5f/fc/a7bf5b6e4e617b45f90f2d9d2a68519c249c81dd4fc2658c7a2a61c4f4b7/aiohappyeyeballs-2.6.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/61/2b/e260d50e64690d2a9e405d52ccd18a63c286c5088937dd0107cb23eb3195/diffpy_utils-3.7.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/61/7c/5c0d34aa3024694d6dcb9271cdbdd08c4e47c1c0ad95ec7e7bc74cdea145/propcache-0.5.2-cp312-cp312-win_amd64.whl - - pypi: https://files.pythonhosted.org/packages/64/a8/c09fbe44b12fa919c5bfe0afb71e60d1231a7dc93405e54c30496c57c9d3/arviz-1.1.0-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/69/d1/705e6c19b437a4105bf3b9ae7945fcfc3ad2abb73d14bae0a3f2d58b305b/arviz_base-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/6e/94/be70f8ee9c45f2f62b39a1f0e9303bc20e138a8f3b8e50ffd89498e177e1/mkdocstrings-1.0.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/6f/0c/8297c8d978c919ad6318011631a6123082d5da940da5f8612e75a247d739/diffpy_pdffit2-1.6.0-cp312-cp312-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/72/b9/313e8f2f2e9517ae050a692ae7b3e4b3f17cc5e6dfea0db51fe14e586580/jinja2_ansible_filters-1.3.2-py3-none-any.whl @@ -1796,7 +1756,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/88/29/744136411e785c4b0b744d5413e56555265939ab3a104c6a4b719dad33fd/mkdocs_get_deps-0.2.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/88/39/799be3f2f0f38cc727ee3b4f1445fe6d5e4133064ec2e4115069418a5bb6/cloudpickle-3.1.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/88/b2/d0896bdcdc8d28a7fc5717c305f1a861c26e18c05047949fb371034d98bd/nodeenv-1.10.0-py2.py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/8a/a1/8d812e53a5da1687abb10445275d41a8b13adb781bbf7196ddbcf8d88505/lazy_loader-0.5-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8c/ec/d431eb7941fb55a31dd6ca3404d41fbb52d99172df2e7707754488390910/msgpack-1.1.2-cp312-cp312-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/8e/5a/7fd1b784a87e96e0078f49a0a13a98b4c5f644ba5597a4a3b70a2ba3e613/py3dmol-2.5.5-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8f/5e/f1e1dd319e35e962a4e00b33150a8868b6329cc1d19fd533436ba5488f09/uncertainties-3.2.3-py3-none-any.whl @@ -1851,7 +1810,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/d7/27/a58ddaf8c588a3ef080db9d0b7e0b97215cee3a45df74f3a94dbbf5c893a/pycodestyle-2.14.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/db/3c/33bac158f8ab7f89b2e59426d5fe2e4f63f7ed25df84c036890172b412b5/cfgv-3.5.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/db/d9/c495884c6e548fce18a8f40568ff120bc3a4b7b99813081c8ac0c936fa64/watchdog-6.0.0-py3-none-win_amd64.whl - - pypi: https://files.pythonhosted.org/packages/dc/83/6d810a8a9ebc9c307989b418840c20e46907c74d707beb67ab566773e6fc/xarray-2026.4.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl @@ -1961,7 +1919,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/httpx-0.28.1-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/hyperframe-6.1.0-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/idna-3.15-pyhcf101f3_0.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-8.8.0-pyhcf101f3_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-9.0.0-pyhcf101f3_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipykernel-7.2.0-pyha191276_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython-9.13.0-pyh53cf698_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython_pygments_lexers-1.1.1-pyhd8ed1ab_0.conda @@ -2043,7 +2001,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/0b/2e/7eea398450457ecb54e18e9d10110993fa65561c4f3add5e8eccd2b9cd41/scipy-1.17.1-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/0b/f9/f15c95d6b200167cb22c5eca5eecfa9d28a8ee3f74095f1cd2345c71f2f9/pydoclint-0.8.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/0d/12/bbce9472f489cb5c4c23b0d13e5c59c37c1aab11b7ac637dfe6bbdccebe7/copier-9.15.1-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/0d/1f/d398de1612f7a611e22d743280339c9af4903675635e41be3370091c704b/arviz_stats-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/0d/fe/6bea5c9162869c5beba5d9c8abbed835ec85bf1ec1fba05a3822325c45f3/build-1.5.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/10/bd/c038d7cc38edc1aa5bf91ab8068b63d4308c66c4c8bb3cbba7dfbc049f9c/pyparsing-3.3.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/10/cb/f2ad4230dc2eb1a74edf38f1a38b9b52277f75bef262d8908e60d957e13c/blinker-1.9.0-py3-none-any.whl @@ -2051,7 +2008,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/11/8d/d2532ad2a603ca2b93ad9f5135732124e57811d0168155852f37fbce2458/pillow-12.2.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/12/c9/6869a1dcf4aaf309b9543ec070be3ec3adebee7c9bec9af8c230494134b9/interrogate-1.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/13/95/cf3f7fe4910cf0365fa8ea0c731f4b8a624d97cd76ea777913ac8d0868e2/mkdocs_jupyter-0.26.3-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/19/d4/225027a913621a879b429a043674aa35220e6ce67785acad4f7bd0c4ff33/xarray_einstats-0.10.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/1c/59/964ecb8008722d27d8a835baea81f56a91cea8e097b3be992bc6ccde6367/versioningit-3.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/1d/77/928ea2e70641ca177a11140062cc5840d421795f2e82749d408d0cce900a/narwhals-2.21.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/1e/77/dc8c558f7593132cf8fefec57c4f60c83b16941c574ac5f619abb3ae7933/dill-0.4.1-py3-none-any.whl @@ -2067,7 +2023,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2c/01/bc663630c510822c95c47a66af9fa7a443c295b47d5f041e5e6ae62ef659/mkdocs_material-9.7.6-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2f/40/dc34d1a8d5f1e51fc64640b62b191684da52ca469da9cd74e84936ffa4a6/msgpack-1.1.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - - pypi: https://files.pythonhosted.org/packages/2f/c8/005d1de3af80f54411703d1263a0b9d31276411ec9f273d9432c59b17299/arviz_plots-1.1.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/31/26/ef168b184a25701f9995e8fb7e503fafd7a99c1c77cda1bc1a26ea2ed486/sqlalchemy-2.0.50-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/5d/f7e914f7d9325abff4057cee62c0fa70263683189f774473cbfb534cd13b/matplotlib-3.10.9-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl @@ -2087,7 +2043,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/46/b4/0887c88ddfaba1d7140ea335144eb904af97550786ee58bdb295ff10d255/crysfml-0.6.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl - pypi: https://files.pythonhosted.org/packages/52/59/0782e51887ac6b07ffd1570e0364cf901ebc36345fea669969d2084baebb/simple_websocket-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/57/da/6fbf010c8ebb347679d0d100b22fe9ba5e13fd04046c5df7280d2f0bf706/sqlalchemy-2.0.50.tar.gz - pypi: https://files.pythonhosted.org/packages/58/3b/1cdec6772bdbaf7b25dab360c59f03cadf05492dd724c6540af905389b07/pandas-3.0.3-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/58/e0/f1871f520c359e4e3a2eb7437c9e7e792bb6c356414e8617937561167caf/pycifrw-5.0.1.tar.gz - pypi: https://files.pythonhosted.org/packages/5b/29/74eeb4d3f3ae61ca096b018ad486b3b3c74b17bec09ab4edab721cbefec3/typeguard-4.5.2-py3-none-any.whl @@ -2095,8 +2050,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/5b/69/93b34728cc386efdde0c342f8c680b9187dea7beb7adaf6b58a0713be101/mpld3-0.5.12-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/5f/fc/a7bf5b6e4e617b45f90f2d9d2a68519c249c81dd4fc2658c7a2a61c4f4b7/aiohappyeyeballs-2.6.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/61/2b/e260d50e64690d2a9e405d52ccd18a63c286c5088937dd0107cb23eb3195/diffpy_utils-3.7.2-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/64/a8/c09fbe44b12fa919c5bfe0afb71e60d1231a7dc93405e54c30496c57c9d3/arviz-1.1.0-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/69/d1/705e6c19b437a4105bf3b9ae7945fcfc3ad2abb73d14bae0a3f2d58b305b/arviz_base-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/6e/94/be70f8ee9c45f2f62b39a1f0e9303bc20e138a8f3b8e50ffd89498e177e1/mkdocstrings-1.0.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/72/b9/313e8f2f2e9517ae050a692ae7b3e4b3f17cc5e6dfea0db51fe14e586580/jinja2_ansible_filters-1.3.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/72/dc/0decaf5da92a7a969374474025787102d811d42aed1d32191fa338620e15/python_socketio-5.16.2-py3-none-any.whl @@ -2115,7 +2068,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/88/39/799be3f2f0f38cc727ee3b4f1445fe6d5e4133064ec2e4115069418a5bb6/cloudpickle-3.1.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/88/b2/d0896bdcdc8d28a7fc5717c305f1a861c26e18c05047949fb371034d98bd/nodeenv-1.10.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/89/1d/8eff589b45bb8190a9d12c49cfad0f176a5cbd1534908a6b5125e2886239/pydantic_core-2.46.4-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl - - pypi: https://files.pythonhosted.org/packages/8a/a1/8d812e53a5da1687abb10445275d41a8b13adb781bbf7196ddbcf8d88505/lazy_loader-0.5-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8e/5a/7fd1b784a87e96e0078f49a0a13a98b4c5f644ba5597a4a3b70a2ba3e613/py3dmol-2.5.5-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8f/5e/f1e1dd319e35e962a4e00b33150a8868b6329cc1d19fd533436ba5488f09/uncertainties-3.2.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/90/96/04b8e52da071d28f5e21a805b19cb9390aa17a47462ac87f5e2696b9566d/paginate-0.5.7-py2.py3-none-any.whl @@ -2166,7 +2118,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/d7/27/a58ddaf8c588a3ef080db9d0b7e0b97215cee3a45df74f3a94dbbf5c893a/pycodestyle-2.14.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/d7/51/ec641c26e6dca1b25a7d2035ba6ecb7c884ef1a100a9e42fbe4ce4405139/coverage-7.14.0-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl - pypi: https://files.pythonhosted.org/packages/db/3c/33bac158f8ab7f89b2e59426d5fe2e4f63f7ed25df84c036890172b412b5/cfgv-3.5.0-py2.py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/dc/83/6d810a8a9ebc9c307989b418840c20e46907c74d707beb67ab566773e6fc/xarray-2026.4.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl @@ -2221,7 +2172,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/httpx-0.28.1-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/hyperframe-6.1.0-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/idna-3.15-pyhcf101f3_0.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-8.8.0-pyhcf101f3_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-9.0.0-pyhcf101f3_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipykernel-7.2.0-pyh5552912_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython-9.13.0-pyh53cf698_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython_pygments_lexers-1.1.1-pyhd8ed1ab_0.conda @@ -2354,14 +2305,12 @@ environments: - pypi: https://files.pythonhosted.org/packages/06/41/4e70dea1d0311016c0b0b1c53a24a266f9f8a34c6bc1af0f17cfca20aa1d/gemmi-0.7.5-cp314-cp314-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/0b/f9/f15c95d6b200167cb22c5eca5eecfa9d28a8ee3f74095f1cd2345c71f2f9/pydoclint-0.8.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/0d/12/bbce9472f489cb5c4c23b0d13e5c59c37c1aab11b7ac637dfe6bbdccebe7/copier-9.15.1-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/0d/1f/d398de1612f7a611e22d743280339c9af4903675635e41be3370091c704b/arviz_stats-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/0d/fe/6bea5c9162869c5beba5d9c8abbed835ec85bf1ec1fba05a3822325c45f3/build-1.5.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/10/bd/c038d7cc38edc1aa5bf91ab8068b63d4308c66c4c8bb3cbba7dfbc049f9c/pyparsing-3.3.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/10/cb/f2ad4230dc2eb1a74edf38f1a38b9b52277f75bef262d8908e60d957e13c/blinker-1.9.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/11/8c/c9138d881c79aa0ea9ed83cbd58d5ca75624378b38cee225dcf5c42cc91f/griffelib-2.0.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/12/c9/6869a1dcf4aaf309b9543ec070be3ec3adebee7c9bec9af8c230494134b9/interrogate-1.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/13/95/cf3f7fe4910cf0365fa8ea0c731f4b8a624d97cd76ea777913ac8d0868e2/mkdocs_jupyter-0.26.3-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/19/d4/225027a913621a879b429a043674aa35220e6ce67785acad4f7bd0c4ff33/xarray_einstats-0.10.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/1b/9f/38ff9e57a2eade7bf9dfee5eae17f39fc0e998658050279cbb14d97d36d9/msgpack-1.1.2-cp314-cp314-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/1c/59/964ecb8008722d27d8a835baea81f56a91cea8e097b3be992bc6ccde6367/versioningit-3.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/1d/77/928ea2e70641ca177a11140062cc5840d421795f2e82749d408d0cce900a/narwhals-2.21.2-py3-none-any.whl @@ -2377,7 +2326,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/2a/2d/d4bf65e47cea8ff2c794a600c4fd1273a7902f268757c531e0ee9f18aa58/pooch-1.9.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/01/bc663630c510822c95c47a66af9fa7a443c295b47d5f041e5e6ae62ef659/mkdocs_material-9.7.6-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/2f/c8/005d1de3af80f54411703d1263a0b9d31276411ec9f273d9432c59b17299/arviz_plots-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/b2/986d1220f6ee931e338d272bc1f3ec02cfe5f9b5fad84e95afdad57f1ebc/format_docstring-0.2.7-py3-none-any.whl @@ -2405,9 +2353,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/60/61/23f27c172f022e04025b7dc2367f4d63c1a398120607ec896228649a6f48/numpy-2.4.6-cp314-cp314-macosx_14_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/61/2b/e260d50e64690d2a9e405d52ccd18a63c286c5088937dd0107cb23eb3195/diffpy_utils-3.7.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/63/b1/4260d67d6bd85e58a66b72d54ce15d5de789b6f3870cc6bedf8ff9667401/propcache-0.5.2-cp314-cp314-macosx_11_0_arm64.whl - - pypi: https://files.pythonhosted.org/packages/64/a8/c09fbe44b12fa919c5bfe0afb71e60d1231a7dc93405e54c30496c57c9d3/arviz-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/68/10/bf2d6738d72748b961a3751ab89522d58c54efc36a8e1a12161216cd45cf/pandas-3.0.3-cp314-cp314-macosx_11_0_arm64.whl - - pypi: https://files.pythonhosted.org/packages/69/d1/705e6c19b437a4105bf3b9ae7945fcfc3ad2abb73d14bae0a3f2d58b305b/arviz_base-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/6a/b7/9366ed44ced9b7ef357ab48c94205280276db9d7f064aa3012a97227e966/h5py-3.16.0-cp314-cp314-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/6e/94/be70f8ee9c45f2f62b39a1f0e9303bc20e138a8f3b8e50ffd89498e177e1/mkdocstrings-1.0.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/72/b9/313e8f2f2e9517ae050a692ae7b3e4b3f17cc5e6dfea0db51fe14e586580/jinja2_ansible_filters-1.3.2-py3-none-any.whl @@ -2425,7 +2371,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/88/29/744136411e785c4b0b744d5413e56555265939ab3a104c6a4b719dad33fd/mkdocs_get_deps-0.2.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/88/39/799be3f2f0f38cc727ee3b4f1445fe6d5e4133064ec2e4115069418a5bb6/cloudpickle-3.1.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/88/b2/d0896bdcdc8d28a7fc5717c305f1a861c26e18c05047949fb371034d98bd/nodeenv-1.10.0-py2.py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/8a/a1/8d812e53a5da1687abb10445275d41a8b13adb781bbf7196ddbcf8d88505/lazy_loader-0.5-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8e/5a/7fd1b784a87e96e0078f49a0a13a98b4c5f644ba5597a4a3b70a2ba3e613/py3dmol-2.5.5-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8f/5e/f1e1dd319e35e962a4e00b33150a8868b6329cc1d19fd533436ba5488f09/uncertainties-3.2.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/90/96/04b8e52da071d28f5e21a805b19cb9390aa17a47462ac87f5e2696b9566d/paginate-0.5.7-py2.py3-none-any.whl @@ -2477,7 +2422,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/db/3c/33bac158f8ab7f89b2e59426d5fe2e4f63f7ed25df84c036890172b412b5/cfgv-3.5.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/db/7b/8624a203326675d7746a254083a187398090a179335b2e4a20e2ddc46e83/scipy-1.17.1-cp314-cp314-macosx_14_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/db/7d/7f3d619e951c88ed75c6037b246ddcf2d322812ee8ea189be89511721d54/watchdog-6.0.0.tar.gz - - pypi: https://files.pythonhosted.org/packages/dc/83/6d810a8a9ebc9c307989b418840c20e46907c74d707beb67ab566773e6fc/xarray-2026.4.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/df/32/10ac51b4be7cdecd7e93d069251c86dfbf70b7adbd7c67b48ccea6c49e1c/sqlalchemy-2.0.50-cp314-cp314-macosx_11_0_arm64.whl @@ -2531,7 +2475,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/httpx-0.28.1-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/hyperframe-6.1.0-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/idna-3.15-pyhcf101f3_0.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-8.8.0-pyhcf101f3_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-9.0.0-pyhcf101f3_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipykernel-7.2.0-pyh6dadd2b_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython-9.13.0-pyhe2676ad_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython_pygments_lexers-1.1.1-pyhd8ed1ab_0.conda @@ -2657,14 +2601,12 @@ environments: - pypi: https://files.pythonhosted.org/packages/0b/f9/f15c95d6b200167cb22c5eca5eecfa9d28a8ee3f74095f1cd2345c71f2f9/pydoclint-0.8.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/0c/53/b50773ecf1d1e4a5858ee13011e30317ba02639ae4a1411a34967951fc9b/crysfml-0.6.2-cp314-cp314-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/0d/12/bbce9472f489cb5c4c23b0d13e5c59c37c1aab11b7ac637dfe6bbdccebe7/copier-9.15.1-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/0d/1f/d398de1612f7a611e22d743280339c9af4903675635e41be3370091c704b/arviz_stats-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/0d/fe/6bea5c9162869c5beba5d9c8abbed835ec85bf1ec1fba05a3822325c45f3/build-1.5.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/10/bd/c038d7cc38edc1aa5bf91ab8068b63d4308c66c4c8bb3cbba7dfbc049f9c/pyparsing-3.3.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/10/cb/f2ad4230dc2eb1a74edf38f1a38b9b52277f75bef262d8908e60d957e13c/blinker-1.9.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/11/8c/c9138d881c79aa0ea9ed83cbd58d5ca75624378b38cee225dcf5c42cc91f/griffelib-2.0.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/12/c9/6869a1dcf4aaf309b9543ec070be3ec3adebee7c9bec9af8c230494134b9/interrogate-1.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/13/95/cf3f7fe4910cf0365fa8ea0c731f4b8a624d97cd76ea777913ac8d0868e2/mkdocs_jupyter-0.26.3-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/19/d4/225027a913621a879b429a043674aa35220e6ce67785acad4f7bd0c4ff33/xarray_einstats-0.10.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/1c/59/964ecb8008722d27d8a835baea81f56a91cea8e097b3be992bc6ccde6367/versioningit-3.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/1d/77/928ea2e70641ca177a11140062cc5840d421795f2e82749d408d0cce900a/narwhals-2.21.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/1e/77/dc8c558f7593132cf8fefec57c4f60c83b16941c574ac5f619abb3ae7933/dill-0.4.1-py3-none-any.whl @@ -2678,7 +2620,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/2a/2d/d4bf65e47cea8ff2c794a600c4fd1273a7902f268757c531e0ee9f18aa58/pooch-1.9.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/01/bc663630c510822c95c47a66af9fa7a443c295b47d5f041e5e6ae62ef659/mkdocs_material-9.7.6-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/2f/c8/005d1de3af80f54411703d1263a0b9d31276411ec9f273d9432c59b17299/arviz_plots-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/b2/986d1220f6ee931e338d272bc1f3ec02cfe5f9b5fad84e95afdad57f1ebc/format_docstring-0.2.7-py3-none-any.whl @@ -2705,8 +2646,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/5f/fc/a7bf5b6e4e617b45f90f2d9d2a68519c249c81dd4fc2658c7a2a61c4f4b7/aiohappyeyeballs-2.6.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/61/2b/e260d50e64690d2a9e405d52ccd18a63c286c5088937dd0107cb23eb3195/diffpy_utils-3.7.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/61/d2/45c9defbaa1ea297035d9d4cce9e8f80daafbf19319c6007f157c6256ea9/propcache-0.5.2-cp314-cp314-win_amd64.whl - - pypi: https://files.pythonhosted.org/packages/64/a8/c09fbe44b12fa919c5bfe0afb71e60d1231a7dc93405e54c30496c57c9d3/arviz-1.1.0-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/69/d1/705e6c19b437a4105bf3b9ae7945fcfc3ad2abb73d14bae0a3f2d58b305b/arviz_base-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/6e/94/be70f8ee9c45f2f62b39a1f0e9303bc20e138a8f3b8e50ffd89498e177e1/mkdocstrings-1.0.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/6e/f1/abd09c2ae91228c5f3998dbd7f41353def9eac64253de3c8105efa2082f7/msgpack-1.1.2-cp314-cp314-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/72/b9/313e8f2f2e9517ae050a692ae7b3e4b3f17cc5e6dfea0db51fe14e586580/jinja2_ansible_filters-1.3.2-py3-none-any.whl @@ -2727,7 +2666,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/88/29/744136411e785c4b0b744d5413e56555265939ab3a104c6a4b719dad33fd/mkdocs_get_deps-0.2.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/88/39/799be3f2f0f38cc727ee3b4f1445fe6d5e4133064ec2e4115069418a5bb6/cloudpickle-3.1.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/88/b2/d0896bdcdc8d28a7fc5717c305f1a861c26e18c05047949fb371034d98bd/nodeenv-1.10.0-py2.py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/8a/a1/8d812e53a5da1687abb10445275d41a8b13adb781bbf7196ddbcf8d88505/lazy_loader-0.5-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8a/bd/e11a108317485075e68af9d23039619b86b28130c3b50d227d42edece64b/greenlet-3.5.1-cp314-cp314-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/8e/5a/7fd1b784a87e96e0078f49a0a13a98b4c5f644ba5597a4a3b70a2ba3e613/py3dmol-2.5.5-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8f/5e/f1e1dd319e35e962a4e00b33150a8868b6329cc1d19fd533436ba5488f09/uncertainties-3.2.3-py3-none-any.whl @@ -2779,7 +2717,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/d7/27/a58ddaf8c588a3ef080db9d0b7e0b97215cee3a45df74f3a94dbbf5c893a/pycodestyle-2.14.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/db/3c/33bac158f8ab7f89b2e59426d5fe2e4f63f7ed25df84c036890172b412b5/cfgv-3.5.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/db/d9/c495884c6e548fce18a8f40568ff120bc3a4b7b99813081c8ac0c936fa64/watchdog-6.0.0-py3-none-win_amd64.whl - - pypi: https://files.pythonhosted.org/packages/dc/83/6d810a8a9ebc9c307989b418840c20e46907c74d707beb67ab566773e6fc/xarray-2026.4.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/df/ac/46de6dda46478f7942f839e094970be2d4a861e005c4b3bf07c92e291a09/numpy-2.4.6-cp314-cp314-win_amd64.whl @@ -2877,7 +2814,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/httpx-0.28.1-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/hyperframe-6.1.0-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/idna-3.15-pyhcf101f3_0.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-8.8.0-pyhcf101f3_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-9.0.0-pyhcf101f3_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipykernel-7.2.0-pyha191276_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython-9.13.0-pyh53cf698_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython_pygments_lexers-1.1.1-pyhd8ed1ab_0.conda @@ -2967,6 +2904,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2b/76/7f1bfd6afff4c5e38e36a3c6d68eb5f4b7311ea80baf693db78d95b603c4/propcache-0.5.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/2f/40/dc34d1a8d5f1e51fc64640b62b191684da52ca469da9cd74e84936ffa4a6/msgpack-1.1.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/2f/c8/005d1de3af80f54411703d1263a0b9d31276411ec9f273d9432c59b17299/arviz_plots-1.1.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/31/26/ef168b184a25701f9995e8fb7e503fafd7a99c1c77cda1bc1a26ea2ed486/sqlalchemy-2.0.50-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/32/5d/f7e914f7d9325abff4057cee62c0fa70263683189f774473cbfb534cd13b/matplotlib-3.10.9-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/33/eb/f9f1ded8e4db9638f9530c3782eb01f5ab04945f4cb9e597a51c203fa4c5/diffpy_pdffit2-1.6.0.tar.gz - pypi: https://files.pythonhosted.org/packages/36/e1/a8933a72c45a87177fbde2696e0d0755c8c9062f8c077a961c6215fa27b1/fonttools-4.63.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl @@ -2978,7 +2916,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/46/b4/0887c88ddfaba1d7140ea335144eb904af97550786ee58bdb295ff10d255/crysfml-0.6.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl - pypi: https://files.pythonhosted.org/packages/52/59/0782e51887ac6b07ffd1570e0364cf901ebc36345fea669969d2084baebb/simple_websocket-1.1.0-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/57/da/6fbf010c8ebb347679d0d100b22fe9ba5e13fd04046c5df7280d2f0bf706/sqlalchemy-2.0.50.tar.gz - pypi: https://files.pythonhosted.org/packages/58/3b/1cdec6772bdbaf7b25dab360c59f03cadf05492dd724c6540af905389b07/pandas-3.0.3-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/58/e0/f1871f520c359e4e3a2eb7437c9e7e792bb6c356414e8617937561167caf/pycifrw-5.0.1.tar.gz - pypi: https://files.pythonhosted.org/packages/5b/29/74eeb4d3f3ae61ca096b018ad486b3b3c74b17bec09ab4edab721cbefec3/typeguard-4.5.2-py3-none-any.whl @@ -3059,7 +2996,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/httpx-0.28.1-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/hyperframe-6.1.0-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/idna-3.15-pyhcf101f3_0.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-8.8.0-pyhcf101f3_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-9.0.0-pyhcf101f3_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipykernel-7.2.0-pyh5552912_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython-9.13.0-pyh53cf698_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython_pygments_lexers-1.1.1-pyhd8ed1ab_0.conda @@ -3272,7 +3209,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/noarch/httpx-0.28.1-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/hyperframe-6.1.0-pyhd8ed1ab_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/idna-3.15-pyhcf101f3_0.conda - - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-8.8.0-pyhcf101f3_0.conda + - conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-9.0.0-pyhcf101f3_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipykernel-7.2.0-pyh6dadd2b_1.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython-9.13.0-pyhe2676ad_0.conda - conda: https://conda.anaconda.org/conda-forge/noarch/ipython_pygments_lexers-1.1.1-pyhd8ed1ab_0.conda @@ -4811,19 +4748,18 @@ packages: - pkg:pypi/idna?source=compressed-mapping size: 62642 timestamp: 1779294335905 -- conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-8.8.0-pyhcf101f3_0.conda - sha256: 82ab2a0d91ca1e7e63ab6a4939356667ef683905dea631bc2121aa534d347b16 - md5: 080594bf4493e6bae2607e65390c520a +- conda: https://conda.anaconda.org/conda-forge/noarch/importlib-metadata-9.0.0-pyhcf101f3_0.conda + sha256: 43e2a5497cad1598ff88a3e69f69bc88b7b8f141fa63c60eab5db296317318b8 + md5: ffc17e785d64e12fc311af9184221839 depends: - python >=3.10 - zipp >=3.20 - python license: Apache-2.0 - license_family: APACHE purls: - - pkg:pypi/importlib-metadata?source=hash-mapping - size: 34387 - timestamp: 1773931568510 + - pkg:pypi/importlib-metadata?source=compressed-mapping + size: 34766 + timestamp: 1779714582554 - conda: https://conda.anaconda.org/conda-forge/noarch/ipykernel-7.2.0-pyh5552912_1.conda sha256: 5c1f3e874adaf603449f2b135d48f168c5d510088c78c229bda0431268b43b27 md5: 4b53d436f3fbc02ce3eeaf8ae9bebe01 @@ -7787,7 +7723,6 @@ packages: - pypi: . name: easydiffraction requires_dist: - - arviz - asciichartpy - asteval - bumps @@ -8639,6 +8574,44 @@ packages: - psutil ; extra == 'test' - setuptools ; extra == 'test' requires_python: '>=3.10' +- pypi: https://files.pythonhosted.org/packages/31/26/ef168b184a25701f9995e8fb7e503fafd7a99c1c77cda1bc1a26ea2ed486/sqlalchemy-2.0.50-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl + name: sqlalchemy + version: 2.0.50 + sha256: 6c206aec519a2e7bd08abbfb33436e325fd22c632d9c21a9047e376ce241646e + requires_dist: + - importlib-metadata ; python_full_version < '3.8' + - greenlet>=1 ; platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64' + - typing-extensions>=4.6.0 + - greenlet>=1 ; extra == 'asyncio' + - mypy>=0.910 ; extra == 'mypy' + - pyodbc ; extra == 'mssql' + - pymssql ; extra == 'mssql-pymssql' + - pyodbc ; extra == 'mssql-pyodbc' + - mysqlclient>=1.4.0 ; extra == 'mysql' + - mysql-connector-python ; extra == 'mysql-connector' + - mariadb>=1.0.1,!=1.1.2,!=1.1.5,!=1.1.10 ; extra == 'mariadb-connector' + - cx-oracle>=8 ; extra == 'oracle' + - oracledb>=1.0.1 ; extra == 'oracle-oracledb' + - psycopg2>=2.7 ; extra == 'postgresql' + - pg8000>=1.29.1 ; extra == 'postgresql-pg8000' + - greenlet>=1 ; extra == 'postgresql-asyncpg' + - asyncpg ; extra == 'postgresql-asyncpg' + - psycopg2-binary ; extra == 'postgresql-psycopg2binary' + - psycopg2cffi ; extra == 'postgresql-psycopg2cffi' + - psycopg>=3.0.7 ; extra == 'postgresql-psycopg' + - psycopg[binary]>=3.0.7 ; extra == 'postgresql-psycopgbinary' + - pymysql ; extra == 'pymysql' + - greenlet>=1 ; extra == 'aiomysql' + - aiomysql>=0.2.0 ; extra == 'aiomysql' + - greenlet>=1 ; extra == 'aioodbc' + - aioodbc ; extra == 'aioodbc' + - greenlet>=1 ; extra == 'asyncmy' + - asyncmy>=0.2.3,!=0.2.4,!=0.2.6 ; extra == 'asyncmy' + - greenlet>=1 ; extra == 'aiosqlite' + - aiosqlite ; extra == 'aiosqlite' + - typing-extensions!=3.10.0.1 ; extra == 'aiosqlite' + - sqlcipher3-binary ; extra == 'sqlcipher' + requires_python: '>=3.7' - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl name: mkdocstrings-python version: 2.0.3 @@ -9199,44 +9172,6 @@ packages: - brotlicffi>=1.2 ; platform_python_implementation != 'CPython' and extra == 'speedups' - backports-zstd ; python_full_version < '3.14' and platform_python_implementation == 'CPython' and extra == 'speedups' requires_python: '>=3.9' -- pypi: https://files.pythonhosted.org/packages/57/da/6fbf010c8ebb347679d0d100b22fe9ba5e13fd04046c5df7280d2f0bf706/sqlalchemy-2.0.50.tar.gz - name: sqlalchemy - version: 2.0.50 - sha256: af5607d11ef90fd6a5c0549fe0045dce1663d427426bcfb506dcb5346a85a3b9 - requires_dist: - - importlib-metadata ; python_full_version < '3.8' - - greenlet>=1 ; platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64' - - typing-extensions>=4.6.0 - - greenlet>=1 ; extra == 'asyncio' - - mypy>=0.910 ; extra == 'mypy' - - pyodbc ; extra == 'mssql' - - pymssql ; extra == 'mssql-pymssql' - - pyodbc ; extra == 'mssql-pyodbc' - - mysqlclient>=1.4.0 ; extra == 'mysql' - - mysql-connector-python ; extra == 'mysql-connector' - - mariadb>=1.0.1,!=1.1.2,!=1.1.5,!=1.1.10 ; extra == 'mariadb-connector' - - cx-oracle>=8 ; extra == 'oracle' - - oracledb>=1.0.1 ; extra == 'oracle-oracledb' - - psycopg2>=2.7 ; extra == 'postgresql' - - pg8000>=1.29.1 ; extra == 'postgresql-pg8000' - - greenlet>=1 ; extra == 'postgresql-asyncpg' - - asyncpg ; extra == 'postgresql-asyncpg' - - psycopg2-binary ; extra == 'postgresql-psycopg2binary' - - psycopg2cffi ; extra == 'postgresql-psycopg2cffi' - - psycopg>=3.0.7 ; extra == 'postgresql-psycopg' - - psycopg[binary]>=3.0.7 ; extra == 'postgresql-psycopgbinary' - - pymysql ; extra == 'pymysql' - - greenlet>=1 ; extra == 'aiomysql' - - aiomysql>=0.2.0 ; extra == 'aiomysql' - - greenlet>=1 ; extra == 'aioodbc' - - aioodbc ; extra == 'aioodbc' - - greenlet>=1 ; extra == 'asyncmy' - - asyncmy>=0.2.3,!=0.2.4,!=0.2.6 ; extra == 'asyncmy' - - greenlet>=1 ; extra == 'aiosqlite' - - aiosqlite ; extra == 'aiosqlite' - - typing-extensions!=3.10.0.1 ; extra == 'aiosqlite' - - sqlcipher3-binary ; extra == 'sqlcipher' - requires_python: '>=3.7' - pypi: https://files.pythonhosted.org/packages/58/3b/1cdec6772bdbaf7b25dab360c59f03cadf05492dd724c6540af905389b07/pandas-3.0.3-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl name: pandas version: 3.0.3 @@ -10066,6 +10001,44 @@ packages: requires_dist: - typing-extensions>=4.14.1 requires_python: '>=3.9' +- pypi: https://files.pythonhosted.org/packages/8a/2b/514fce8a7df81cf5bad7ff7865de7ac0c5776a38cc043475c4703eb7fe8b/sqlalchemy-2.0.50-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl + name: sqlalchemy + version: 2.0.50 + sha256: 110fdac56ace278949f00de805edacbd6141e382d992f9ba28238b3a0827a600 + requires_dist: + - importlib-metadata ; python_full_version < '3.8' + - greenlet>=1 ; platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64' + - typing-extensions>=4.6.0 + - greenlet>=1 ; extra == 'asyncio' + - mypy>=0.910 ; extra == 'mypy' + - pyodbc ; extra == 'mssql' + - pymssql ; extra == 'mssql-pymssql' + - pyodbc ; extra == 'mssql-pyodbc' + - mysqlclient>=1.4.0 ; extra == 'mysql' + - mysql-connector-python ; extra == 'mysql-connector' + - mariadb>=1.0.1,!=1.1.2,!=1.1.5,!=1.1.10 ; extra == 'mariadb-connector' + - cx-oracle>=8 ; extra == 'oracle' + - oracledb>=1.0.1 ; extra == 'oracle-oracledb' + - psycopg2>=2.7 ; extra == 'postgresql' + - pg8000>=1.29.1 ; extra == 'postgresql-pg8000' + - greenlet>=1 ; extra == 'postgresql-asyncpg' + - asyncpg ; extra == 'postgresql-asyncpg' + - psycopg2-binary ; extra == 'postgresql-psycopg2binary' + - psycopg2cffi ; extra == 'postgresql-psycopg2cffi' + - psycopg>=3.0.7 ; extra == 'postgresql-psycopg' + - psycopg[binary]>=3.0.7 ; extra == 'postgresql-psycopgbinary' + - pymysql ; extra == 'pymysql' + - greenlet>=1 ; extra == 'aiomysql' + - aiomysql>=0.2.0 ; extra == 'aiomysql' + - greenlet>=1 ; extra == 'aioodbc' + - aioodbc ; extra == 'aioodbc' + - greenlet>=1 ; extra == 'asyncmy' + - asyncmy>=0.2.3,!=0.2.4,!=0.2.6 ; extra == 'asyncmy' + - greenlet>=1 ; extra == 'aiosqlite' + - aiosqlite ; extra == 'aiosqlite' + - typing-extensions!=3.10.0.1 ; extra == 'aiosqlite' + - sqlcipher3-binary ; extra == 'sqlcipher' + requires_python: '>=3.7' - pypi: https://files.pythonhosted.org/packages/8a/a1/8d812e53a5da1687abb10445275d41a8b13adb781bbf7196ddbcf8d88505/lazy_loader-0.5-py3-none-any.whl name: lazy-loader version: '0.5' diff --git a/pyproject.toml b/pyproject.toml index 790d72195..421686854 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -47,7 +47,6 @@ dependencies = [ 'darkdetect', # Detecting dark mode (system-level) 'pandas', # Displaying tables in Jupyter notebooks 'plotly', # Interactive plots - 'arviz', # Bayesian analysis summaries and posterior plotting 'py3Dmol', # Visualisation of crystal structures ] diff --git a/src/easydiffraction/analysis/fit_helpers/_diagnostics.py b/src/easydiffraction/analysis/fit_helpers/_diagnostics.py new file mode 100644 index 000000000..8696992d5 --- /dev/null +++ b/src/easydiffraction/analysis/fit_helpers/_diagnostics.py @@ -0,0 +1,172 @@ +# SPDX-FileCopyrightText: 2026 EasyScience contributors +# SPDX-License-Identifier: BSD-3-Clause +""" +MCMC convergence diagnostics computed in pure NumPy + SciPy. + +The two diagnostics this module produces — split-chain Gelman–Rubin +R̂ and bulk effective sample size (ESS) — are the only Bayesian +diagnostics EasyDiffraction reports. The implementations follow the +standard formulas described in Vehtari, Gelman, Simpson, Carpenter +and Bürkner (2019), *Rank-normalization, folding, and localization: +An improved R̂ for assessing convergence of MCMC* +(https://arxiv.org/abs/1903.08008), Stan's reference manual, and +Geyer (1992), *Practical Markov chain Monte Carlo*. + +Inputs use the project's preserved layout: a 2-D NumPy array of +shape ``(n_draws, n_chains)`` per parameter, never an ArviZ +``InferenceData`` object. +""" + +from __future__ import annotations + +import numpy as np +from scipy import stats + +_MIN_DRAWS = 4 + + +def compute_r_hat(samples: np.ndarray) -> float: + """ + Split-chain Gelman–Rubin R̂ for one parameter. + + Each chain is split in half (the standard "split R̂" variant); + the within-chain (W) and between-chain (B) variances are + computed on the doubled chain set, and R̂ is returned as + ``sqrt(V̂ / W)`` where ``V̂ = ((n-1)/n) · W + B/n``. + + Parameters + ---------- + samples : np.ndarray + Posterior samples for one parameter with shape + ``(n_draws, n_chains)``. + + Returns + ------- + float + R̂ value. ``nan`` when there are fewer than 4 draws, fewer + than 2 chains, or zero within-chain variance. + + Raises + ------ + ValueError + If ``samples`` is not 2-D. + """ + if samples.ndim != 2: + msg = 'samples must have shape (n_draws, n_chains)' + raise ValueError(msg) + n_draws, n_chains = samples.shape + if n_draws < _MIN_DRAWS or n_chains < 2: + return float('nan') + + # Split each chain in half. With odd n_draws, drop the middle sample. + half = n_draws // 2 + splits = np.concatenate( + [samples[:half, :], samples[-half:, :]], + axis=1, + ) + + n_split = splits.shape[0] + chain_means = splits.mean(axis=0) + chain_vars = splits.var(axis=0, ddof=1) + + within = float(chain_vars.mean()) + if within == 0 or not np.isfinite(within): + return float('nan') + + between = n_split * float(chain_means.var(ddof=1)) + var_hat = ((n_split - 1) / n_split) * within + between / n_split + return float(np.sqrt(var_hat / within)) + + +def compute_ess_bulk(samples: np.ndarray) -> float: + """ + Bulk effective sample size for one parameter. + + Samples are rank-normalized across all chain/draw pairs (so the + diagnostic is robust to heavy-tailed marginals); the + autocorrelation function is then averaged across chains and + summed with Geyer's initial positive sequence: pairs of + consecutive lags are added to the running variance estimate + until a pair first becomes non-positive (Geyer 1992; Vehtari + et al. 2019 §3.1). + + Parameters + ---------- + samples : np.ndarray + Posterior samples for one parameter with shape + ``(n_draws, n_chains)``. + + Returns + ------- + float + Effective sample size in the bulk of the posterior. ``nan`` + when there are fewer than 4 draws, no chains, zero + variance, or the autocorrelation sum is non-positive. + + Raises + ------ + ValueError + If ``samples`` is not 2-D. + """ + if samples.ndim != 2: + msg = 'samples must have shape (n_draws, n_chains)' + raise ValueError(msg) + n_draws, n_chains = samples.shape + if n_draws < _MIN_DRAWS or n_chains < 1: + return float('nan') + + total = n_draws * n_chains + + # Rank-normalize across all samples → standard normal scores. + ranks = stats.rankdata(samples.ravel()).reshape(samples.shape) + z = stats.norm.ppf((ranks - 0.5) / total) + + # Per-chain autocorrelation, then average across chains. + acf_sum = np.zeros(n_draws) + for chain_index in range(n_chains): + acf_sum += _autocorr_fft(z[:, chain_index]) + rho = acf_sum / n_chains + + if not np.isfinite(rho[0]) or rho[0] <= 0: + return float('nan') + + # Geyer's initial positive sequence on pairs of lags. + tau = 1.0 + for t in range(1, n_draws // 2): + pair = rho[2 * t - 1] + rho[2 * t] + if pair <= 0: + break + tau += 2.0 * pair + + if tau <= 0 or not np.isfinite(tau): + return float('nan') + return float(total / tau) + + +def _autocorr_fft(series: np.ndarray) -> np.ndarray: + """ + Return the normalized autocorrelation function via FFT. + + Pads to the next power of two so the FFT is well-conditioned + for arbitrary chain lengths. The returned ACF has the same + length as the input series and starts at ``rho[0] = 1`` when + the input has non-zero variance. + + Parameters + ---------- + series : np.ndarray + One-dimensional sequence of samples. + + Returns + ------- + np.ndarray + Autocorrelation values at lags 0, 1, …, ``len(series) - 1``. + """ + n = series.size + centered = series - series.mean() + size = 1 << (2 * n - 1).bit_length() + fx = np.fft.fft(centered, n=size) + acf = np.fft.ifft(fx * np.conj(fx))[:n].real + if acf[0] == 0: + return acf + return acf / acf[0] diff --git a/src/easydiffraction/analysis/fit_helpers/bayesian.py b/src/easydiffraction/analysis/fit_helpers/bayesian.py index 67da76d9a..42ce7d770 100644 --- a/src/easydiffraction/analysis/fit_helpers/bayesian.py +++ b/src/easydiffraction/analysis/fit_helpers/bayesian.py @@ -4,17 +4,19 @@ from __future__ import annotations -import warnings from dataclasses import dataclass -import arviz as az import numpy as np + +from easydiffraction.analysis.fit_helpers._diagnostics import compute_ess_bulk +from easydiffraction.analysis.fit_helpers._diagnostics import compute_r_hat from easydiffraction.analysis.fit_helpers.metrics import calculate_r_factor from easydiffraction.analysis.fit_helpers.metrics import calculate_r_factor_squared from easydiffraction.analysis.fit_helpers.metrics import calculate_rb_factor from easydiffraction.analysis.fit_helpers.metrics import calculate_weighted_r_factor from easydiffraction.analysis.fit_helpers.reporting import FitResults from easydiffraction.analysis.fit_helpers.reporting import _build_parameter_row +from easydiffraction.analysis.fit_helpers.reporting import _overall_status_row_label from easydiffraction.core.posterior import PosteriorParameterSummary from easydiffraction.utils.logging import console from easydiffraction.utils.utils import print_metrics_table @@ -115,20 +117,21 @@ def flattened(self) -> np.ndarray: """ return np.asarray(self.parameter_samples).reshape(-1, len(self.parameter_names)) - def to_arviz(self) -> object: + def validate_shapes(self) -> tuple[int, int, int]: """ - Convert posterior samples to an ArviZ ``InferenceData`` object. + Validate stored sample shapes and return ``(n_draws, n_chains, n_parameters)``. Returns ------- - object - ArviZ ``InferenceData`` instance built from the stored - posterior samples. + tuple[int, int, int] + Tuple ``(n_draws, n_chains, n_parameters)``. Raises ------ ValueError - If the stored arrays do not have the expected shapes. + If the sample array is not 3-D, the parameter axis does + not match ``parameter_names``, or ``log_posterior`` (when + present) does not match the first two sample axes. """ posterior_array = np.asarray(self.parameter_samples, dtype=float) if posterior_array.ndim != POSTERIOR_SAMPLE_NDIM: @@ -140,31 +143,13 @@ def to_arviz(self) -> object: msg = 'Posterior sample array does not match the parameter name list length.' raise ValueError(msg) - posterior_dict = { - name: np.transpose(posterior_array[:, :, index], (1, 0)) - for index, name in enumerate(self.parameter_names) - } - - sample_stats: dict[str, np.ndarray] | None = None if self.log_posterior is not None: log_posterior = np.asarray(self.log_posterior, dtype=float) if log_posterior.shape != (n_draws, n_chains): msg = 'Log-posterior array must match the first two posterior sample axes.' raise ValueError(msg) - sample_stats = {'lp': np.transpose(log_posterior, (1, 0))} - - data = {'posterior': posterior_dict} - if sample_stats is not None: - data['sample_stats'] = sample_stats - - with warnings.catch_warnings(): - warnings.filterwarnings( - 'ignore', - message='Found chain dimension to be longer than draw dimension.*', - category=UserWarning, - module='arviz_base.base', - ) - return az.from_dict(data) + + return n_draws, n_chains, n_parameters SummaryList = list[PosteriorParameterSummary] | None @@ -329,7 +314,7 @@ def _build_fit_results_rows(self, metrics: dict[str, float | None]) -> list[list sampler_label = self.minimizer_type or self.sampler_name if sampler_label: rows.append(['🧪 Sampler', str(sampler_label)]) - rows.append(['✅ Overall status', overall_status]) + rows.append([_overall_status_row_label(overall_status), overall_status]) if self.message: rows.append(['💬 Engine message', self.message]) if self.fitting_time is not None: @@ -393,12 +378,15 @@ def compute_convergence_diagnostics(posterior_samples: PosteriorSamples) -> dict dict[str, object] Convergence metrics keyed by diagnostic name. """ - inference_data = posterior_samples.to_arviz() - rhat_dataset = az.rhat(inference_data) - ess_dataset = az.ess(inference_data, method='bulk') + n_draws, n_chains, _n_parameters = posterior_samples.validate_shapes() + parameter_samples = np.asarray(posterior_samples.parameter_samples, dtype=float) - r_hat_by_parameter = _dataset_to_scalar_dict(rhat_dataset) - ess_bulk_by_parameter = _dataset_to_scalar_dict(ess_dataset) + r_hat_by_parameter: dict[str, float | None] = {} + ess_bulk_by_parameter: dict[str, float | None] = {} + for index, name in enumerate(posterior_samples.parameter_names): + per_parameter = parameter_samples[:, :, index] + r_hat_by_parameter[name] = _maybe_scalar(compute_r_hat(per_parameter)) + ess_bulk_by_parameter[name] = _maybe_scalar(compute_ess_bulk(per_parameter)) finite_r_hat = [value for value in r_hat_by_parameter.values() if value is not None] finite_ess_bulk = [value for value in ess_bulk_by_parameter.values() if value is not None] @@ -420,8 +408,8 @@ def compute_convergence_diagnostics(posterior_samples: PosteriorSamples) -> dict 'ess_bulk_by_parameter': ess_bulk_by_parameter, 'max_r_hat': max_r_hat, 'min_ess_bulk': min_ess_bulk, - 'n_draws': int(posterior_samples.parameter_samples.shape[0]), - 'n_chains': int(posterior_samples.parameter_samples.shape[1]), + 'n_draws': n_draws, + 'n_chains': n_chains, 'n_parameters': len(posterior_samples.parameter_names), } @@ -522,13 +510,6 @@ def standard_deviations_from_summaries( return np.array([summary.standard_deviation for summary in summaries], dtype=float) -def _dataset_to_scalar_dict(dataset: object) -> dict[str, float | None]: - values: dict[str, float | None] = {} - for name, data_array in dataset.data_vars.items(): - values[name] = _maybe_scalar(np.asarray(data_array).reshape(-1)[0]) - return values - - def _maybe_scalar(value: object) -> float | None: if value is None: return None diff --git a/src/easydiffraction/analysis/fit_helpers/reporting.py b/src/easydiffraction/analysis/fit_helpers/reporting.py index 56ef0e526..37a933c2d 100644 --- a/src/easydiffraction/analysis/fit_helpers/reporting.py +++ b/src/easydiffraction/analysis/fit_helpers/reporting.py @@ -12,6 +12,12 @@ from easydiffraction.utils.utils import render_table +def _overall_status_row_label(status: str) -> str: + """Return the metric label for an overall status row.""" + icon = '✅' if status == 'success' else '❌' + return f'{icon} Overall status' + + class FitResults: """ Container for results of a single optimization run. @@ -158,7 +164,8 @@ def _build_fit_results_rows( rows: list[list[str]] = [] if self.minimizer_type is not None: rows.append(['🧪 Minimizer', str(self.minimizer_type)]) - rows.append(['✅ Overall status', 'success' if self.success else 'failed']) + overall_status = 'success' if self.success else 'failed' + rows.append([_overall_status_row_label(overall_status), overall_status]) if self.fitting_time is not None: rows.append(['⏱️ Fitting time (seconds)', f'{self.fitting_time:.2f}']) if self.iterations: @@ -278,5 +285,3 @@ def _compute_relative_change(param: object) -> str: change = ((param.value - param._fit_start_value) / param._fit_start_value) * 100 arrow = '↑' if change > 0 else '↓' return f'{abs(change):.2f} % {arrow}' - - diff --git a/src/easydiffraction/display/plotting.py b/src/easydiffraction/display/plotting.py index 200843411..ce838dbdd 100644 --- a/src/easydiffraction/display/plotting.py +++ b/src/easydiffraction/display/plotting.py @@ -86,7 +86,7 @@ class PosteriorPairPlotStyleEnum(StrEnum): DEFAULT_BRAGG_PEAKS_HEIGHT_FRACTION = 0.10 DEFAULT_RESID_HEIGHT = DEFAULT_RESIDUAL_HEIGHT_FRACTION DEFAULT_BRAGG_ROW = DEFAULT_BRAGG_PEAKS_HEIGHT_FRACTION -DEFAULT_POSTERIOR_PREDICTIVE_DRAWS = 200 +DEFAULT_POSTERIOR_PREDICTIVE_DRAWS = 50 DEFAULT_POSTERIOR_PREDICTIVE_DRAW_PLOT_CAP = 50 FULL_POSTERIOR_PAIR_COVARIANCE_RANK = 2 POSTERIOR_FLATTENED_SAMPLE_NDIM = 2 @@ -3820,33 +3820,6 @@ def _posterior_predictive_draw_indices(n_draws: int) -> np.ndarray: ) ) - def _get_posterior_inference_data( - self, - ) -> tuple[object | None, object | None]: - """ - Return posterior inference data for the current Bayesian fit. - - Returns - ------- - tuple[object | None, object | None] - ``(inference_data, fit_results)`` when posterior samples are - available, otherwise ``(None, None)``. - """ - if self.engine != PlotterEngineEnum.PLOTLY.value: - log.warning('Posterior plots currently require the Plotly plotting backend.') - return None, None - - fit_results = self._get_fit_result_for_correlation() - if fit_results is None: - return None, None - - posterior_samples = getattr(fit_results, 'posterior_samples', None) - if posterior_samples is None: - log.warning('Posterior samples are unavailable. Run a Bayesian fit first.') - return None, None - - return posterior_samples.to_arviz(), fit_results - def _get_posterior_samples_and_fit_results( self, ) -> tuple[object | None, object | None]: diff --git a/src/easydiffraction/utils/utils.py b/src/easydiffraction/utils/utils.py index 7834df1dd..d98e3ac64 100644 --- a/src/easydiffraction/utils/utils.py +++ b/src/easydiffraction/utils/utils.py @@ -606,7 +606,9 @@ def download_tutorial( id : int | str Numeric tutorial id (e.g. 1). destination : str, default='tutorials' - Directory to save the file into (created if missing). + Directory to save the file into (created if missing). Relative + destinations are resolved against the configured artifact root + when ``EASYDIFFRACTION_ARTIFACT_ROOT`` is set. overwrite : bool, default=False Whether to overwrite the file if it already exists. @@ -637,7 +639,7 @@ def download_tutorial( fname = f'ed-{id}.ipynb' - dest_path = pathlib.Path(destination) + dest_path = resolve_artifact_path(destination) dest_path.mkdir(parents=True, exist_ok=True) file_path = dest_path / fname @@ -683,7 +685,9 @@ def download_all_tutorials( Parameters ---------- destination : str, default='tutorials' - Directory to save the files into (created if missing). + Directory to save the files into (created if missing). Relative + destinations are resolved against the configured artifact root + when ``EASYDIFFRACTION_ARTIFACT_ROOT`` is set. overwrite : bool, default=False Whether to overwrite files if they already exist. @@ -712,8 +716,10 @@ def download_all_tutorials( except (OSError, ValueError) as e: log.warning(f'Failed to download tutorial #{tutorial_id}: {e}') + resolved_destination = resolve_artifact_path(destination) console.print( - f"✅ Downloaded {len(downloaded_paths)} tutorials to '{display_path(destination)}'" + f"✅ Downloaded {len(downloaded_paths)} tutorials to " + f"'{display_path(resolved_destination)}'" ) return downloaded_paths diff --git a/tests/integration/fitting/test_bayesian_helper_support.py b/tests/integration/fitting/test_bayesian_helper_support.py index 45a89be89..1c3d67169 100644 --- a/tests/integration/fitting/test_bayesian_helper_support.py +++ b/tests/integration/fitting/test_bayesian_helper_support.py @@ -33,7 +33,7 @@ def __init__(self, unique_name: str, start: float, value: float, uncertainty: fl self.units = 'arb' -def test_posterior_samples_flatten_and_to_arviz(): +def test_posterior_samples_flatten(): from easydiffraction.analysis.fit_helpers.bayesian import PosteriorSamples posterior_samples = PosteriorSamples( @@ -49,17 +49,13 @@ def test_posterior_samples_flatten_and_to_arviz(): ) flattened = posterior_samples.flattened() - inference_data = posterior_samples.to_arviz() assert flattened.shape == (4, 2) np.testing.assert_allclose(flattened[:, 0], np.array([1.0, 2.0, 3.0, 4.0])) np.testing.assert_allclose(flattened[:, 1], np.array([10.0, 20.0, 30.0, 40.0])) - assert set(inference_data.posterior.data_vars) == {'a', 'b'} - assert inference_data.posterior['a'].shape == (2, 2) - assert inference_data.sample_stats['lp'].shape == (2, 2) -def test_posterior_samples_to_arviz_validates_shapes(): +def test_posterior_samples_validate_shapes_rejects_wrong_ndim(): from easydiffraction.analysis.fit_helpers.bayesian import PosteriorSamples posterior_samples = PosteriorSamples( @@ -71,10 +67,10 @@ def test_posterior_samples_to_arviz_validates_shapes(): ValueError, match=r'Posterior sample array must have shape \(n_draws, n_chains, n_parameters\)\.', ): - posterior_samples.to_arviz() + posterior_samples.validate_shapes() -def test_posterior_samples_to_arviz_validates_name_and_log_posterior_lengths(): +def test_posterior_samples_validate_shapes_rejects_name_and_log_posterior_mismatches(): from easydiffraction.analysis.fit_helpers.bayesian import PosteriorSamples wrong_names = PosteriorSamples( @@ -85,7 +81,7 @@ def test_posterior_samples_to_arviz_validates_name_and_log_posterior_lengths(): ValueError, match=r'Posterior sample array does not match the parameter name list length\.', ): - wrong_names.to_arviz() + wrong_names.validate_shapes() wrong_log_posterior = PosteriorSamples( parameter_names=['a'], @@ -96,7 +92,7 @@ def test_posterior_samples_to_arviz_validates_name_and_log_posterior_lengths(): ValueError, match=r'Log-posterior array must match the first two posterior sample axes\.', ): - wrong_log_posterior.to_arviz() + wrong_log_posterior.validate_shapes() def test_compute_convergence_diagnostics_treats_non_finite_values_as_not_converged( @@ -110,17 +106,13 @@ def test_compute_convergence_diagnostics_treats_non_finite_values_as_not_converg parameter_samples=np.ones((4, 2, 1), dtype=float), ) - fake_dataset = type('FakeDataset', (), {'data_vars': {'a': np.array([np.nan], dtype=float)}}) - monkeypatch.setattr( - 'easydiffraction.analysis.fit_helpers.bayesian.az.rhat', - lambda inference_data: fake_dataset, + 'easydiffraction.analysis.fit_helpers.bayesian.compute_r_hat', + lambda _samples: float('nan'), ) monkeypatch.setattr( - 'easydiffraction.analysis.fit_helpers.bayesian.az.ess', - lambda inference_data, method='bulk': type( - 'FakeDataset', (), {'data_vars': {'a': np.array([4000.0], dtype=float)}} - ), + 'easydiffraction.analysis.fit_helpers.bayesian.compute_ess_bulk', + lambda _samples: 4000.0, ) diagnostics = compute_convergence_diagnostics(posterior_samples) @@ -219,19 +211,11 @@ def test_standard_deviations_from_summaries_returns_float_array(): def test_bayesian_format_helpers_cover_edge_cases(): from easydiffraction.analysis.fit_helpers.bayesian import _bayesian_overall_status from easydiffraction.analysis.fit_helpers.bayesian import _calculate_fit_quality_metrics - from easydiffraction.analysis.fit_helpers.bayesian import _dataset_to_scalar_dict from easydiffraction.analysis.fit_helpers.bayesian import _maybe_scalar - dataset = type( - 'FakeDataset', - (), - {'data_vars': {'a': np.array([np.nan], dtype=float), 'b': np.array([3.0], dtype=float)}}, - ) - assert _maybe_scalar(None) is None assert _maybe_scalar(float('inf')) is None assert _maybe_scalar(3.0) == pytest.approx(3.0) - assert _dataset_to_scalar_dict(dataset) == {'a': None, 'b': 3.0} # Two-state overall-status helper: 'success' only when sampler # completed AND convergence passed. diff --git a/tests/unit/easydiffraction/analysis/fit_helpers/test__diagnostics.py b/tests/unit/easydiffraction/analysis/fit_helpers/test__diagnostics.py new file mode 100644 index 000000000..aade1b4eb --- /dev/null +++ b/tests/unit/easydiffraction/analysis/fit_helpers/test__diagnostics.py @@ -0,0 +1,111 @@ +# SPDX-FileCopyrightText: 2026 EasyScience contributors +# SPDX-License-Identifier: BSD-3-Clause + +from __future__ import annotations + +import math + +import numpy as np +import pytest + +from easydiffraction.analysis.fit_helpers._diagnostics import _autocorr_fft +from easydiffraction.analysis.fit_helpers._diagnostics import compute_ess_bulk +from easydiffraction.analysis.fit_helpers._diagnostics import compute_r_hat + + +def _independent_samples(n_draws: int, n_chains: int, seed: int) -> np.ndarray: + rng = np.random.default_rng(seed) + return rng.standard_normal((n_draws, n_chains)) + + +def _correlated_chain(n_draws: int, n_chains: int, rho: float, seed: int) -> np.ndarray: + """Generate AR(1) chains with autocorrelation ``rho``.""" + rng = np.random.default_rng(seed) + samples = np.zeros((n_draws, n_chains)) + samples[0, :] = rng.standard_normal(n_chains) + sigma = math.sqrt(1.0 - rho * rho) + for t in range(1, n_draws): + samples[t, :] = rho * samples[t - 1, :] + sigma * rng.standard_normal(n_chains) + return samples + + +def test_r_hat_returns_nan_for_too_few_draws_or_chains(): + assert math.isnan(compute_r_hat(np.ones((3, 4)))) + assert math.isnan(compute_r_hat(np.ones((100, 1)))) + + +def test_r_hat_returns_nan_for_zero_variance(): + assert math.isnan(compute_r_hat(np.ones((100, 4)))) + + +def test_r_hat_rejects_non_2d_arrays(): + with pytest.raises(ValueError, match=r'samples must have shape \(n_draws, n_chains\)'): + compute_r_hat(np.ones((10, 4, 2))) + + +def test_r_hat_close_to_one_for_well_mixed_independent_chains(): + samples = _independent_samples(n_draws=2000, n_chains=4, seed=42) + r_hat = compute_r_hat(samples) + + assert math.isfinite(r_hat) + assert 0.98 < r_hat < 1.05 + + +def test_r_hat_above_one_when_chains_disagree(): + rng = np.random.default_rng(1) + chains_with_offsets = ( + rng.standard_normal((1000, 4)) + + np.array([-2.0, -1.0, 1.0, 2.0]) + ) + r_hat = compute_r_hat(chains_with_offsets) + + assert math.isfinite(r_hat) + assert r_hat > 1.5 + + +def test_r_hat_handles_odd_number_of_draws(): + samples = _independent_samples(n_draws=2001, n_chains=4, seed=7) + r_hat = compute_r_hat(samples) + + assert math.isfinite(r_hat) + assert 0.97 < r_hat < 1.05 + + +def test_ess_bulk_returns_nan_for_too_few_draws(): + assert math.isnan(compute_ess_bulk(np.ones((3, 4)))) + + +def test_ess_bulk_rejects_non_2d_arrays(): + with pytest.raises(ValueError, match=r'samples must have shape \(n_draws, n_chains\)'): + compute_ess_bulk(np.ones((10, 4, 2))) + + +def test_ess_bulk_near_total_for_independent_samples(): + samples = _independent_samples(n_draws=2000, n_chains=4, seed=123) + ess = compute_ess_bulk(samples) + + assert math.isfinite(ess) + # Rank-normalized ACF on truly independent samples is dominated by + # noise around zero; the Geyer initial-positive-sequence sum truncates + # quickly and ESS is close to the total sample count. + assert ess > 0.5 * samples.size + + +def test_ess_bulk_lower_for_strongly_autocorrelated_chains(): + independent = _independent_samples(n_draws=2000, n_chains=4, seed=11) + correlated = _correlated_chain(n_draws=2000, n_chains=4, rho=0.9, seed=11) + + ess_independent = compute_ess_bulk(independent) + ess_correlated = compute_ess_bulk(correlated) + + assert math.isfinite(ess_independent) + assert math.isfinite(ess_correlated) + assert ess_correlated < 0.4 * ess_independent + + +def test_autocorr_fft_handles_zero_variance(): + flat = np.ones(64) + acf = _autocorr_fft(flat) + + # Zero-variance input: ACF is all zeros except the lag-0 element. + assert acf[0] == 0.0 diff --git a/tests/unit/easydiffraction/analysis/fit_helpers/test_bayesian.py b/tests/unit/easydiffraction/analysis/fit_helpers/test_bayesian.py index 960214bbd..da29f65ba 100644 --- a/tests/unit/easydiffraction/analysis/fit_helpers/test_bayesian.py +++ b/tests/unit/easydiffraction/analysis/fit_helpers/test_bayesian.py @@ -3,8 +3,6 @@ from __future__ import annotations -import warnings - import numpy as np import pytest @@ -33,7 +31,7 @@ def test_module_import(): assert MUT.__name__ == 'easydiffraction.analysis.fit_helpers.bayesian' -def test_posterior_samples_flatten_and_to_arviz(): +def test_posterior_samples_flatten(): from easydiffraction.analysis.fit_helpers.bayesian import PosteriorSamples posterior_samples = PosteriorSamples( @@ -49,17 +47,13 @@ def test_posterior_samples_flatten_and_to_arviz(): ) flattened = posterior_samples.flattened() - inference_data = posterior_samples.to_arviz() assert flattened.shape == (4, 2) np.testing.assert_allclose(flattened[:, 0], np.array([1.0, 2.0, 3.0, 4.0])) np.testing.assert_allclose(flattened[:, 1], np.array([10.0, 20.0, 30.0, 40.0])) - assert set(inference_data.posterior.data_vars) == {'a', 'b'} - assert inference_data.posterior['a'].shape == (2, 2) - assert inference_data.sample_stats['lp'].shape == (2, 2) -def test_posterior_samples_to_arviz_allows_more_chains_than_draws_without_warning(): +def test_posterior_samples_validate_shapes_returns_dimensions(): from easydiffraction.analysis.fit_helpers.bayesian import PosteriorSamples posterior_samples = PosteriorSamples( @@ -68,16 +62,10 @@ def test_posterior_samples_to_arviz_allows_more_chains_than_draws_without_warnin log_posterior=np.ones((2, 32), dtype=float), ) - with warnings.catch_warnings(record=True) as caught_warnings: - warnings.simplefilter('always') - inference_data = posterior_samples.to_arviz() - - warning_messages = [str(warning.message) for warning in caught_warnings] - assert not any('Found chain dimension' in message for message in warning_messages) - assert inference_data.posterior['a'].shape == (32, 2) + assert posterior_samples.validate_shapes() == (2, 32, 1) -def test_posterior_samples_to_arviz_validates_shapes(): +def test_posterior_samples_validate_shapes_rejects_wrong_ndim(): from easydiffraction.analysis.fit_helpers.bayesian import PosteriorSamples posterior_samples = PosteriorSamples( @@ -89,7 +77,7 @@ def test_posterior_samples_to_arviz_validates_shapes(): ValueError, match=r'Posterior sample array must have shape \(n_draws, n_chains, n_parameters\)\.', ): - posterior_samples.to_arviz() + posterior_samples.validate_shapes() def test_compute_convergence_diagnostics_treats_non_finite_values_as_not_converged(monkeypatch): @@ -101,17 +89,13 @@ def test_compute_convergence_diagnostics_treats_non_finite_values_as_not_converg parameter_samples=np.ones((4, 2, 1), dtype=float), ) - fake_dataset = type('FakeDataset', (), {'data_vars': {'a': np.array([np.nan], dtype=float)}}) - monkeypatch.setattr( - 'easydiffraction.analysis.fit_helpers.bayesian.az.rhat', - lambda inference_data: fake_dataset, + 'easydiffraction.analysis.fit_helpers.bayesian.compute_r_hat', + lambda _samples: float('nan'), ) monkeypatch.setattr( - 'easydiffraction.analysis.fit_helpers.bayesian.az.ess', - lambda inference_data, method='bulk': type( - 'FakeDataset', (), {'data_vars': {'a': np.array([4000.0], dtype=float)}} - ), + 'easydiffraction.analysis.fit_helpers.bayesian.compute_ess_bulk', + lambda _samples: 4000.0, ) diagnostics = compute_convergence_diagnostics(posterior_samples) @@ -208,7 +192,8 @@ def test_bayesian_fit_results_display_results_prints_sampler_and_convergence(cap out = capsys.readouterr().out assert 'Bayesian fit results' in out - assert 'Overall status' in out + assert '❌ Overall status' in out + assert '✅ Overall status' not in out assert 'failed' in out # convergence failed → overall failed assert 'DREAM sampling completed' in out # engine message assert 'Sampler' in out diff --git a/tests/unit/easydiffraction/analysis/fit_helpers/test_reporting.py b/tests/unit/easydiffraction/analysis/fit_helpers/test_reporting.py index ab21c0d79..d5a241d17 100644 --- a/tests/unit/easydiffraction/analysis/fit_helpers/test_reporting.py +++ b/tests/unit/easydiffraction/analysis/fit_helpers/test_reporting.py @@ -10,6 +10,13 @@ def test_module_import(): assert expected_module_name == actual_module_name +def test_overall_status_row_label_uses_failure_icon(): + from easydiffraction.analysis.fit_helpers.reporting import _overall_status_row_label + + assert _overall_status_row_label('success') == '✅ Overall status' + assert _overall_status_row_label('failed') == '❌ Overall status' + + def test_fitresults_display_results_prints_and_table(capsys, monkeypatch): # Arrange: build a minimal fake parameter object with required attributes class Identity: @@ -50,7 +57,7 @@ def __init__(self, start, value, uncertainty, name='p', units='u'): # Assert: key lines printed and a table rendered out = capsys.readouterr().out assert 'Least-squares fit results:' in out - assert 'Overall status' in out + assert '✅ Overall status' in out assert 'success' in out assert 'reduced χ²' in out assert 'R-factor (Rf' in out diff --git a/tests/unit/easydiffraction/utils/test_utils.py b/tests/unit/easydiffraction/utils/test_utils.py index 1d6a8b23e..87f2d71b3 100644 --- a/tests/unit/easydiffraction/utils/test_utils.py +++ b/tests/unit/easydiffraction/utils/test_utils.py @@ -314,6 +314,39 @@ def __exit__(self, *args): assert (tmp_path / 'ed-1.ipynb').exists() +def test_download_tutorial_uses_artifact_root(monkeypatch, tmp_path): + import easydiffraction.utils.utils as MUT + + fake_index = { + '1': { + 'url': 'https://example.com/{version}/tutorials/ed-1/ed-1.ipynb', + 'title': 'Quick Start', + }, + } + artifact_root = tmp_path / 'artifacts' + monkeypatch.setenv('EASYDIFFRACTION_ARTIFACT_ROOT', str(artifact_root)) + monkeypatch.setattr(MUT, '_fetch_tutorials_index', lambda: fake_index) + monkeypatch.setattr(MUT, '_get_version_for_url', lambda: '0.8.0') + + class DummyResp: + def read(self): + return b'{"cells": []}' + + def __enter__(self): + return self + + def __exit__(self, *args): + return False + + monkeypatch.setattr(MUT, '_safe_urlopen', lambda url: DummyResp()) + + result = MUT.download_tutorial(id=1, destination='tutorials') + + expected_path = artifact_root / 'tutorials' / 'ed-1.ipynb' + assert result == str(expected_path) + assert expected_path.exists() + + def test_download_tutorial_already_exists_no_overwrite(monkeypatch, tmp_path, capsys): import easydiffraction.utils.utils as MUT @@ -390,6 +423,46 @@ def __exit__(self, *args): assert (tmp_path / 'ed-2.ipynb').exists() +def test_download_all_tutorials_reports_resolved_artifact_root( + monkeypatch, + tmp_path, + capsys, +): + import easydiffraction.utils.utils as MUT + + fake_index = { + '1': { + 'url': 'https://example.com/{version}/tutorials/ed-1/ed-1.ipynb', + 'title': 'Quick Start', + }, + } + artifact_root = tmp_path / 'artifacts' + monkeypatch.setenv('EASYDIFFRACTION_ARTIFACT_ROOT', str(artifact_root)) + monkeypatch.setattr(MUT, '_fetch_tutorials_index', lambda: fake_index) + monkeypatch.setattr(MUT, '_get_version_for_url', lambda: '0.8.0') + + class DummyResp: + def read(self): + return b'{"cells": []}' + + def __enter__(self): + return self + + def __exit__(self, *args): + return False + + monkeypatch.setattr(MUT, '_safe_urlopen', lambda url: DummyResp()) + + result = MUT.download_all_tutorials(destination='tutorials') + + expected_dir = artifact_root / 'tutorials' + assert result == [str(expected_dir / 'ed-1.ipynb')] + assert (expected_dir / 'ed-1.ipynb').exists() + out = capsys.readouterr().out + assert 'Downloaded 1 tutorials' in out + assert 'artifacts/tutorials' in out + + def test_resolve_tutorial_url(): # Test with a specific version From 477460c55ae8985194fa6fc76a37a6316e0d5bfb Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 21:26:56 +0200 Subject: [PATCH 37/65] Update tutorial archives and reduce steps --- docs/docs/tutorials/ed-21.py | 4 ++-- docs/docs/tutorials/ed-24.py | 46 +----------------------------------- docs/docs/tutorials/ed-25.py | 7 +++--- docs/docs/tutorials/ed-26.py | 5 +++- 4 files changed, 11 insertions(+), 51 deletions(-) diff --git a/docs/docs/tutorials/ed-21.py b/docs/docs/tutorials/ed-21.py index 087fa6718..13838dbb2 100644 --- a/docs/docs/tutorials/ed-21.py +++ b/docs/docs/tutorials/ed-21.py @@ -297,8 +297,8 @@ project.analysis.minimizer.type = 'bumps (dream)' # %% -project.analysis.minimizer.sampling_steps = 1000 # lower than the default 3000 -project.analysis.minimizer.burn_in_steps = 200 # lower than the default 600 +project.analysis.minimizer.sampling_steps = 100 # lower than the default 3000 +project.analysis.minimizer.burn_in_steps = 20 # lower than the default 600 # %% project.analysis.fit() diff --git a/docs/docs/tutorials/ed-24.py b/docs/docs/tutorials/ed-24.py index e1efcecf5..f0330bc95 100644 --- a/docs/docs/tutorials/ed-24.py +++ b/docs/docs/tutorials/ed-24.py @@ -12,51 +12,8 @@ # ## Import Library # %% -from pathlib import Path - import easydiffraction as ed - -# The ID 35 archive used below was saved before the -# switchable-category-owned-selectors refactor renamed several CIF -# tags. The helper below rewrites the archive in place so the tutorial -# can load it; it is intentionally narrow (ID 35 only, hrpt only, -# line-segment background only) and not a general legacy migration -# path. EasyDiffraction is in beta and does not ship legacy CIF -# shims, so saved projects in the old layout must be regenerated. The -# helper will be deleted once the upstream archive is republished -# under the current tag names. -def _normalize_id35_archive_for_tutorial(project_dir): - """Rewrite the ID 35 archive's CIF tags for the current API.""" - project_path = Path(project_dir) - - replacements_by_file = { - project_path / 'project.cif': { - '_rendering.chart_engine': '_chart.type', - '_rendering.table_engine': '_table.type', - }, - project_path / 'analysis' / 'analysis.cif': { - '_fitting.mode_type': '_fitting_mode.type', - '_fitting.minimizer_type': '_minimizer.type', - }, - project_path / 'experiments' / 'hrpt.cif': { - '_calculation.calculator_type': '_calculator.type', - '_peak.profile_type': '_peak.type', - }, - } - - for file_path, replacements in replacements_by_file.items(): - text = file_path.read_text(encoding='utf-8') - for old, new in replacements.items(): - text = text.replace(old, new) - if file_path.name == 'hrpt.cif' and '_background.type' not in text: - text = text.replace( - '\nloop_\n_pd_background.id\n', - '\n_background.type line-segment\nloop_\n_pd_background.id\n', - ) - file_path.write_text(text, encoding='utf-8') - - # %% [markdown] # ## Download Saved Project # @@ -65,8 +22,7 @@ def _normalize_id35_archive_for_tutorial(project_dir): # caches. # %% -project_dir = ed.download_data(id=35, destination='projects') -_normalize_id35_archive_for_tutorial(project_dir) +project_dir = ed.download_data(id=39, destination='projects') # %% [markdown] # ## Load the Saved Bayesian Project diff --git a/docs/docs/tutorials/ed-25.py b/docs/docs/tutorials/ed-25.py index d1f5a57c7..8a6d3035b 100644 --- a/docs/docs/tutorials/ed-25.py +++ b/docs/docs/tutorials/ed-25.py @@ -291,11 +291,12 @@ project.analysis.minimizer.type = 'emcee' # %% -project.analysis.minimizer.sampling_steps = 10000 # lower than the default 5000 -project.analysis.minimizer.burn_in_steps = 2000 # lower than the default 1000 +project.analysis.minimizer.sampling_steps = 100 # lower than the default 5000 +project.analysis.minimizer.burn_in_steps = 20 # lower than the default 1000 +project.analysis.minimizer.population_size = 16 # lower than the default 32 # %% -project.analysis.fit(resume=False) +project.analysis.fit() # %% [markdown] # ## Step 7: Inspect Bayesian Results diff --git a/docs/docs/tutorials/ed-26.py b/docs/docs/tutorials/ed-26.py index 29a39fc3f..42dc88803 100644 --- a/docs/docs/tutorials/ed-26.py +++ b/docs/docs/tutorials/ed-26.py @@ -35,7 +35,7 @@ # caches. # %% -project_dir = ed.download_data(id=35, destination='projects') +project_dir = ed.download_data(id=38, destination='projects') # %% [markdown] # ## Load the Saved Bayesian Project @@ -111,6 +111,9 @@ # # After resume, the posterior plots use the extended chain. +# %% +project.display.posterior.pairs() + # %% project.display.posterior.distribution() From 3ed647886d34c7791162317414269bf28e5cd924 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 21:28:28 +0200 Subject: [PATCH 38/65] Update notebooks --- docs/docs/tutorials/ed-13.ipynb | 2 +- docs/docs/tutorials/ed-21.ipynb | 4 +- docs/docs/tutorials/ed-22.py | 2 +- docs/docs/tutorials/ed-24.ipynb | 49 +--- docs/docs/tutorials/ed-25.ipynb | 380 ++++++++++++++++++++++++-------- docs/docs/tutorials/ed-26.ipynb | 307 ++++++++++++++++++++++++++ docs/docs/tutorials/ed-5.ipynb | 62 +++--- docs/docs/tutorials/index.md | 54 +++-- 8 files changed, 667 insertions(+), 193 deletions(-) create mode 100644 docs/docs/tutorials/ed-26.ipynb diff --git a/docs/docs/tutorials/ed-13.ipynb b/docs/docs/tutorials/ed-13.ipynb index a99270afa..36b56afea 100644 --- a/docs/docs/tutorials/ed-13.ipynb +++ b/docs/docs/tutorials/ed-13.ipynb @@ -2657,7 +2657,7 @@ ], "metadata": { "jupytext": { - "cell_metadata_filter": "title,tags,-all", + "cell_metadata_filter": "tags,title,-all", "main_language": "python", "notebook_metadata_filter": "-all" } diff --git a/docs/docs/tutorials/ed-21.ipynb b/docs/docs/tutorials/ed-21.ipynb index 2b02a7c63..285084dc5 100644 --- a/docs/docs/tutorials/ed-21.ipynb +++ b/docs/docs/tutorials/ed-21.ipynb @@ -24,7 +24,7 @@ "id": "1", "metadata": {}, "source": [ - "# Bayesian Analysis: LBCO, HRPT\n", + "# Bayesian Analysis (`bumps-dream`): LBCO, HRPT\n", "\n", "This tutorial demonstrates a practical two-stage workflow for powder\n", "diffraction analysis with EasyDiffraction.\n", @@ -97,7 +97,7 @@ "metadata": {}, "outputs": [], "source": [ - "project.save_as('projects/lbco_hrpt_bayesian')" + "project.save_as('projects/lbco_hrpt_bumps-dream')" ] }, { diff --git a/docs/docs/tutorials/ed-22.py b/docs/docs/tutorials/ed-22.py index 4b6c99eab..24fcd4369 100644 --- a/docs/docs/tutorials/ed-22.py +++ b/docs/docs/tutorials/ed-22.py @@ -1,5 +1,5 @@ # %% [markdown] -# # Bayesian Analysis: Tb2TiO7, HEiDi +# # Bayesian Analysis: Tb2TiO7 (`bumps-dream`), HEiDi # # This tutorial demonstrates a practical two-stage workflow for single-crystal # diffraction analysis with EasyDiffraction. diff --git a/docs/docs/tutorials/ed-24.ipynb b/docs/docs/tutorials/ed-24.ipynb index 4e77ed733..904635418 100644 --- a/docs/docs/tutorials/ed-24.ipynb +++ b/docs/docs/tutorials/ed-24.ipynb @@ -24,7 +24,7 @@ "id": "1", "metadata": {}, "source": [ - "# Load Saved Bayesian Project: LBCO, HRPT\n", + "# Bayesian Analysis Display (`bumps-dream`): LBCO, HRPT\n", "\n", "This tutorial shows how to reopen the Bayesian project created in\n", "`ed-21.py` and inspect the saved fit results without rerunning DREAM.\n", @@ -49,49 +49,7 @@ "metadata": {}, "outputs": [], "source": [ - "from pathlib import Path\n", - "\n", - "import easydiffraction as ed\n", - "\n", - "\n", - "# The ID 35 archive used below was saved before the\n", - "# switchable-category-owned-selectors refactor renamed several CIF\n", - "# tags. The helper below rewrites the archive in place so the tutorial\n", - "# can load it; it is intentionally narrow (ID 35 only, hrpt only,\n", - "# line-segment background only) and not a general legacy migration\n", - "# path. EasyDiffraction is in beta and does not ship legacy CIF\n", - "# shims, so saved projects in the old layout must be regenerated. The\n", - "# helper will be deleted once the upstream archive is republished\n", - "# under the current tag names.\n", - "def _normalize_id35_archive_for_tutorial(project_dir):\n", - " \"\"\"Rewrite the ID 35 archive's CIF tags for the current API.\"\"\"\n", - " project_path = Path(project_dir)\n", - "\n", - " replacements_by_file = {\n", - " project_path / 'project.cif': {\n", - " '_rendering.chart_engine': '_chart.type',\n", - " '_rendering.table_engine': '_table.type',\n", - " },\n", - " project_path / 'analysis' / 'analysis.cif': {\n", - " '_fitting.mode_type': '_fitting_mode.type',\n", - " '_fitting.minimizer_type': '_minimizer.type',\n", - " },\n", - " project_path / 'experiments' / 'hrpt.cif': {\n", - " '_calculation.calculator_type': '_calculator.type',\n", - " '_peak.profile_type': '_peak.type',\n", - " },\n", - " }\n", - "\n", - " for file_path, replacements in replacements_by_file.items():\n", - " text = file_path.read_text(encoding='utf-8')\n", - " for old, new in replacements.items():\n", - " text = text.replace(old, new)\n", - " if file_path.name == 'hrpt.cif' and '_background.type' not in text:\n", - " text = text.replace(\n", - " '\\nloop_\\n_pd_background.id\\n',\n", - " '\\n_background.type line-segment\\nloop_\\n_pd_background.id\\n',\n", - " )\n", - " file_path.write_text(text, encoding='utf-8')" + "import easydiffraction as ed" ] }, { @@ -113,8 +71,7 @@ "metadata": {}, "outputs": [], "source": [ - "project_dir = ed.download_data(id=35, destination='projects')\n", - "_normalize_id35_archive_for_tutorial(project_dir)" + "project_dir = ed.download_data(id=39, destination='projects')" ] }, { diff --git a/docs/docs/tutorials/ed-25.ipynb b/docs/docs/tutorials/ed-25.ipynb index e4f899efa..d29a0d746 100644 --- a/docs/docs/tutorials/ed-25.ipynb +++ b/docs/docs/tutorials/ed-25.ipynb @@ -24,20 +24,27 @@ "id": "1", "metadata": {}, "source": [ - "# Bayesian Analysis with emcee: LBCO, HRPT\n", + "# Bayesian Analysis (`emcee`): LBCO, HRPT\n", "\n", - "This tutorial demonstrates how to run Bayesian sampling with the\n", - "emcee minimizer and then resume the same chain from the saved project.\n", + "This tutorial demonstrates a practical two-stage workflow for powder\n", + "diffraction analysis with EasyDiffraction.\n", "\n", - "The workflow uses the same La0.5Ba0.5CoO3 powder diffraction example\n", - "as the DREAM Bayesian tutorial:\n", + "In the first stage, we run a fast local refinement to obtain a sensible\n", + "point estimate and parameter uncertainties. In the second stage, we use\n", + "these refined values to define fit bounds and then sample the posterior\n", + "distribution with emcee.\n", "\n", - "- run a short local refinement,\n", - "- derive finite fit bounds for the sampled parameters,\n", - "- switch to emcee and sample the posterior,\n", - "- save the project with the emcee chain,\n", - "- resume the chain with additional steps,\n", - "- inspect posterior plots after each sampling stage." + "The example uses constant-wavelength neutron powder diffraction data\n", + "for La0.5Ba0.5CoO3 measured on HRPT at PSI.\n", + "\n", + "The goal is not only to obtain a good fit, but also to answer Bayesian\n", + "questions such as:\n", + "\n", + "- Which parameter values are most probable?\n", + "- How broad are the credible intervals?\n", + "- Which parameters are strongly correlated?\n", + "- How much uncertainty propagates into the calculated diffraction\n", + " pattern?" ] }, { @@ -63,10 +70,14 @@ "id": "4", "metadata": {}, "source": [ - "## Create a Project Container\n", + "## Step 1: Create a Project Container\n", "\n", - "The project is saved before sampling because emcee stores its chain in\n", - "the project's analysis sidecar file." + "The project object keeps structures, experiments, fit settings, and\n", + "plotting utilities together in a single place. We will build the full\n", + "workflow inside this object.\n", + "\n", + "Save the project to a directory early on so that you can easily reload\n", + "it later if needed." ] }, { @@ -94,9 +105,11 @@ "id": "7", "metadata": {}, "source": [ - "## Build the Structural Model\n", + "## Step 2: Build the Structural Model\n", "\n", - "Define a compact cubic perovskite model for La0.5Ba0.5CoO3." + "We define a simple cubic perovskite model for LBCO. La and Ba share the\n", + "same crystallographic site with equal occupancy, while Co and O occupy\n", + "the remaining ideal perovskite positions." ] }, { @@ -140,10 +153,20 @@ "structure.cell.length_a = 3.88" ] }, + { + "cell_type": "markdown", + "id": "12", + "metadata": {}, + "source": [ + "The atom-site definitions below form the starting structural model. The\n", + "parameters are intentionally reasonable rather than fully optimized,\n", + "because the refinement step will improve them." + ] + }, { "cell_type": "code", "execution_count": null, - "id": "12", + "id": "13", "metadata": {}, "outputs": [], "source": [ @@ -193,29 +216,49 @@ }, { "cell_type": "markdown", - "id": "13", + "id": "14", "metadata": {}, "source": [ - "## Define the Diffraction Experiment\n", + "## Step 3: Define the Diffraction Experiment\n", "\n", - "Download the HRPT powder pattern, create a neutron powder experiment,\n", - "and set the key instrument, peak-profile, and background values." + "Next we download the measured powder pattern, create a neutron powder\n", + "experiment, and configure the instrument, profile, background, and\n", + "excluded regions." + ] + }, + { + "cell_type": "markdown", + "id": "15", + "metadata": {}, + "source": [ + "Download the measured data from the repository. Alternatively, you\n", + "could use your own data file by providing the path to it instead of\n", + "downloading from the repository." ] }, { "cell_type": "code", "execution_count": null, - "id": "14", + "id": "16", "metadata": {}, "outputs": [], "source": [ "data_path = ed.download_data(id=3, destination='data')" ] }, + { + "cell_type": "markdown", + "id": "17", + "metadata": {}, + "source": [ + "Create the experiment object and specify the sample form, beam mode,\n", + "and radiation probe." + ] + }, { "cell_type": "code", "execution_count": null, - "id": "15", + "id": "18", "metadata": {}, "outputs": [], "source": [ @@ -231,27 +274,46 @@ { "cell_type": "code", "execution_count": null, - "id": "16", + "id": "19", "metadata": {}, "outputs": [], "source": [ "experiment = project.experiments['hrpt']" ] }, + { + "cell_type": "markdown", + "id": "20", + "metadata": {}, + "source": [ + "Link the structural phase to the experiment." + ] + }, { "cell_type": "code", "execution_count": null, - "id": "17", + "id": "21", "metadata": {}, "outputs": [], "source": [ "experiment.linked_phases.create(id='lbco', scale=9.1351)" ] }, + { + "cell_type": "markdown", + "id": "22", + "metadata": {}, + "source": [ + "Set instrument and peak profile parameters.\n", + "\n", + "These values provide the initial instrument description for the local\n", + "refinement. Later, a subset of them will be refined." + ] + }, { "cell_type": "code", "execution_count": null, - "id": "18", + "id": "23", "metadata": {}, "outputs": [], "source": [ @@ -262,7 +324,7 @@ { "cell_type": "code", "execution_count": null, - "id": "19", + "id": "24", "metadata": {}, "outputs": [], "source": [ @@ -272,10 +334,21 @@ "experiment.peak.broad_lorentz_y = 0.0844" ] }, + { + "cell_type": "markdown", + "id": "25", + "metadata": {}, + "source": [ + "Add background points and excluded regions.\n", + "\n", + "The line-segment background is defined by a few anchor points. We also\n", + "exclude regions that are not intended to contribute to the fit." + ] + }, { "cell_type": "code", "execution_count": null, - "id": "20", + "id": "26", "metadata": {}, "outputs": [], "source": [ @@ -288,7 +361,7 @@ { "cell_type": "code", "execution_count": null, - "id": "21", + "id": "27", "metadata": {}, "outputs": [], "source": [ @@ -298,19 +371,27 @@ }, { "cell_type": "markdown", - "id": "22", + "id": "28", "metadata": {}, "source": [ - "## Run a Local Refinement First\n", + "## Step 4: Run an Initial Local Refinement\n", + "\n", + "Before Bayesian sampling, it is useful to run a deterministic fit. This\n", + "gives us:\n", + "\n", + "- a good point estimate near the best-fit region,\n", + "- uncertainties from the local optimizer,\n", + "- a quick check that the model and experiment are configured\n", + " sensibly.\n", "\n", - "The local fit provides starting values and uncertainties that are used\n", - "to build finite bounds for emcee." + "In this tutorial we refine only a small set of parameters that are easy\n", + "to interpret in the later Bayesian stage." ] }, { "cell_type": "code", "execution_count": null, - "id": "23", + "id": "29", "metadata": {}, "outputs": [], "source": [ @@ -320,7 +401,7 @@ { "cell_type": "code", "execution_count": null, - "id": "24", + "id": "30", "metadata": {}, "outputs": [], "source": [ @@ -330,20 +411,30 @@ "experiment.instrument.calib_twotheta_offset.free = True" ] }, + { + "cell_type": "markdown", + "id": "31", + "metadata": {}, + "source": [ + "We keep LMFIT Levenberg-Marquardt minimizer as a fast local optimizer.\n", + "Its main purpose here is to provide a stable starting point and\n", + "uncertainty estimates for the Bayesian run." + ] + }, { "cell_type": "code", "execution_count": null, - "id": "25", + "id": "32", "metadata": {}, "outputs": [], "source": [ - "project.analysis.minimizer.type = 'bumps (lm)'" + "project.analysis.minimizer.show_supported()" ] }, { "cell_type": "code", "execution_count": null, - "id": "26", + "id": "33", "metadata": {}, "outputs": [], "source": [ @@ -353,212 +444,307 @@ { "cell_type": "code", "execution_count": null, - "id": "27", + "id": "34", "metadata": {}, "outputs": [], "source": [ "project.display.fit.results()" ] }, + { + "cell_type": "markdown", + "id": "35", + "metadata": {}, + "source": [ + "The correlation plot shows how strongly the fitted parameters move\n", + "together in the local refinement. The measured-vs-calculated plots show\n", + "how well the refined model reproduces the data globally and in a zoomed\n", + "region." + ] + }, { "cell_type": "code", "execution_count": null, - "id": "28", + "id": "36", "metadata": {}, "outputs": [], "source": [ - "for param in project.free_parameters:\n", - " param.set_fit_bounds_from_uncertainty()" + "project.display.fit.correlations()" ] }, { "cell_type": "code", "execution_count": null, - "id": "29", + "id": "37", "metadata": {}, "outputs": [], "source": [ - "project.display.parameters.free()" + "project.display.pattern(expt_name='hrpt')" ] }, { "cell_type": "markdown", - "id": "30", + "id": "38", "metadata": {}, "source": [ - "## Run emcee Sampling\n", + "## Step 5: Prepare for Bayesian Sampling\n", + "\n", + "Bayesian samplers require finite bounds for the free parameters. Instead of\n", + "setting them manually, we derive them from the uncertainties estimated\n", + "in the local refinement.\n", "\n", - "The sampling settings are intentionally small for tutorial runtime.\n", - "Use more steps and inspect convergence diagnostics for production\n", - "analysis." + "The helper method `set_fit_bounds_from_uncertainty` centers the bounds\n", + "on the current parameter value and expands them by a chosen multiple of\n", + "the reported uncertainty.\n", + "\n", + "The default `multiplier` is 4. If the local refinement is very tight,\n", + "or if you expect a broader posterior, increase it explicitly.\n", + "\n", + "Show unset fit bounds before setting them from the local refinement uncertainties." ] }, { "cell_type": "code", "execution_count": null, - "id": "31", + "id": "39", "metadata": {}, "outputs": [], "source": [ - "project.analysis.minimizer.type = 'emcee'" + "project.display.parameters.free()" ] }, { - "cell_type": "code", - "execution_count": null, - "id": "32", + "cell_type": "markdown", + "id": "40", "metadata": {}, - "outputs": [], "source": [ - "project.analysis.minimizer.sampling_steps = 1000\n", - "project.analysis.minimizer.burn_in_steps = 200\n", - "project.analysis.minimizer.thinning_interval = 10\n", - "project.analysis.minimizer.population_size = 32\n", - "project.analysis.minimizer.initialization_method = 'ball'\n", - "project.analysis.minimizer.random_seed = 12345" + "Set fit bounds for all free parameters using the default multiplier of\n", + "4. In this tutorial that means the posterior pair plot will later\n", + "refer to a `±4 × uncertainty` region in its title. To use a different\n", + "region, pass another value, for example `multiplier=6`." ] }, { "cell_type": "code", "execution_count": null, - "id": "33", + "id": "41", "metadata": {}, "outputs": [], "source": [ - "project.analysis.fit()" + "for param in project.free_parameters:\n", + " param.set_fit_bounds_from_uncertainty()" + ] + }, + { + "cell_type": "markdown", + "id": "42", + "metadata": {}, + "source": [ + "Displaying the free parameters again is a convenient way to confirm\n", + "that the fit bounds have been assigned as expected before launching the\n", + "sampler." ] }, { "cell_type": "code", "execution_count": null, - "id": "34", + "id": "43", "metadata": {}, "outputs": [], "source": [ - "project.display.fit.results()" + "project.display.parameters.free()" ] }, { "cell_type": "markdown", - "id": "35", + "id": "44", "metadata": {}, "source": [ - "## Inspect the Posterior\n", + "## Step 6: Configure and Run emcee\n", "\n", - "The posterior distribution plot shows the sampled marginal\n", - "distributions after the first emcee run." + "We now switch from the local minimizer to the Bayesian emcee sampler.\n", + "\n", + "The settings below are intentionally small so the tutorial runs\n", + "quickly. For production analysis you would usually increase the number\n", + "of steps and often the burn-in as well. emcee also lets you tune how\n", + "walkers are initialized, how many walkers are used, and which proposal\n", + "move drives the ensemble.\n", + "\n", + "The default emcee proposal is the stretch move. This tutorial uses the\n", + "differential-evolution move instead, because it mixes better for the\n", + "strongly correlated LBCO/HRPT parameters. The walker count is kept\n", + "below the default to keep runtime close to the DREAM tutorial while\n", + "retaining good convergence diagnostics for this five-parameter example." ] }, { "cell_type": "code", "execution_count": null, - "id": "36", + "id": "45", "metadata": {}, "outputs": [], "source": [ - "project.display.posterior.distribution()" + "project.analysis.minimizer.show_supported()" ] }, { - "cell_type": "markdown", - "id": "37", + "cell_type": "code", + "execution_count": null, + "id": "46", "metadata": {}, + "outputs": [], "source": [ - "The posterior predictive plot propagates the sampled parameter\n", - "uncertainty into the calculated diffraction pattern." + "project.analysis.minimizer.type = 'emcee'" ] }, { "cell_type": "code", "execution_count": null, - "id": "38", + "id": "47", "metadata": {}, "outputs": [], "source": [ - "project.display.posterior.predictive(expt_name='hrpt')" + "project.analysis.minimizer.sampling_steps = 100 # lower than the default 5000\n", + "project.analysis.minimizer.burn_in_steps = 20 # lower than the default 1000\n", + "project.analysis.minimizer.population_size = 16 # lower than the default 32" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "48", + "metadata": {}, + "outputs": [], + "source": [ + "project.analysis.fit()" ] }, { "cell_type": "markdown", - "id": "39", + "id": "49", "metadata": {}, "source": [ - "## Save the Sampled Project\n", + "## Step 7: Inspect Bayesian Results\n", "\n", - "Saving persists both the analysis state and the emcee chain sidecar so\n", - "the same chain can be resumed later." + "The fit-results display now includes sampler settings, convergence\n", + "diagnostics, committed parameter values, and posterior summary\n", + "statistics." ] }, { "cell_type": "code", "execution_count": null, - "id": "40", + "id": "50", "metadata": {}, "outputs": [], "source": [ - "project.save()" + "project.display.fit.results()" ] }, { "cell_type": "markdown", - "id": "41", + "id": "51", "metadata": {}, "source": [ - "## Resume emcee Sampling\n", + "The correlation and posterior-pair plots are complementary:\n", "\n", - "Resume from the saved backend and append 500 more emcee steps to the\n", - "existing chain." + "- `plot_param_correlations` summarizes pairwise structure in a compact\n", + " matrix.\n", + "- `plot_posterior_pairs` shows marginal densities on the diagonal and\n", + " posterior contours off-diagonal. In this tutorial its title also\n", + " reminds you that the display region follows the `±4 × uncertainty`\n", + " bounds defined above, while numeric subplot ranges are omitted to\n", + " keep the grid readable." ] }, { "cell_type": "code", "execution_count": null, - "id": "42", + "id": "52", "metadata": {}, "outputs": [], "source": [ - "project.analysis.fit(resume=True, extra_steps=500)" + "project.display.fit.correlations()" ] }, { "cell_type": "code", "execution_count": null, - "id": "43", + "id": "53", "metadata": {}, "outputs": [], "source": [ - "project.display.fit.results()" + "project.display.posterior.pairs()" ] }, { "cell_type": "markdown", - "id": "44", + "id": "54", "metadata": {}, "source": [ - "## Inspect the Resumed Posterior\n", - "\n", - "After resume, the posterior plots use the extended chain." + "The one-dimensional posterior distributions below make it easier to\n", + "inspect individual parameters in isolation, including asymmetry or\n", + "multimodality." ] }, { "cell_type": "code", "execution_count": null, - "id": "45", + "id": "55", "metadata": {}, "outputs": [], "source": [ "project.display.posterior.distribution()" ] }, + { + "cell_type": "markdown", + "id": "56", + "metadata": {}, + "source": [ + "Finally, the posterior predictive plot propagates the sampled parameter\n", + "uncertainty into the calculated diffraction pattern. Comparing this to\n", + "the zoomed measured-vs-calculated view helps assess whether the sampled\n", + "model family explains the data in the region of interest." + ] + }, { "cell_type": "code", "execution_count": null, - "id": "46", + "id": "57", "metadata": {}, "outputs": [], "source": [ "project.display.posterior.predictive(expt_name='hrpt')" ] + }, + { + "cell_type": "markdown", + "id": "58", + "metadata": {}, + "source": [ + "A final zoomed measured-vs-calculated plot is useful for checking how\n", + "the posterior-supported model behaves in a narrow region of the pattern\n", + "after the Bayesian run." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "59", + "metadata": {}, + "outputs": [], + "source": [ + "project.display.posterior.predictive(expt_name='hrpt', x_min=92, x_max=93)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "60", + "metadata": {}, + "outputs": [], + "source": [] } ], "metadata": { diff --git a/docs/docs/tutorials/ed-26.ipynb b/docs/docs/tutorials/ed-26.ipynb new file mode 100644 index 000000000..1740dfcdb --- /dev/null +++ b/docs/docs/tutorials/ed-26.ipynb @@ -0,0 +1,307 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "id": "0", + "metadata": { + "tags": [ + "hide-in-docs" + ] + }, + "outputs": [], + "source": [ + "# Check whether easydiffraction is installed; install it if needed.\n", + "# Required for remote environments such as Google Colab.\n", + "import importlib.util\n", + "\n", + "if importlib.util.find_spec('easydiffraction') is None:\n", + " %pip install easydiffraction" + ] + }, + { + "cell_type": "markdown", + "id": "1", + "metadata": {}, + "source": [ + "# Bayesian Analysis Resume (`emcee`): LBCO, HRPT\n", + "\n", + "This tutorial shows how to reopen the Bayesian project created previously,\n", + "inspect the saved fit results and then run more sampling steps to extend the existing chain.\n", + "Resuming only works with EMCEE because the current BUMPS-DREAM implementation does not support\n", + "saving and resuming its state.\n", + "\n", + "This workflow is useful when:\n", + "- the initial sampling run has not yet converged and more steps are needed,\n", + "- the initial sampling run has converged but more steps are desired for better posterior resolution,\n", + "- the initial sampling run has converged but the posterior plots have not yet been inspected and the user wants to see the plots before deciding whether to run more steps.\n", + "\n", + "The workflow uses the same La0.5Ba0.5CoO3 powder diffraction example\n", + "as the DREAM Bayesian tutorial:\n", + "\n", + "- run a short local refinement,\n", + "- derive finite fit bounds for the sampled parameters,\n", + "- switch to emcee and sample the posterior,\n", + "- save the project with the emcee chain,\n", + "- resume the chain with additional steps,\n", + "- inspect posterior plots after each sampling stage." + ] + }, + { + "cell_type": "markdown", + "id": "2", + "metadata": {}, + "source": [ + "## Import Library" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3", + "metadata": {}, + "outputs": [], + "source": [ + "import easydiffraction as ed" + ] + }, + { + "cell_type": "markdown", + "id": "4", + "metadata": {}, + "source": [ + "## Download Saved Project\n", + "\n", + "The returned path points directly to the saved project directory with\n", + "the completed Bayesian fit and persisted posterior samples and plot\n", + "caches." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5", + "metadata": {}, + "outputs": [], + "source": [ + "project_dir = ed.download_data(id=38, destination='projects')" + ] + }, + { + "cell_type": "markdown", + "id": "6", + "metadata": {}, + "source": [ + "## Load the Saved Bayesian Project\n", + "\n", + "Loading restores the persisted fit state, posterior samples, and plot\n", + "caches. No new fit is launched in this tutorial." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "7", + "metadata": {}, + "outputs": [], + "source": [ + "project = ed.Project.load(project_dir)" + ] + }, + { + "cell_type": "markdown", + "id": "8", + "metadata": {}, + "source": [ + "## Review the Saved Fit Summary\n", + "\n", + "The fit summary reports the committed point estimate, sampler\n", + "settings, convergence diagnostics, and posterior parameter summaries\n", + "from the saved Bayesian run." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9", + "metadata": {}, + "outputs": [], + "source": [ + "project.display.fit.results()" + ] + }, + { + "cell_type": "markdown", + "id": "10", + "metadata": {}, + "source": [ + "## Show Correlations\n", + "\n", + "The correlation matrix is restored from the saved project state." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "11", + "metadata": {}, + "outputs": [], + "source": [ + "project.display.fit.correlations()" + ] + }, + { + "cell_type": "markdown", + "id": "12", + "metadata": {}, + "source": [ + "## Inspect Posterior Densities and Pair Structure\n", + "\n", + "The pair plot and one-dimensional posterior distributions now load\n", + "from the persisted caches generated when the Bayesian fit was saved." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "13", + "metadata": {}, + "outputs": [], + "source": [ + "project.display.posterior.pairs()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "14", + "metadata": {}, + "outputs": [], + "source": [ + "project.display.posterior.distribution()" + ] + }, + { + "cell_type": "markdown", + "id": "15", + "metadata": {}, + "source": [ + "## Plot Posterior Predictive Checks\n", + "\n", + "The posterior predictive view reuses the cached predictive summary\n", + "stored in the project rather than recalculating it on first display.\n", + "It overlays the 95% credible interval propagated from the posterior\n", + "samples." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "16", + "metadata": {}, + "outputs": [], + "source": [ + "project.display.posterior.predictive(expt_name='hrpt')" + ] + }, + { + "cell_type": "markdown", + "id": "17", + "metadata": {}, + "source": [ + "A zoomed view is useful for checking the propagated uncertainty in a\n", + "narrow region of the diffraction pattern." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "18", + "metadata": {}, + "outputs": [], + "source": [ + "project.display.posterior.predictive(expt_name='hrpt', x_min=92, x_max=93)" + ] + }, + { + "cell_type": "markdown", + "id": "19", + "metadata": {}, + "source": [ + "## Resume emcee Sampling\n", + "\n", + "Resume from the saved backend and append 100 more emcee steps to the\n", + "existing chain. We use only 100 steps here to keep the tutorial fast, but in practice you would typically run more steps to ensure convergence and better posterior resolution." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "20", + "metadata": {}, + "outputs": [], + "source": [ + "project.analysis.fit(resume=True, extra_steps=100)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "21", + "metadata": {}, + "outputs": [], + "source": [ + "project.display.fit.results()" + ] + }, + { + "cell_type": "markdown", + "id": "22", + "metadata": {}, + "source": [ + "## Inspect the Resumed Posterior\n", + "\n", + "After resume, the posterior plots use the extended chain." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "23", + "metadata": {}, + "outputs": [], + "source": [ + "project.display.posterior.pairs()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "24", + "metadata": {}, + "outputs": [], + "source": [ + "project.display.posterior.distribution()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "25", + "metadata": {}, + "outputs": [], + "source": [ + "project.display.posterior.predictive(expt_name='hrpt', x_min=92, x_max=93)" + ] + } + ], + "metadata": { + "jupytext": { + "cell_metadata_filter": "-all", + "main_language": "python", + "notebook_metadata_filter": "-all" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/docs/tutorials/ed-5.ipynb b/docs/docs/tutorials/ed-5.ipynb index 5dd0e7a06..6971955d7 100644 --- a/docs/docs/tutorials/ed-5.ipynb +++ b/docs/docs/tutorials/ed-5.ipynb @@ -371,9 +371,19 @@ ] }, { - "cell_type": "markdown", + "cell_type": "code", + "execution_count": null, "id": "29", "metadata": {}, + "outputs": [], + "source": [ + "project.save_as('projects/cosio_d20')" + ] + }, + { + "cell_type": "markdown", + "id": "30", + "metadata": {}, "source": [ "#### Add Structure" ] @@ -381,7 +391,7 @@ { "cell_type": "code", "execution_count": null, - "id": "30", + "id": "31", "metadata": {}, "outputs": [], "source": [ @@ -390,7 +400,7 @@ }, { "cell_type": "markdown", - "id": "31", + "id": "32", "metadata": {}, "source": [ "#### Add Experiment" @@ -399,7 +409,7 @@ { "cell_type": "code", "execution_count": null, - "id": "32", + "id": "33", "metadata": {}, "outputs": [], "source": [ @@ -408,7 +418,7 @@ }, { "cell_type": "markdown", - "id": "33", + "id": "34", "metadata": {}, "source": [ "## Perform Analysis\n", @@ -422,7 +432,7 @@ { "cell_type": "code", "execution_count": null, - "id": "34", + "id": "35", "metadata": {}, "outputs": [], "source": [ @@ -432,7 +442,7 @@ { "cell_type": "code", "execution_count": null, - "id": "35", + "id": "36", "metadata": {}, "outputs": [], "source": [ @@ -441,7 +451,7 @@ }, { "cell_type": "markdown", - "id": "36", + "id": "37", "metadata": {}, "source": [ "#### Set Free Parameters" @@ -450,7 +460,7 @@ { "cell_type": "code", "execution_count": null, - "id": "37", + "id": "38", "metadata": {}, "outputs": [], "source": [ @@ -473,7 +483,7 @@ { "cell_type": "code", "execution_count": null, - "id": "38", + "id": "39", "metadata": {}, "outputs": [], "source": [ @@ -494,7 +504,7 @@ }, { "cell_type": "markdown", - "id": "39", + "id": "40", "metadata": {}, "source": [ "Show free parameters after selection." @@ -503,7 +513,7 @@ { "cell_type": "code", "execution_count": null, - "id": "40", + "id": "41", "metadata": {}, "outputs": [], "source": [ @@ -512,7 +522,7 @@ }, { "cell_type": "markdown", - "id": "41", + "id": "42", "metadata": {}, "source": [ "#### Set Constraints\n", @@ -523,7 +533,7 @@ { "cell_type": "code", "execution_count": null, - "id": "42", + "id": "43", "metadata": {}, "outputs": [], "source": [ @@ -539,7 +549,7 @@ }, { "cell_type": "markdown", - "id": "43", + "id": "44", "metadata": {}, "source": [ "Set constraints." @@ -548,7 +558,7 @@ { "cell_type": "code", "execution_count": null, - "id": "44", + "id": "45", "metadata": { "lines_to_next_cell": 2 }, @@ -559,7 +569,7 @@ }, { "cell_type": "markdown", - "id": "45", + "id": "46", "metadata": {}, "source": [ "#### Run Fitting" @@ -568,7 +578,7 @@ { "cell_type": "code", "execution_count": null, - "id": "46", + "id": "47", "metadata": {}, "outputs": [], "source": [ @@ -578,7 +588,7 @@ { "cell_type": "code", "execution_count": null, - "id": "47", + "id": "48", "metadata": {}, "outputs": [], "source": [ @@ -588,7 +598,7 @@ { "cell_type": "code", "execution_count": null, - "id": "48", + "id": "49", "metadata": {}, "outputs": [], "source": [ @@ -597,7 +607,7 @@ }, { "cell_type": "markdown", - "id": "49", + "id": "50", "metadata": {}, "source": [ "#### Plot Measured vs Calculated" @@ -606,7 +616,7 @@ { "cell_type": "code", "execution_count": null, - "id": "50", + "id": "51", "metadata": {}, "outputs": [], "source": [ @@ -616,7 +626,7 @@ { "cell_type": "code", "execution_count": null, - "id": "51", + "id": "52", "metadata": {}, "outputs": [], "source": [ @@ -625,7 +635,7 @@ }, { "cell_type": "markdown", - "id": "52", + "id": "53", "metadata": {}, "source": [ "## Summary\n", @@ -635,7 +645,7 @@ }, { "cell_type": "markdown", - "id": "53", + "id": "54", "metadata": {}, "source": [ "#### Show Project Summary" @@ -644,7 +654,7 @@ { "cell_type": "code", "execution_count": null, - "id": "54", + "id": "55", "metadata": {}, "outputs": [], "source": [ diff --git a/docs/docs/tutorials/index.md b/docs/docs/tutorials/index.md index 13ed5d11a..7ec5bf301 100644 --- a/docs/docs/tutorials/index.md +++ b/docs/docs/tutorials/index.md @@ -46,10 +46,11 @@ The tutorials are organized into the following categories: - [Co2SiO4 Sequential Fit](ed-23.ipynb) – Resumes a sequential refinement from an existing `analysis/results.csv` after an incomplete previous run. -- [LBCO Bayesian Display](ed-24.ipynb) – Shows how to load the saved - project after a Bayesian analysis and inspect the persisted fit - summary, correlation matrix, posterior plots, and predictive checks - without rerunning MCMC sampling. + +See also under [Bayesian Analysis](#bayesian-analysis): +[LBCO Bayesian Display (`bumps-dream`)](ed-24.ipynb) and +[LBCO Bayesian Resume (`emcee`)](ed-26.ipynb) — both load saved +projects containing Bayesian fit state. ## Powder Diffraction @@ -76,7 +77,7 @@ The tutorials are organized into the following categories: refinement of Taurine using time-of-flight neutron single crystal diffraction data from SENJU at J-PARC. -## Pair Distribution Function (PDF) +## Pair Distribution Function - [Ni `pd-neut-cwl`](ed-10.ipynb) – Demonstrates a PDF analysis of Ni using data collected from a constant wavelength neutron powder @@ -87,7 +88,7 @@ The tutorials are organized into the following categories: - [NaCl `pd-xray`](ed-12.ipynb) – Demonstrates a PDF analysis of NaCl using data collected from an X-ray powder diffraction experiment. -## Multi-Structure & Multi-Experiment Refinement +## Multiple Data Blocks - [PbSO4 NPD+XRD](ed-4.ipynb) – Joint fit of PbSO4 using X-ray and neutron constant wavelength powder diffraction data. @@ -109,20 +110,33 @@ The tutorials are organized into the following categories: ## Bayesian Analysis -- [LBCO Bayesian](ed-21.ipynb) – Demonstrates how to perform a Bayesian - analysis of the La0.5Ba0.5CoO3 crystal structure using constant - wavelength neutron powder diffraction data from HRPT at PSI. This - tutorial covers the use of Markov Chain Monte Carlo (MCMC) sampling to - explore the posterior distribution of the refined parameters, - providing insights into parameter uncertainties and correlations. -- [LBCO emcee Resume](ed-25.ipynb) – Demonstrates how to run Bayesian - sampling with the emcee minimizer, save the project with its chain, - resume sampling with additional steps, and inspect the posterior - before and after resume. -- [Tb2TiO7 Bayesian](ed-22.ipynb) – Another example of a Bayesian - analysis. This tutorial focuses on the Tb2TiO7 crystal structure using - constant wavelength neutron single crystal diffraction data from HEiDi - at FRM II. Similar to the LBCO Bayesian tutorial, it covers the use of +- [LBCO Bayesian (`bumps-dream`)](ed-21.ipynb) – Demonstrates how to + perform a Bayesian analysis of the La0.5Ba0.5CoO3 crystal structure + using constant wavelength neutron powder diffraction data from HRPT + at PSI. Covers the use of Markov Chain Monte Carlo (MCMC) sampling + with the bumps-DREAM minimizer to explore the posterior distribution + of the refined parameters, providing insights into parameter + uncertainties and correlations. +- [LBCO Bayesian Display (`bumps-dream`)](ed-24.ipynb) – Shows how to + reopen the saved Bayesian project produced by the LBCO Bayesian + tutorial and inspect persisted fit summaries, correlation matrix, + posterior distribution plots, and predictive checks — without + rerunning MCMC sampling. +- [LBCO Bayesian (`emcee`)](ed-25.ipynb) – Two-stage workflow on the + LBCO HRPT dataset: first a quick local refinement to obtain a point + estimate and uncertainties, then full posterior sampling with the + emcee minimizer. Covers credible intervals, parameter correlations, + and propagation of uncertainty into the calculated diffraction + pattern. +- [LBCO Bayesian Resume (`emcee`)](ed-26.ipynb) – Loads a Bayesian + project that already contains an emcee chain, inspects the posterior, + and resumes sampling with additional steps. The full project state + (parameters, chain, plot caches) round-trips through disk. Resuming + is currently supported only for emcee, not for bumps-DREAM. +- [Tb2TiO7 Bayesian (`bumps-dream`)](ed-22.ipynb) – Another example of + a Bayesian analysis, focused on the Tb2TiO7 crystal structure using + constant wavelength neutron single crystal diffraction data from + HEiDi at FRM II. Similar to the LBCO Bayesian tutorial, it covers MCMC sampling to explore the posterior distribution of the refined parameters, providing insights into parameter uncertainties and correlations in the context of single crystal diffraction data. From 08b7cccc27b6681d0e519a46a9c336dc4a4b3ad8 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 22:00:06 +0200 Subject: [PATCH 39/65] Share emcee minimizer defaults --- .../analysis/categories/minimizer/emcee.py | 18 ++++++------- .../analysis/minimizers/emcee.py | 26 ++++++++----------- .../analysis/minimizers/emcee_defaults.py | 26 +++++++++++++++++++ 3 files changed, 45 insertions(+), 25 deletions(-) create mode 100644 src/easydiffraction/analysis/minimizers/emcee_defaults.py diff --git a/src/easydiffraction/analysis/categories/minimizer/emcee.py b/src/easydiffraction/analysis/categories/minimizer/emcee.py index 651c5be7b..d988241e7 100644 --- a/src/easydiffraction/analysis/categories/minimizer/emcee.py +++ b/src/easydiffraction/analysis/categories/minimizer/emcee.py @@ -8,6 +8,14 @@ from easydiffraction.analysis.categories.minimizer.bayesian_base import BayesianMinimizerBase from easydiffraction.analysis.categories.minimizer.factory import MinimizerCategoryFactory +from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_INITIALIZATION_METHOD +from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_NBURN as DEFAULT_BURN_IN_STEPS +from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_NSTEPS as DEFAULT_SAMPLING_STEPS +from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_NWALKERS as DEFAULT_POPULATION_SIZE +from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_PARALLEL_WORKERS +from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_PROPOSAL_MOVES +from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_THIN as DEFAULT_THINNING_INTERVAL +from easydiffraction.analysis.minimizers.emcee_defaults import SUPPORTED_PROPOSAL_MOVES from easydiffraction.analysis.minimizers.enums import InitializationMethodEnum from easydiffraction.analysis.minimizers.enums import MinimizerTypeEnum from easydiffraction.core.metadata import TypeInfo @@ -16,16 +24,6 @@ from easydiffraction.core.variable import StringDescriptor from easydiffraction.io.cif.handler import CifHandler -DEFAULT_SAMPLING_STEPS = 5000 -DEFAULT_BURN_IN_STEPS = 1000 -DEFAULT_THINNING_INTERVAL = 1 -DEFAULT_POPULATION_SIZE = 32 -DEFAULT_PARALLEL_WORKERS = 0 -DEFAULT_INITIALIZATION_METHOD = InitializationMethodEnum.BALL -DEFAULT_PROPOSAL_MOVES = 'de' -SUPPORTED_PROPOSAL_MOVES = ('stretch', 'de', 'de_snooker', 'walk') - - @MinimizerCategoryFactory.register class EmceeMinimizer(BayesianMinimizerBase): """Persisted settings for the emcee minimizer.""" diff --git a/src/easydiffraction/analysis/minimizers/emcee.py b/src/easydiffraction/analysis/minimizers/emcee.py index d44002826..98a38c982 100644 --- a/src/easydiffraction/analysis/minimizers/emcee.py +++ b/src/easydiffraction/analysis/minimizers/emcee.py @@ -23,31 +23,27 @@ from easydiffraction.analysis.fit_helpers.metrics import calculate_reduced_chi_square from easydiffraction.analysis.fit_helpers.tracking import SamplerProgressUpdate from easydiffraction.analysis.minimizers.base import MinimizerBase +from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_INITIALIZATION_METHOD +from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_METHOD +from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_NBURN +from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_NSTEPS +from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_NWALKERS +from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_PARALLEL_WORKERS +from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_PROPOSAL_MOVES +from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_THIN +from easydiffraction.analysis.minimizers.emcee_defaults import MAX_RANDOM_SEED +from easydiffraction.analysis.minimizers.emcee_defaults import SUPPORTED_INITIALIZATION_METHOD_SET +from easydiffraction.analysis.minimizers.emcee_defaults import SUPPORTED_INITIALIZATION_METHODS from easydiffraction.analysis.minimizers.enums import InitializationMethodEnum from easydiffraction.analysis.minimizers.enums import MinimizerTypeEnum from easydiffraction.analysis.minimizers.factory import MinimizerFactory from easydiffraction.core.metadata import TypeInfo from easydiffraction.utils.enums import VerbosityEnum -DEFAULT_METHOD = 'de' -DEFAULT_NSTEPS = 5000 -DEFAULT_NBURN = 1000 -DEFAULT_THIN = 1 -DEFAULT_NWALKERS = 32 -DEFAULT_PARALLEL_WORKERS = 0 -DEFAULT_INITIALIZATION_METHOD = InitializationMethodEnum.BALL -DEFAULT_PROPOSAL_MOVES = 'de' -MAX_RANDOM_SEED = int(np.iinfo(np.uint32).max) EMCEE_CHAIN_GROUP = 'emcee_chain' EMCEE_FAILURES = (ArithmeticError, RuntimeError, TypeError, ValueError) EMCEE_SAMPLE_ARRAY_NDIM = 3 TOTAL_PROGRESS_POINTS = 25 -SUPPORTED_INITIALIZATION_METHODS = ( - InitializationMethodEnum.BALL, - InitializationMethodEnum.UNIFORM, - InitializationMethodEnum.PRIOR, -) -SUPPORTED_INITIALIZATION_METHOD_SET = frozenset(SUPPORTED_INITIALIZATION_METHODS) if TYPE_CHECKING: from collections.abc import Callable diff --git a/src/easydiffraction/analysis/minimizers/emcee_defaults.py b/src/easydiffraction/analysis/minimizers/emcee_defaults.py new file mode 100644 index 000000000..6f590660f --- /dev/null +++ b/src/easydiffraction/analysis/minimizers/emcee_defaults.py @@ -0,0 +1,26 @@ +# SPDX-FileCopyrightText: 2026 EasyScience contributors +# SPDX-License-Identifier: BSD-3-Clause +"""Shared defaults for the emcee minimizer.""" + +from __future__ import annotations + +import numpy as np + +from easydiffraction.analysis.minimizers.enums import InitializationMethodEnum + +DEFAULT_METHOD = 'de' +DEFAULT_NSTEPS = 5000 +DEFAULT_NBURN = 1000 +DEFAULT_THIN = 1 +DEFAULT_NWALKERS = 32 +DEFAULT_PARALLEL_WORKERS = 0 +DEFAULT_INITIALIZATION_METHOD = InitializationMethodEnum.BALL +DEFAULT_PROPOSAL_MOVES = 'de' +MAX_RANDOM_SEED = int(np.iinfo(np.uint32).max) +SUPPORTED_PROPOSAL_MOVES = ('stretch', 'de', 'de_snooker', 'walk') +SUPPORTED_INITIALIZATION_METHODS = ( + InitializationMethodEnum.BALL, + InitializationMethodEnum.UNIFORM, + InitializationMethodEnum.PRIOR, +) +SUPPORTED_INITIALIZATION_METHOD_SET = frozenset(SUPPORTED_INITIALIZATION_METHODS) From 3a3f5a4d06716493931524332140fbc727f65b4b Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 22:00:33 +0200 Subject: [PATCH 40/65] Remove unused sidecar warning wrapper --- src/easydiffraction/io/results_sidecar.py | 5 ----- 1 file changed, 5 deletions(-) diff --git a/src/easydiffraction/io/results_sidecar.py b/src/easydiffraction/io/results_sidecar.py index 79abeceff..b570a26b5 100644 --- a/src/easydiffraction/io/results_sidecar.py +++ b/src/easydiffraction/io/results_sidecar.py @@ -71,11 +71,6 @@ def _warn_existing_sidecar_overwrite(sidecar_path: Path) -> None: ) -def warn_analysis_results_sidecar_overwrite(*, analysis_dir: Path) -> None: - """Warn when a new fit will overwrite existing sidecar arrays.""" - _warn_existing_sidecar_overwrite(_sidecar_path(analysis_dir=analysis_dir)) - - def prepare_analysis_results_sidecar_for_new_fit(*, analysis_dir: Path) -> None: """Warn and remove the results sidecar before a fresh fit starts.""" sidecar_path = _sidecar_path(analysis_dir=analysis_dir) From f7ff71fae09ffa8d59fef509520d1bfe3d02d765 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 22:00:54 +0200 Subject: [PATCH 41/65] Keep DREAM parallel worker syncing --- src/easydiffraction/analysis/categories/minimizer/base.py | 5 +---- src/easydiffraction/analysis/categories/minimizer/emcee.py | 4 ++++ 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/src/easydiffraction/analysis/categories/minimizer/base.py b/src/easydiffraction/analysis/categories/minimizer/base.py index a57696ef4..8d0130bcd 100644 --- a/src/easydiffraction/analysis/categories/minimizer/base.py +++ b/src/easydiffraction/analysis/categories/minimizer/base.py @@ -24,10 +24,7 @@ class MinimizerCategoryBase(CategoryItem, SwitchableCategoryBase): _owner_attr_name = 'minimizer' _swap_method_name = '_swap_minimizer' _native_key_map: ClassVar[dict[str, str]] = {} - _engine_sync_skip_keys: ClassVar[frozenset[str]] = frozenset({ - 'random_seed', - 'parallel_workers', - }) + _engine_sync_skip_keys: ClassVar[frozenset[str]] = frozenset({'random_seed'}) _setting_descriptor_names: ClassVar[tuple[str, ...]] = () _result_descriptor_names: ClassVar[tuple[str, ...]] = () _fit_result_class: ClassVar[type[FitResultBase]] = FitResultBase diff --git a/src/easydiffraction/analysis/categories/minimizer/emcee.py b/src/easydiffraction/analysis/categories/minimizer/emcee.py index d988241e7..e5f841b1c 100644 --- a/src/easydiffraction/analysis/categories/minimizer/emcee.py +++ b/src/easydiffraction/analysis/categories/minimizer/emcee.py @@ -46,6 +46,10 @@ class EmceeMinimizer(BayesianMinimizerBase): 'random_seed': 'random_seed', 'proposal_moves': 'proposal_moves', } + _engine_sync_skip_keys: ClassVar[frozenset[str]] = frozenset({ + 'random_seed', + 'parallel_workers', + }) _setting_descriptor_names: ClassVar[tuple[str, ...]] = ( *BayesianMinimizerBase._setting_descriptor_names, 'proposal_moves', From 063f9933e2158d5c61bd043f9f6d515103661b99 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 22:01:57 +0200 Subject: [PATCH 42/65] Use native emcee sampler setting keys --- src/easydiffraction/analysis/analysis.py | 17 ++++++++++------- .../analysis/minimizers/emcee.py | 9 ++------- 2 files changed, 12 insertions(+), 14 deletions(-) diff --git a/src/easydiffraction/analysis/analysis.py b/src/easydiffraction/analysis/analysis.py index cca6fd244..d39e6b5ec 100644 --- a/src/easydiffraction/analysis/analysis.py +++ b/src/easydiffraction/analysis/analysis.py @@ -888,12 +888,15 @@ def _restored_bayesian_sampler_settings( """Return display settings for restored Bayesian results.""" if self.minimizer.type == MinimizerTypeEnum.EMCEE.value: restored_settings = { - 'steps': self._int_sampler_setting(sampler_settings, 'nsteps'), - 'burn': self._int_sampler_setting(sampler_settings, 'nburn'), + 'nsteps': self._int_sampler_setting(sampler_settings, 'nsteps'), + 'nburn': self._int_sampler_setting(sampler_settings, 'nburn'), 'thin': self._int_sampler_setting(sampler_settings, 'thin'), - 'pop': self._int_sampler_setting(sampler_settings, 'nwalkers'), - 'parallel': self._int_sampler_setting(sampler_settings, 'parallel_workers'), - 'init': str(sampler_settings.get('initialization_method', '')), + 'nwalkers': self._int_sampler_setting(sampler_settings, 'nwalkers'), + 'parallel_workers': self._int_sampler_setting( + sampler_settings, + 'parallel_workers', + ), + 'initialization_method': str(sampler_settings.get('initialization_method', '')), 'proposal_moves': str(sampler_settings.get('proposal_moves', '')), 'random_seed': random_seed, } @@ -944,8 +947,8 @@ def _sampler_sample_count( n_parameters: int, ) -> int: """Return restored total sampled scalar count.""" - steps = int(sampler_settings.get('steps') or 0) - population = int(sampler_settings.get('pop') or 0) + steps = int(sampler_settings.get('steps') or sampler_settings.get('nsteps') or 0) + population = int(sampler_settings.get('pop') or sampler_settings.get('nwalkers') or 0) return max(0, steps) * max(0, population) * max(0, int(n_parameters)) def help(self) -> None: diff --git a/src/easydiffraction/analysis/minimizers/emcee.py b/src/easydiffraction/analysis/minimizers/emcee.py index 98a38c982..a6c81c4eb 100644 --- a/src/easydiffraction/analysis/minimizers/emcee.py +++ b/src/easydiffraction/analysis/minimizers/emcee.py @@ -968,13 +968,7 @@ def _sampler_settings( samples = self.nsteps * self.nwalkers * n_parameters return { 'random_seed': int(random_seed), - 'steps': int(self.nsteps), - 'burn': int(self.nburn), 'thin': int(self.thin), - 'pop': int(self.nwalkers), - 'parallel': int(self.parallel_workers), - 'init': self.initialization_method.value, - 'proposal_moves': self.proposal_moves, 'samples': int(samples), 'total_steps': int(total_steps), 'nsteps': int(self.nsteps), @@ -982,6 +976,7 @@ def _sampler_settings( 'nwalkers': int(self.nwalkers), 'parallel_workers': int(self.parallel_workers), 'initialization_method': self.initialization_method.value, + 'proposal_moves': self.proposal_moves, } @staticmethod @@ -1271,7 +1266,7 @@ def _build_fit_results( best_log_posterior=getattr(raw_result, 'best_log_posterior', None), ) fit_results.message = getattr(raw_result, 'message', '') - fit_results.iterations = int(fit_results.sampler_settings.get('steps', self.nsteps)) + fit_results.iterations = int(fit_results.sampler_settings.get('nsteps', self.nsteps)) fit_results.result = raw_result return fit_results From 20167ebd80de043eb987059d227c39e57a167c40 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 22:02:46 +0200 Subject: [PATCH 43/65] Document emcee tutorial split --- docs/dev/plans/emcee-minimizer.md | 38 +++++++++++++++---------------- 1 file changed, 18 insertions(+), 20 deletions(-) diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index 3380bed32..295f3ada4 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -652,7 +652,7 @@ Mark `[x]` as each step lands. Commit: `Route emcee posterior through fit_result and sidecar` -- [x] **P1.7 — Add `ed-25.py` tutorial.** Verify first that +- [x] **P1.7 — Add emcee tutorials.** Verify first that `docs/docs/tutorials/ed-25.py` is unused. `ed-23.py` is the "Co2SiO4 Sequential Fit" tutorial and `ed-24.py` is the "LBCO Bayesian Display" tutorial — do **not** overwrite either. If @@ -660,23 +660,20 @@ Mark `[x]` as each step lands. next free integer slot and adjust the file name + references below to match. - New notebook source at `docs/docs/tutorials/ed-25.py` - covering: + New notebook sources at `docs/docs/tutorials/ed-25.py` and + `docs/docs/tutorials/ed-26.py` covering: - - `project.analysis.minimizer.type = 'emcee'` (post-switchable - syntax). - - `project.analysis.minimizer.sampling_steps = 1000` (small for - tutorial speed). - - `project.analysis.fit()` and a posterior plot. - - `project.save()`. - - `project.analysis.fit(resume=True, extra_steps=500)` continues the - chain. - - Final posterior plot after resume. + - `ed-25.py`: `project.analysis.minimizer.type = 'emcee'` + (post-switchable syntax), tutorial-sized sampler settings, + `project.analysis.fit()`, posterior plots, and `project.save()`. + - `ed-26.py`: reopening the saved project, displaying restored + Bayesian results, and `project.analysis.fit(resume=True, extra_steps=500)` + to continue the chain. Update the docs navigation in the same step: - Add an entry under "MCMC / Bayesian" (or the appropriate section) in [`docs/docs/tutorials/index.md`](../../docs/tutorials/index.md) - pointing at `ed-25.ipynb`. + pointing at `ed-25.ipynb` and `ed-26.ipynb`. - Add a navigation entry under the matching section in [`docs/mkdocs.yml`](../../../docs/mkdocs.yml). @@ -686,14 +683,15 @@ Mark `[x]` as each step lands. ``` test -f docs/docs/tutorials/ed-25.py - git grep -nE '\banalysis\.minimizer_type\b|\bminimizer\.runtime_seconds\b|\bminimizer\.gelman_rubin_max\b' docs/docs/tutorials/ed-25.py - git grep -n 'ed-25' docs/docs/tutorials/index.md docs/mkdocs.yml + test -f docs/docs/tutorials/ed-26.py + git grep -nE '\banalysis\.minimizer_type\b|\bminimizer\.runtime_seconds\b|\bminimizer\.gelman_rubin_max\b' docs/docs/tutorials/ed-25.py docs/docs/tutorials/ed-26.py + git grep -n 'ed-2[56]' docs/docs/tutorials/index.md docs/mkdocs.yml ``` - The first must be true; the second must be empty; the third must - return at least one hit in each file. + The first two must be true; the third must be empty; the fourth must + return at least one hit for both tutorial files. - Commit: `Add ed-25 emcee tutorial` + Commit: `Introduce emcee minimizer tutorials` - [x] **P1.8 — Phase 1 review gate.** No code change. Stop and request user review. After approval, proceed to Phase 2. @@ -806,5 +804,5 @@ single-file affair. Plots, parameter posteriors, and tables work the same as for DREAM, so switching between samplers to cross-check results is straightforward. -A new tutorial (`ed-25`) walks through a short run, saving the project, -and resuming for additional steps. +New tutorials walk through a short run (`ed-25`) and reopening the +saved project to resume for additional steps (`ed-26`). From c3c023d3d54ca03379edb39d1da9b2a1ca80581a Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 22:03:11 +0200 Subject: [PATCH 44/65] Use eager emcee sidecar imports --- src/easydiffraction/analysis/analysis.py | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/src/easydiffraction/analysis/analysis.py b/src/easydiffraction/analysis/analysis.py index d39e6b5ec..b45a9f9ac 100644 --- a/src/easydiffraction/analysis/analysis.py +++ b/src/easydiffraction/analysis/analysis.py @@ -8,6 +8,7 @@ from pathlib import Path from typing import TYPE_CHECKING +import h5py import numpy as np import pandas as pd @@ -37,6 +38,7 @@ from easydiffraction.analysis.fit_helpers.bayesian import posterior_predictive_cache_key from easydiffraction.analysis.fit_helpers.reporting import FitResults from easydiffraction.analysis.fitting import Fitter +from easydiffraction.analysis.minimizers.emcee import EMCEE_CHAIN_GROUP from easydiffraction.analysis.minimizers.enums import MinimizerTypeEnum from easydiffraction.core.category_owner import CategoryOwner from easydiffraction.core.guard import _apply_help_filter @@ -1218,12 +1220,6 @@ def _has_resumable_emcee_sidecar(self) -> bool: return False try: - import h5py # noqa: PLC0415 - - from easydiffraction.analysis.minimizers.emcee import ( # noqa: PLC0415 - EMCEE_CHAIN_GROUP, - ) - with h5py.File(sidecar_path, 'r') as handle: group = handle.get(EMCEE_CHAIN_GROUP) if group is None: From 028bfe817d2f455240ab76f8908cde84dc2bc31a Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 22:03:54 +0200 Subject: [PATCH 45/65] Clarify emcee integration test phase --- docs/dev/plans/emcee-minimizer.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index 295f3ada4..ae9efe9dd 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -227,14 +227,18 @@ When the matching open-issue is fully resolved, move it to the `HDFBackend`. - `tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py`. - `tests/unit/easydiffraction/analysis/minimizers/test_emcee.py`. -- `tests/integration/fitting/test_emcee.py` (cross-check vs DREAM on a - shared toy fit; assert posterior medians agree to within tolerance). -- `docs/docs/tutorials/ed-25.py` (emcee + resume tutorial). The next - free tutorial slot — `ed-23` is already the "Co2SiO4 Sequential Fit" +- `docs/docs/tutorials/ed-25.py` and `docs/docs/tutorials/ed-26.py` + (emcee fresh-run tutorial and resume/reload tutorial). The next free + tutorial slot — `ed-23` is already the "Co2SiO4 Sequential Fit" tutorial and `ed-24` is the "LBCO Bayesian Display" tutorial. Verify `ed-25.py` is unused before creating it at P1.7 start; bump if a newer slot is already occupied. +### Phase 2 verification files + +- `tests/integration/fitting/test_emcee.py` (cross-check vs DREAM on a + shared toy fit; assert posterior medians agree to within tolerance). + ### Modified - `src/easydiffraction/analysis/minimizers/enums.py` — add From ac229a48145738964656be531ad87b0fc65e8ea0 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 22:04:26 +0200 Subject: [PATCH 46/65] Remove redundant emcee initialization map --- src/easydiffraction/analysis/categories/minimizer/emcee.py | 6 ------ 1 file changed, 6 deletions(-) diff --git a/src/easydiffraction/analysis/categories/minimizer/emcee.py b/src/easydiffraction/analysis/categories/minimizer/emcee.py index e5f841b1c..af07f0cd4 100644 --- a/src/easydiffraction/analysis/categories/minimizer/emcee.py +++ b/src/easydiffraction/analysis/categories/minimizer/emcee.py @@ -59,12 +59,6 @@ class EmceeMinimizer(BayesianMinimizerBase): InitializationMethodEnum.UNIFORM, InitializationMethodEnum.PRIOR, ) - _native_initialization_methods: ClassVar[dict[InitializationMethodEnum, str]] = { - InitializationMethodEnum.BALL: InitializationMethodEnum.BALL.value, - InitializationMethodEnum.UNIFORM: InitializationMethodEnum.UNIFORM.value, - InitializationMethodEnum.PRIOR: InitializationMethodEnum.PRIOR.value, - } - type_info = TypeInfo( tag=MinimizerTypeEnum.EMCEE, description='emcee affine-invariant ensemble Bayesian sampling', From 6bbb3ddf210733814dfa7068cb7cd45b3dd6119f Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 22:07:21 +0200 Subject: [PATCH 47/65] Apply non-py formatting --- .../adrs/accepted/analysis-cif-fit-state.md | 4 +- .../accepted/fit-results-display-naming.md | 243 +++++++++--------- .../minimizer-category-consolidation.md | 22 +- docs/dev/issues/closed.md | 20 +- docs/dev/plans/emcee-minimizer.md | 22 +- docs/docs/tutorials/ed-22.py | 29 ++- docs/docs/tutorials/ed-25.py | 8 +- docs/docs/tutorials/index.md | 28 +- pyproject.toml | 2 +- 9 files changed, 183 insertions(+), 195 deletions(-) diff --git a/docs/dev/adrs/accepted/analysis-cif-fit-state.md b/docs/dev/adrs/accepted/analysis-cif-fit-state.md index 3bd4fa388..8b31da419 100644 --- a/docs/dev/adrs/accepted/analysis-cif-fit-state.md +++ b/docs/dev/adrs/accepted/analysis-cif-fit-state.md @@ -66,8 +66,8 @@ pre-fit scalar snapshots: - `start_value` - `start_uncertainty` -When any row has uncertainty-derived bounds, `_fit_parameter` also stores -the provenance field: +When any row has uncertainty-derived bounds, `_fit_parameter` also +stores the provenance field: - `fit_bounds_uncertainty_multiplier` diff --git a/docs/dev/adrs/accepted/fit-results-display-naming.md b/docs/dev/adrs/accepted/fit-results-display-naming.md index 5947565ef..5704102e1 100644 --- a/docs/dev/adrs/accepted/fit-results-display-naming.md +++ b/docs/dev/adrs/accepted/fit-results-display-naming.md @@ -31,23 +31,21 @@ Three problems: 1. `best posterior sample` (21 chars) is too wide for HTML / markdown layouts and forces the other columns into narrow space. -2. `uncertainty` is the column header in both LSQ and Bayesian - committed tables but the underlying quantities differ - (covariance-derived σ vs posterior SD). The display layer does not - annotate the difference. +2. `uncertainty` is the column header in both LSQ and Bayesian committed + tables but the underlying quantities differ (covariance-derived σ vs + posterior SD). The display layer does not annotate the difference. 3. LSQ's `fitted` and Bayesian's `best posterior sample` are conceptually parallel (the value committed back to the project) but - the headers do not signal that parallelism, complicating - side-by-side reading. + the headers do not signal that parallelism, complicating side-by-side + reading. Two conventions guide the cross-method naming choice: - **IUCr CIF** prefers the `_su` suffix (standard uncertainty); `_esd` (estimated standard deviation) is deprecated. -- **GUM** (Guide to the Expression of Uncertainty in Measurement) - treats Bayesian posterior SD and frequentist standard uncertainty as - the same physical quantity — 1σ of the inferred distribution of the - measurand. +- **GUM** (Guide to the Expression of Uncertainty in Measurement) treats + Bayesian posterior SD and frequentist standard uncertainty as the same + physical quantity — 1σ of the inferred distribution of the measurand. Both converge on `s.u.` as the appropriate cross-method label. @@ -55,18 +53,18 @@ Both converge on `s.u.` as the appropriate cross-method label. column headers or footnotes; [`iucr-cif-tag-alignment.md`](../suggestions/iucr-cif-tag-alignment.md) defines persisted CIF tag names but not display labels; -[`analysis-cif-fit-state.md`](analysis-cif-fit-state.md) defines -Python and CIF attribute names but not user-visible labels. Display -naming for fit-results tables is a real gap. +[`analysis-cif-fit-state.md`](analysis-cif-fit-state.md) defines Python +and CIF attribute names but not user-visible labels. Display naming for +fit-results tables is a real gap. ## Decision ### 1. Short headers paired with a footnote glossary Every fit-results table emits a glossary block immediately below the -table that expands the short column headers into one-line -descriptions. The footnote disambiguates per fitting mode so the -column header itself can stay short. +table that expands the short column headers into one-line descriptions. +The footnote disambiguates per fitting mode so the column header itself +can stay short. ### 2. Cross-method consistency where the physical quantity is the same @@ -132,9 +130,9 @@ ess bulk = bulk effective sample size (typically ≥ 400) - `📈 Fitted parameters:` → `📈 Refined parameters:` (IUCr-style "refinement" wording, also matches the cross-method `value` column). -- `📈 Committed parameters:` stays unchanged — the duality of - "committed values" vs "posterior distribution" is meaningful and - worth preserving on the Bayesian side. +- `📈 Committed parameters:` stays unchanged — the duality of "committed + values" vs "posterior distribution" is meaningful and worth preserving + on the Bayesian side. - `📊 Posterior parameter summaries:` → `📊 Posterior distribution:` (shorter and explicit about what the second table shows). @@ -150,115 +148,107 @@ table-header form: - Measured-vs-calculated plots: `measured`, `calculated`. Existing chart legends that describe plot **type** (e.g. -`Marginal density`, `Posterior contours`, `Posterior samples`) are -not parameter-value labels and are out of scope for this ADR. +`Marginal density`, `Posterior contours`, `Posterior samples`) are not +parameter-value labels and are out of scope for this ADR. ### 6. Internal attribute names unchanged `Parameter.value`, `Parameter.uncertainty`, -`Parameter.posterior_uncertainty`, and every persisted CIF tag stay -as they are. This ADR governs **display strings only**, not the -Python or CIF API. +`Parameter.posterior_uncertainty`, and every persisted CIF tag stay as +they are. This ADR governs **display strings only**, not the Python or +CIF API. ## Addendum (2026-05-25): Fit-results table replaces emoji-line summary -The original ADR specified two parameter-level tables for Bayesian -fits (`Committed parameters`, `Posterior distribution`) and one for -LSQ (`Refined parameters`), each below an emoji-line summary block +The original ADR specified two parameter-level tables for Bayesian fits +(`Committed parameters`, `Posterior distribution`) and one for LSQ +(`Refined parameters`), each below an emoji-line summary block (`✅ Success: True`, `📏 Goodness-of-fit (reduced χ²): 1.29`, …). In practice the emoji-line block grew long, mixed multi-value lines -(`📊 Convergence: status=passed, max_r_hat=1.004, …`) with -single-value lines (`📏 R-factor (Rf): 5.65%`), and split related -information across visually-different formats. +(`📊 Convergence: status=passed, max_r_hat=1.004, …`) with single-value +lines (`📏 R-factor (Rf): 5.65%`), and split related information across +visually-different formats. -The block is now rendered as **one additional 2-column table** per -fit method, sitting directly above the parameter tables: +The block is now rendered as **one additional 2-column table** per fit +method, sitting directly above the parameter tables: - LSQ: `📋 Least-squares fit results:` — title. - Bayesian: `📋 Bayesian fit results:` — title. -Column layout: `Metric | Value`, left/right alignment. Each row -carries one emoji-prefixed metric name in the first column and one -scalar value in the second. The previous `console.paragraph('Fit -results')` / `console.paragraph('Bayesian fit results')` section -header is dropped — the table title now signals the section. +Column layout: `Metric | Value`, left/right alignment. Each row carries +one emoji-prefixed metric name in the first column and one scalar value +in the second. The previous `console.paragraph('Fit results')` / +`console.paragraph('Bayesian fit results')` section header is dropped — +the table title now signals the section. Canonical row order (top-to-bottom): -1. `🧪 Minimizer` / `🧪 Sampler` — the minimizer.type string - (e.g. `lmfit (leastsq)`, `bumps (dream)`). -2. `✅ Overall status` — single shared value vocabulary: - `success` / `failed`. For LSQ this mirrors `FitResults.success`. - For Bayesian this is `success` only when the sampler completed - *and* convergence passed, else `failed`. Per-metric convergence - detail goes in rows 12–16 below. -3. `💬 Engine message` *(Bayesian, optional)* — the engine's - free-form status message, e.g. `DREAM sampling completed`. +1. `🧪 Minimizer` / `🧪 Sampler` — the minimizer.type string (e.g. + `lmfit (leastsq)`, `bumps (dream)`). +2. `✅ Overall status` — single shared value vocabulary: `success` / + `failed`. For LSQ this mirrors `FitResults.success`. For Bayesian + this is `success` only when the sampler completed _and_ convergence + passed, else `failed`. Per-metric convergence detail goes in rows + 12–16 below. +3. `💬 Engine message` _(Bayesian, optional)_ — the engine's free-form + status message, e.g. `DREAM sampling completed`. 4. `⏱️ Fitting time (seconds)` — `fitting_time`. -5. `🔁 Iterations` *(LSQ, optional)* — shown only when +5. `🔁 Iterations` _(LSQ, optional)_ — shown only when `FitResults.iterations > 0`. -6. `📏 Goodness-of-fit (reduced χ²)` — `reduced_chi_square`. -7–10. `📏 R-factor (Rf, %)`, `📏 R-factor squared (Rf², %)`, - `📏 Weighted R-factor (wR, %)`, `📏 Bragg R-factor (BR, %)` — - each row when the corresponding inputs are available. Units - appear in the metric name, so the value cell holds a bare - number. (R-factors come immediately after goodness-of-fit and - before `Best log-posterior` — both methods agree on this - order.) -11. `📉 Best log-posterior` *(Bayesian, optional)* — shown when - `best_log_posterior is not None`. -12–16. *(Bayesian only)* Convergence rows derived from - `convergence_diagnostics`: - - `📊 Convergence status` — `passed` / `failed`. - - `📊 Max r-hat` — formatted to 3 decimals. - - `📊 Min ess bulk` — formatted to 1 decimal. - - `📊 Draws per chain`. - - `📊 Chains`. +6. `📏 Goodness-of-fit (reduced χ²)` — `reduced_chi_square`. 7–10. + `📏 R-factor (Rf, %)`, `📏 R-factor squared (Rf², %)`, + `📏 Weighted R-factor (wR, %)`, `📏 Bragg R-factor (BR, %)` — each + row when the corresponding inputs are available. Units appear in the + metric name, so the value cell holds a bare number. (R-factors come + immediately after goodness-of-fit and before `Best log-posterior` — + both methods agree on this order.) +7. `📉 Best log-posterior` _(Bayesian, optional)_ — shown when + `best_log_posterior is not None`. 12–16. _(Bayesian only)_ + Convergence rows derived from `convergence_diagnostics`: - + `📊 Convergence status` — `passed` / `failed`. - `📊 Max r-hat` — + formatted to 3 decimals. - `📊 Min ess bulk` — formatted to 1 + decimal. - `📊 Draws per chain`. - `📊 Chains`. The shared-vocabulary `success` / `failed` for `Overall status` is -intentional cross-method consistency: a reader scanning LSQ and -Bayesian outputs side-by-side sees the same status word in the -same row position regardless of method. Bayesian-specific nuance -(sampler completed but convergence flagged, etc.) is exposed in -the convergence rows below. +intentional cross-method consistency: a reader scanning LSQ and Bayesian +outputs side-by-side sees the same status word in the same row position +regardless of method. Bayesian-specific nuance (sampler completed but +convergence flagged, etc.) is exposed in the convergence rows below. **Rows dropped relative to the previous emoji-line summary:** - `🎯 Committed point estimate: Best posterior sample` — already documented by the `Committed parameters` table footnote - (`value = estimate written back to the project (best posterior - sample)`). + (`value = estimate written back to the project (best posterior sample)`). - `🔁 Sampler completed: yes` — redundant with `Overall status`. - `⚙️ Sampler settings: steps=…, burn=…, …` — already in the `Settings used` table above the fit-results table. -- The derived `samples = n_draws × n_chains` count — derived from - the `Draws per chain` and `Chains` rows immediately below. +- The derived `samples = n_draws × n_chains` count — derived from the + `Draws per chain` and `Chains` rows immediately below. **Table-title icons.** The four fit-output tables now carry a distinguishing icon in their title so the four blocks are visually separable when scrolling: -| Table | Title prefix | -| --- | --- | -| Minimizer settings | `⚙️ Settings used:` | -| Fit-method summary | `📋 Least-squares fit results:` / `📋 Bayesian fit results:` | -| Committed values | `📈 Refined parameters:` / `📈 Committed parameters:` | -| Posterior summary (Bayesian only) | `📊 Posterior distribution:` | +| Table | Title prefix | +| --------------------------------- | ------------------------------------------------------------ | +| Minimizer settings | `⚙️ Settings used:` | +| Fit-method summary | `📋 Least-squares fit results:` / `📋 Bayesian fit results:` | +| Committed values | `📈 Refined parameters:` / `📈 Committed parameters:` | +| Posterior summary (Bayesian only) | `📊 Posterior distribution:` | The icons are also the same emoji used inside the rows of the -corresponding fit-results-summary table (📏 for goodness-of-fit -metrics, 📊 for convergence diagnostics), so the visual language -is internally consistent. - -**Internal-implementation note.** Helper -`print_metrics_table(rows)` in -`easydiffraction.utils.utils` renders the new 2-column table from -a list of `[label, value]` rows. Both -`reporting.FitResults.display_results()` and -`bayesian.BayesianFitResults.display_results()` build their rows -via a `_build_fit_results_rows()` instance method and feed -`print_metrics_table()`. The shared signature keeps the two -methods structurally parallel. +corresponding fit-results-summary table (📏 for goodness-of-fit metrics, +📊 for convergence diagnostics), so the visual language is internally +consistent. + +**Internal-implementation note.** Helper `print_metrics_table(rows)` in +`easydiffraction.utils.utils` renders the new 2-column table from a list +of `[label, value]` rows. Both `reporting.FitResults.display_results()` +and `bayesian.BayesianFitResults.display_results()` build their rows via +a `_build_fit_results_rows()` instance method and feed +`print_metrics_table()`. The shared signature keeps the two methods +structurally parallel. ## Consequences @@ -266,25 +256,24 @@ methods structurally parallel. - Tables fit standard HTML / markdown width without truncating the formerly 21-character `best posterior sample` column. -- Users can compare LSQ and Bayesian results column-by-column - (`start`, `value`, `s.u.`, `change` line up identically). +- Users can compare LSQ and Bayesian results column-by-column (`start`, + `value`, `s.u.`, `change` line up identically). - IUCr / GUM-aligned terminology. - The inline footnote glossary gives non-programmer users a discoverability path without having to leave the table to read external docs. -- Setting the convention in an ADR keeps future fit-result tables (a - new sampler, an alternative refinement strategy) on the same - naming. +- Setting the convention in an ADR keeps future fit-result tables (a new + sampler, an alternative refinement strategy) on the same naming. ### Trade-offs - Existing tutorials, tests, and integration outputs that pin the literal strings `Fitted parameters`, `Posterior parameter summaries`, - `fitted`, `best posterior sample`, `uncertainty`, `95% interval` - need updating in the implementation PR. -- `s.u.` is unfamiliar to readers who do not know GUM or IUCr CIF. - The footnote covers this; the compactness win at the column header - is the main argument. + `fitted`, `best posterior sample`, `uncertainty`, `95% interval` need + updating in the implementation PR. +- `s.u.` is unfamiliar to readers who do not know GUM or IUCr CIF. The + footnote covers this; the compactness win at the column header is the + main argument. ### ADRs related to this ADR @@ -305,41 +294,43 @@ None directly amended. This ADR complements: ### A. Keep `uncertainty` as the column header for both modes Pros: zero changes. Cons: ambiguous in Bayesian context (users may -confuse it with the 95% credible interval below); inconsistent with -the IUCr CIF `_su` convention; reinforces the wider `best posterior -sample` problem because it does not solve the layout issue. +confuse it with the 95% credible interval below); inconsistent with the +IUCr CIF `_su` convention; reinforces the wider `best posterior sample` +problem because it does not solve the layout issue. ### B. `posterior SD` for Bayesian, `uncertainty` for LSQ -Pros: explicit on the Bayesian side. Cons: different column headers -for the same physical quantity (1σ width), breaking the -column-by-column comparison; longer (10 chars vs 4 for `s.u.`). +Pros: explicit on the Bayesian side. Cons: different column headers for +the same physical quantity (1σ width), breaking the column-by-column +comparison; longer (10 chars vs 4 for `s.u.`). ### C. Different headers for the committed-value column (`refined` vs + `estimate` vs `value`) -Three different headers for "the value committed to the project". -Pros: each method-accurate. Cons: breaks the cross-method consistency -goal; readers seeing `refined` next to `estimate` in side-by-side -tables wonder what the semantic difference is even though the -underlying quantity is the same. Decision: use neutral `value` -everywhere, let the footnote disambiguate. +Three different headers for "the value committed to the project". Pros: +each method-accurate. Cons: breaks the cross-method consistency goal; +readers seeing `refined` next to `estimate` in side-by-side tables +wonder what the semantic difference is even though the underlying +quantity is the same. Decision: use neutral `value` everywhere, let the +footnote disambiguate. ### D. Single Bayesian table covering both committed values and + posterior summary -Pros: one table to read. Cons: nine value columns plus identity -columns exceed standard HTML width and truncate. The two-table split -is forced by layout and meaningfully preserves the "what did I -commit" vs "what does the posterior look like" duality. +Pros: one table to read. Cons: nine value columns plus identity columns +exceed standard HTML width and truncate. The two-table split is forced +by layout and meaningfully preserves the "what did I commit" vs "what +does the posterior look like" duality. ## Deferred Work -- The `acceptance rate` column in the posterior distribution table. - Not displayed by default today; a future ADR can decide whether it - joins the canonical layout or stays in a verbose mode. -- Inline footnote text vs Markdown link to a docs-site glossary. - Inline is the initial form; promotion to a glossary page is a - future ADR if footnote lengths grow. -- Localisation. All display strings are English; non-English UIs are - out of scope. +- The `acceptance rate` column in the posterior distribution table. Not + displayed by default today; a future ADR can decide whether it joins + the canonical layout or stays in a verbose mode. +- Inline footnote text vs Markdown link to a docs-site glossary. Inline + is the initial form; promotion to a glossary page is a future ADR if + footnote lengths grow. +- Localisation. All display strings are English; non-English UIs are out + of scope. diff --git a/docs/dev/adrs/accepted/minimizer-category-consolidation.md b/docs/dev/adrs/accepted/minimizer-category-consolidation.md index f99a169a8..7d0df4e1f 100644 --- a/docs/dev/adrs/accepted/minimizer-category-consolidation.md +++ b/docs/dev/adrs/accepted/minimizer-category-consolidation.md @@ -231,18 +231,18 @@ Verbose CIF tags are user-facing. The canonical MCMC abbreviation field so it appears in `help()` output but does not become a Python attribute or a CIF tag. -**Implementation note (2026-05-25).** The per-parameter R̂ and bulk -ESS values feeding `gelman_rubin_max` and -`effective_sample_size_min` are computed by an in-tree helper at +**Implementation note (2026-05-25).** The per-parameter R̂ and bulk ESS +values feeding `gelman_rubin_max` and `effective_sample_size_min` are +computed by an in-tree helper at [`analysis/fit_helpers/_diagnostics.py`](../../../../src/easydiffraction/analysis/fit_helpers/_diagnostics.py) -— pure NumPy + SciPy implementations of split R̂ and -rank-normalized bulk ESS (Vehtari, Gelman, Simpson, Carpenter and -Bürkner 2019; Geyer 1992). The earlier `arviz` dependency, which -the library only used to call `az.rhat()` and `az.ess(method='bulk')`, -has been removed; the diagnostics' public surface -(`gelman_rubin_max`, `effective_sample_size_min`, -`r_hat_by_parameter`, `ess_bulk_by_parameter`) and the convergence -thresholds (R̂ ≤ 1.01, ESS ≥ 400) are unchanged. +— pure NumPy + SciPy implementations of split R̂ and rank-normalized bulk +ESS (Vehtari, Gelman, Simpson, Carpenter and Bürkner 2019; Geyer 1992). +The earlier `arviz` dependency, which the library only used to call +`az.rhat()` and `az.ess(method='bulk')`, has been removed; the +diagnostics' public surface (`gelman_rubin_max`, +`effective_sample_size_min`, `r_hat_by_parameter`, +`ess_bulk_by_parameter`) and the convergence thresholds (R̂ ≤ 1.01, ESS +≥ 400) are unchanged. ### 6. Unified `initialization_method` enum diff --git a/docs/dev/issues/closed.md b/docs/dev/issues/closed.md index fbe4578c4..4186146ed 100644 --- a/docs/dev/issues/closed.md +++ b/docs/dev/issues/closed.md @@ -6,8 +6,7 @@ Issues that have been fully resolved. Kept for historical reference. ## 103. Make `_sync_engine_from_minimizer_category` Skip-Keys Declarative -Closed by -[`emcee-minimizer.md`](../plans/emcee-minimizer.md). Minimizer +Closed by [`emcee-minimizer.md`](../plans/emcee-minimizer.md). Minimizer categories now declare `_engine_sync_skip_keys`, and analysis sync filters against that set instead of hardcoding skipped keys. @@ -15,20 +14,19 @@ filters against that set instead of hardcoding skipped keys. ## 101. Remove Dead Branch in `_fit_state_categories` -Closed by -[`emcee-minimizer.md`](../plans/emcee-minimizer.md). The deterministic -branch that returned the same category list as the fallthrough path was -removed while preserving unsupported `result_kind` warning behavior. +Closed by [`emcee-minimizer.md`](../plans/emcee-minimizer.md). The +deterministic branch that returned the same category list as the +fallthrough path was removed while preserving unsupported `result_kind` +warning behavior. --- ## 100. Collapse Duplicate Predictive-Cache-Key Helpers -Closed by -[`emcee-minimizer.md`](../plans/emcee-minimizer.md). -`posterior_predictive_cache_key()` in `analysis.fit_helpers.bayesian` -is now the single helper used by analysis, plotting, and project -display code. +Closed by [`emcee-minimizer.md`](../plans/emcee-minimizer.md). +`posterior_predictive_cache_key()` in `analysis.fit_helpers.bayesian` is +now the single helper used by analysis, plotting, and project display +code. --- diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index ae9efe9dd..ccd37cc7e 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -259,9 +259,9 @@ When the matching open-issue is fully resolved, move it to `/emcee_chain` is present, expose a helper to construct an `emcee.backends.HDFBackend(path, name='emcee_chain', read_only=True)` for inspection/visualisation. -- `pyproject.toml` — add `emcee` as a - direct runtime dependency and refresh the lockfile via `pixi lock` (CI - installs from the lockfile, not from the manifest files alone). +- `pyproject.toml` — add `emcee` as a direct runtime dependency and + refresh the lockfile via `pixi lock` (CI installs from the lockfile, + not from the manifest files alone). ### Deleted @@ -272,8 +272,8 @@ When the matching open-issue is fully resolved, move it to Mark `[x]` as each step lands. - [x] **P1.1 — Add emcee dependency and refresh the lockfile.** - - Add `emcee` to `pyproject.toml` (runtime dependencies, not just - the `doc` extra — the existing lockfile carries emcee only as + - Add `emcee` to `pyproject.toml` (runtime dependencies, not just the + `doc` extra — the existing lockfile carries emcee only as `extra == 'doc'` which CI does not install for runtime). - Add the same dependency to `pixi.toml` (runtime feature). - Run `pixi lock` to regenerate `pixi.lock` with `emcee` as a direct @@ -300,8 +300,7 @@ Mark `[x]` as each step lands. native kwargs (see §"Decisions already made" point 3). - Class-level defaults for emcee-specific values: `sampling_steps=5000`, `burn_in_steps=1000`, `thinning_interval=1`, - `population_size=32`, `parallel_workers=0`, - `proposal_moves='de'`. + `population_size=32`, `parallel_workers=0`, `proposal_moves='de'`. - `__init__` constructs descriptors via the inherited helpers (`_sampling_steps_descriptor(default)`, etc. from `BayesianMinimizerBase`) and adds a new `proposal_moves` descriptor @@ -671,8 +670,9 @@ Mark `[x]` as each step lands. (post-switchable syntax), tutorial-sized sampler settings, `project.analysis.fit()`, posterior plots, and `project.save()`. - `ed-26.py`: reopening the saved project, displaying restored - Bayesian results, and `project.analysis.fit(resume=True, extra_steps=500)` - to continue the chain. + Bayesian results, and + `project.analysis.fit(resume=True, extra_steps=500)` to continue the + chain. Update the docs navigation in the same step: - Add an entry under "MCMC / Bayesian" (or the appropriate section) in @@ -808,5 +808,5 @@ single-file affair. Plots, parameter posteriors, and tables work the same as for DREAM, so switching between samplers to cross-check results is straightforward. -New tutorials walk through a short run (`ed-25`) and reopening the -saved project to resume for additional steps (`ed-26`). +New tutorials walk through a short run (`ed-25`) and reopening the saved +project to resume for additional steps (`ed-26`). diff --git a/docs/docs/tutorials/ed-22.py b/docs/docs/tutorials/ed-22.py index 24fcd4369..52fd31662 100644 --- a/docs/docs/tutorials/ed-22.py +++ b/docs/docs/tutorials/ed-22.py @@ -1,5 +1,5 @@ # %% [markdown] -# # Bayesian Analysis: Tb2TiO7 (`bumps-dream`), HEiDi +# # Bayesian Analysis: Tb2TiO7 (`emcee`), HEiDi # # This tutorial demonstrates a practical two-stage workflow for single-crystal # diffraction analysis with EasyDiffraction. @@ -7,7 +7,7 @@ # In the first stage, we run a fast local refinement to obtain a sensible # point estimate and parameter uncertainties. In the second stage, we use # these refined values to define fit bounds and then sample the posterior -# distribution with BUMPS-DREAM. +# distribution with emcee. # # The example uses constant-wavelength neutron single-crystal diffraction data # for Tb2TiO7 measured on HEiDi at FRM II. @@ -37,6 +37,9 @@ # %% project = ed.Project() +# %% +project.save_as('projects/tbti_heidi_emcee') + # %% [markdown] # ## Step 2: Build the Structural Model # @@ -158,7 +161,7 @@ # %% [markdown] # ## Step 5: Prepare for Bayesian Sampling # -# DREAM requires finite bounds for the free parameters. Instead of +# Bayesian samplers require finite bounds for the free parameters. Instead of # setting them manually, we derive them from the uncertainties estimated # in the local refinement. # @@ -195,18 +198,15 @@ project.display.parameters.free() # %% [markdown] -# ## Step 6: Configure and Run DREAM +# ## Step 6: Configure and Run emcee # -# We now switch from the local minimizer to the Bayesian DREAM sampler. +# We now switch from the local minimizer to the Bayesian emcee sampler. # # The settings below are intentionally small so the tutorial runs # quickly. For production analysis you would usually increase the number -# of steps (`steps`) and often the burn-in (`burn`) as well. When -# needed, the DREAM API also lets you tune how chains are initialized -# through the `init` setting. Other sampler settings such as `thin` and -# `pop` can be adjusted as well. The current EasyDiffraction defaults -# use `steps=3000`, `init='lhs'`, and `parallel=0`, which tells -# BUMPS-DREAM to use all available CPUs for population evaluations. +# of steps and often the burn-in as well. emcee also lets you tune how +# walkers are initialized, how many walkers are used, and which proposal +# move drives the ensemble. # # The `burn` setting is auto-resolved when left unset. Here we override # `steps` with a smaller value to keep the tutorial fast, and the @@ -216,11 +216,12 @@ project.analysis.minimizer.show_supported() # %% -project.analysis.minimizer.type = 'bumps (dream)' +project.analysis.minimizer.type = 'emcee' # %% -project.analysis.minimizer.sampling_steps = 100 # lower than the default 3000 -project.analysis.minimizer.burn_in_steps = 20 # lower than the default 600 +project.analysis.minimizer.sampling_steps = 500 # lower than the default 3000 +project.analysis.minimizer.burn_in_steps = 100 # lower than the default 600 +project.analysis.minimizer.population_size = 16 # lower than the default 32 # %% project.analysis.fit() diff --git a/docs/docs/tutorials/ed-25.py b/docs/docs/tutorials/ed-25.py index 8a6d3035b..58de766a7 100644 --- a/docs/docs/tutorials/ed-25.py +++ b/docs/docs/tutorials/ed-25.py @@ -278,11 +278,9 @@ # walkers are initialized, how many walkers are used, and which proposal # move drives the ensemble. # -# The default emcee proposal is the stretch move. This tutorial uses the -# differential-evolution move instead, because it mixes better for the -# strongly correlated LBCO/HRPT parameters. The walker count is kept -# below the default to keep runtime close to the DREAM tutorial while -# retaining good convergence diagnostics for this five-parameter example. +# The `burn` setting is auto-resolved when left unset. Here we override +# `steps` with a smaller value to keep the tutorial fast, and the +# effective burn-in is recomputed automatically. # %% project.analysis.minimizer.show_supported() diff --git a/docs/docs/tutorials/index.md b/docs/docs/tutorials/index.md index 7ec5bf301..7bca1cdc4 100644 --- a/docs/docs/tutorials/index.md +++ b/docs/docs/tutorials/index.md @@ -49,8 +49,8 @@ The tutorials are organized into the following categories: See also under [Bayesian Analysis](#bayesian-analysis): [LBCO Bayesian Display (`bumps-dream`)](ed-24.ipynb) and -[LBCO Bayesian Resume (`emcee`)](ed-26.ipynb) — both load saved -projects containing Bayesian fit state. +[LBCO Bayesian Resume (`emcee`)](ed-26.ipynb) — both load saved projects +containing Bayesian fit state. ## Powder Diffraction @@ -112,11 +112,11 @@ projects containing Bayesian fit state. - [LBCO Bayesian (`bumps-dream`)](ed-21.ipynb) – Demonstrates how to perform a Bayesian analysis of the La0.5Ba0.5CoO3 crystal structure - using constant wavelength neutron powder diffraction data from HRPT - at PSI. Covers the use of Markov Chain Monte Carlo (MCMC) sampling - with the bumps-DREAM minimizer to explore the posterior distribution - of the refined parameters, providing insights into parameter - uncertainties and correlations. + using constant wavelength neutron powder diffraction data from HRPT at + PSI. Covers the use of Markov Chain Monte Carlo (MCMC) sampling with + the bumps-DREAM minimizer to explore the posterior distribution of the + refined parameters, providing insights into parameter uncertainties + and correlations. - [LBCO Bayesian Display (`bumps-dream`)](ed-24.ipynb) – Shows how to reopen the saved Bayesian project produced by the LBCO Bayesian tutorial and inspect persisted fit summaries, correlation matrix, @@ -131,13 +131,13 @@ projects containing Bayesian fit state. - [LBCO Bayesian Resume (`emcee`)](ed-26.ipynb) – Loads a Bayesian project that already contains an emcee chain, inspects the posterior, and resumes sampling with additional steps. The full project state - (parameters, chain, plot caches) round-trips through disk. Resuming - is currently supported only for emcee, not for bumps-DREAM. -- [Tb2TiO7 Bayesian (`bumps-dream`)](ed-22.ipynb) – Another example of - a Bayesian analysis, focused on the Tb2TiO7 crystal structure using - constant wavelength neutron single crystal diffraction data from - HEiDi at FRM II. Similar to the LBCO Bayesian tutorial, it covers - MCMC sampling to explore the posterior distribution of the refined + (parameters, chain, plot caches) round-trips through disk. Resuming is + currently supported only for emcee, not for bumps-DREAM. +- [Tb2TiO7 Bayesian (`emcee`)](ed-22.ipynb) – Another example of a + Bayesian analysis, focused on the Tb2TiO7 crystal structure using + constant wavelength neutron single crystal diffraction data from HEiDi + at FRM II. Similar to the LBCO Bayesian tutorial, it covers MCMC + sampling to explore the posterior distribution of the refined parameters, providing insights into parameter uncertainties and correlations in the context of single crystal diffraction data. diff --git a/pyproject.toml b/pyproject.toml index 421686854..de4f24894 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -34,7 +34,7 @@ dependencies = [ 'sympy', # Symbolic mathematics library 'lmfit', # Non-linear optimization and curve fitting 'bumps', # Non-linear optimization and curve fitting - 'emcee', # Affine-invariant MCMC sampler + 'emcee', # Affine-invariant MCMC sampler 'dfo-ls', # Non-linear optimization and curve fitting 'gemmi', # Crystallography library 'cryspy', # Calculations of diffraction patterns From 30fa356f2b6f25ba3362bc9b3b00c95d2eac6a80 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 22:08:36 +0200 Subject: [PATCH 48/65] Update notebooks --- docs/docs/tutorials/ed-22.ipynb | 124 +++++++++++++++++--------------- docs/docs/tutorials/ed-25.ipynb | 16 +---- docs/docs/tutorials/ed-25.py | 2 - 3 files changed, 69 insertions(+), 73 deletions(-) diff --git a/docs/docs/tutorials/ed-22.ipynb b/docs/docs/tutorials/ed-22.ipynb index 8825c90dc..fa0c91566 100644 --- a/docs/docs/tutorials/ed-22.ipynb +++ b/docs/docs/tutorials/ed-22.ipynb @@ -24,7 +24,7 @@ "id": "1", "metadata": {}, "source": [ - "# Bayesian Analysis: Tb2TiO7, HEiDi\n", + "# Bayesian Analysis: Tb2TiO7 (`emcee`), HEiDi\n", "\n", "This tutorial demonstrates a practical two-stage workflow for single-crystal\n", "diffraction analysis with EasyDiffraction.\n", @@ -32,7 +32,7 @@ "In the first stage, we run a fast local refinement to obtain a sensible\n", "point estimate and parameter uncertainties. In the second stage, we use\n", "these refined values to define fit bounds and then sample the posterior\n", - "distribution with BUMPS-DREAM.\n", + "distribution with emcee.\n", "\n", "The example uses constant-wavelength neutron single-crystal diffraction data\n", "for Tb2TiO7 measured on HEiDi at FRM II.\n", @@ -88,9 +88,19 @@ ] }, { - "cell_type": "markdown", + "cell_type": "code", + "execution_count": null, "id": "6", "metadata": {}, + "outputs": [], + "source": [ + "project.save_as('projects/tbti_heidi_emcee')" + ] + }, + { + "cell_type": "markdown", + "id": "7", + "metadata": {}, "source": [ "## Step 2: Build the Structural Model\n", "\n", @@ -103,7 +113,7 @@ { "cell_type": "code", "execution_count": null, - "id": "7", + "id": "8", "metadata": {}, "outputs": [], "source": [ @@ -113,7 +123,7 @@ { "cell_type": "code", "execution_count": null, - "id": "8", + "id": "9", "metadata": {}, "outputs": [], "source": [ @@ -123,7 +133,7 @@ { "cell_type": "code", "execution_count": null, - "id": "9", + "id": "10", "metadata": {}, "outputs": [], "source": [ @@ -132,7 +142,7 @@ }, { "cell_type": "markdown", - "id": "10", + "id": "11", "metadata": {}, "source": [ "## Step 3: Define the Diffraction Experiment\n", @@ -145,7 +155,7 @@ { "cell_type": "code", "execution_count": null, - "id": "11", + "id": "12", "metadata": {}, "outputs": [], "source": [ @@ -155,7 +165,7 @@ { "cell_type": "code", "execution_count": null, - "id": "12", + "id": "13", "metadata": {}, "outputs": [], "source": [ @@ -171,7 +181,7 @@ { "cell_type": "code", "execution_count": null, - "id": "13", + "id": "14", "metadata": {}, "outputs": [], "source": [ @@ -180,7 +190,7 @@ }, { "cell_type": "markdown", - "id": "14", + "id": "15", "metadata": {}, "source": [ "Link the crystal structure to the experiment and set its scale factor." @@ -189,7 +199,7 @@ { "cell_type": "code", "execution_count": null, - "id": "15", + "id": "16", "metadata": {}, "outputs": [], "source": [ @@ -199,7 +209,7 @@ }, { "cell_type": "markdown", - "id": "16", + "id": "17", "metadata": {}, "source": [ "Set the instrument wavelength and starting extinction parameters.\n", @@ -210,7 +220,7 @@ { "cell_type": "code", "execution_count": null, - "id": "17", + "id": "18", "metadata": {}, "outputs": [], "source": [ @@ -220,7 +230,7 @@ { "cell_type": "code", "execution_count": null, - "id": "18", + "id": "19", "metadata": {}, "outputs": [], "source": [ @@ -230,7 +240,7 @@ }, { "cell_type": "markdown", - "id": "19", + "id": "20", "metadata": {}, "source": [ "## Step 4: Run an Initial Local Refinement\n", @@ -250,7 +260,7 @@ { "cell_type": "code", "execution_count": null, - "id": "20", + "id": "21", "metadata": {}, "outputs": [], "source": [ @@ -269,7 +279,7 @@ { "cell_type": "code", "execution_count": null, - "id": "21", + "id": "22", "metadata": {}, "outputs": [], "source": [ @@ -279,7 +289,7 @@ }, { "cell_type": "markdown", - "id": "22", + "id": "23", "metadata": {}, "source": [ "We keep using the default LMFIT Levenberg-Marquardt minimizer as a fast local\n", @@ -290,7 +300,7 @@ { "cell_type": "code", "execution_count": null, - "id": "23", + "id": "24", "metadata": {}, "outputs": [], "source": [ @@ -300,7 +310,7 @@ { "cell_type": "code", "execution_count": null, - "id": "24", + "id": "25", "metadata": {}, "outputs": [], "source": [ @@ -309,7 +319,7 @@ }, { "cell_type": "markdown", - "id": "25", + "id": "26", "metadata": {}, "source": [ "The fit-results display summarizes the locally refined values and their\n", @@ -319,7 +329,7 @@ { "cell_type": "code", "execution_count": null, - "id": "26", + "id": "27", "metadata": {}, "outputs": [], "source": [ @@ -328,7 +338,7 @@ }, { "cell_type": "markdown", - "id": "27", + "id": "28", "metadata": {}, "source": [ "The correlation plot shows how strongly the refined parameters move\n", @@ -340,7 +350,7 @@ { "cell_type": "code", "execution_count": null, - "id": "28", + "id": "29", "metadata": {}, "outputs": [], "source": [ @@ -350,7 +360,7 @@ { "cell_type": "code", "execution_count": null, - "id": "29", + "id": "30", "metadata": {}, "outputs": [], "source": [ @@ -359,12 +369,12 @@ }, { "cell_type": "markdown", - "id": "30", + "id": "31", "metadata": {}, "source": [ "## Step 5: Prepare for Bayesian Sampling\n", "\n", - "DREAM requires finite bounds for the free parameters. Instead of\n", + "Bayesian samplers require finite bounds for the free parameters. Instead of\n", "setting them manually, we derive them from the uncertainties estimated\n", "in the local refinement.\n", "\n", @@ -383,7 +393,7 @@ { "cell_type": "code", "execution_count": null, - "id": "31", + "id": "32", "metadata": {}, "outputs": [], "source": [ @@ -392,7 +402,7 @@ }, { "cell_type": "markdown", - "id": "32", + "id": "33", "metadata": {}, "source": [ "Set fit bounds for all free parameters using `multiplier=1.5`. In this\n", @@ -404,7 +414,7 @@ { "cell_type": "code", "execution_count": null, - "id": "33", + "id": "34", "metadata": {}, "outputs": [], "source": [ @@ -414,7 +424,7 @@ }, { "cell_type": "markdown", - "id": "34", + "id": "35", "metadata": {}, "source": [ "Displaying the free parameters again is a convenient way to confirm\n", @@ -425,7 +435,7 @@ { "cell_type": "code", "execution_count": null, - "id": "35", + "id": "36", "metadata": {}, "outputs": [], "source": [ @@ -434,21 +444,18 @@ }, { "cell_type": "markdown", - "id": "36", + "id": "37", "metadata": {}, "source": [ - "## Step 6: Configure and Run DREAM\n", + "## Step 6: Configure and Run emcee\n", "\n", - "We now switch from the local minimizer to the Bayesian DREAM sampler.\n", + "We now switch from the local minimizer to the Bayesian emcee sampler.\n", "\n", "The settings below are intentionally small so the tutorial runs\n", "quickly. For production analysis you would usually increase the number\n", - "of steps (`steps`) and often the burn-in (`burn`) as well. When\n", - "needed, the DREAM API also lets you tune how chains are initialized\n", - "through the `init` setting. Other sampler settings such as `thin` and\n", - "`pop` can be adjusted as well. The current EasyDiffraction defaults\n", - "use `steps=3000`, `init='lhs'`, and `parallel=0`, which tells\n", - "BUMPS-DREAM to use all available CPUs for population evaluations.\n", + "of steps and often the burn-in as well. emcee also lets you tune how\n", + "walkers are initialized, how many walkers are used, and which proposal\n", + "move drives the ensemble.\n", "\n", "The `burn` setting is auto-resolved when left unset. Here we override\n", "`steps` with a smaller value to keep the tutorial fast, and the\n", @@ -458,7 +465,7 @@ { "cell_type": "code", "execution_count": null, - "id": "37", + "id": "38", "metadata": {}, "outputs": [], "source": [ @@ -468,28 +475,29 @@ { "cell_type": "code", "execution_count": null, - "id": "38", + "id": "39", "metadata": {}, "outputs": [], "source": [ - "project.analysis.minimizer.type = 'bumps (dream)'" + "project.analysis.minimizer.type = 'emcee'" ] }, { "cell_type": "code", "execution_count": null, - "id": "39", + "id": "40", "metadata": {}, "outputs": [], "source": [ - "project.analysis.minimizer.sampling_steps = 100 # lower than the default 3000\n", - "project.analysis.minimizer.burn_in_steps = 20 # lower than the default 600" + "project.analysis.minimizer.sampling_steps = 500 # lower than the default 3000\n", + "project.analysis.minimizer.burn_in_steps = 100 # lower than the default 600\n", + "project.analysis.minimizer.population_size = 16 # lower than the default 32" ] }, { "cell_type": "code", "execution_count": null, - "id": "40", + "id": "41", "metadata": {}, "outputs": [], "source": [ @@ -498,7 +506,7 @@ }, { "cell_type": "markdown", - "id": "41", + "id": "42", "metadata": {}, "source": [ "## Step 7: Inspect Bayesian Results\n", @@ -511,7 +519,7 @@ { "cell_type": "code", "execution_count": null, - "id": "42", + "id": "43", "metadata": {}, "outputs": [], "source": [ @@ -520,7 +528,7 @@ }, { "cell_type": "markdown", - "id": "43", + "id": "44", "metadata": {}, "source": [ "The correlation and posterior-pair plots are complementary:\n", @@ -537,7 +545,7 @@ { "cell_type": "code", "execution_count": null, - "id": "44", + "id": "45", "metadata": {}, "outputs": [], "source": [ @@ -547,7 +555,7 @@ { "cell_type": "code", "execution_count": null, - "id": "45", + "id": "46", "metadata": {}, "outputs": [], "source": [ @@ -556,7 +564,7 @@ }, { "cell_type": "markdown", - "id": "46", + "id": "47", "metadata": {}, "source": [ "The one-dimensional posterior distributions below make it easier to\n", @@ -567,7 +575,7 @@ { "cell_type": "code", "execution_count": null, - "id": "47", + "id": "48", "metadata": {}, "outputs": [], "source": [ @@ -576,7 +584,7 @@ }, { "cell_type": "markdown", - "id": "48", + "id": "49", "metadata": {}, "source": [ "Finally, the posterior predictive plot propagates the sampled\n", @@ -587,7 +595,7 @@ { "cell_type": "code", "execution_count": null, - "id": "49", + "id": "50", "metadata": {}, "outputs": [], "source": [ diff --git a/docs/docs/tutorials/ed-25.ipynb b/docs/docs/tutorials/ed-25.ipynb index d29a0d746..d24eecfd3 100644 --- a/docs/docs/tutorials/ed-25.ipynb +++ b/docs/docs/tutorials/ed-25.ipynb @@ -570,11 +570,9 @@ "walkers are initialized, how many walkers are used, and which proposal\n", "move drives the ensemble.\n", "\n", - "The default emcee proposal is the stretch move. This tutorial uses the\n", - "differential-evolution move instead, because it mixes better for the\n", - "strongly correlated LBCO/HRPT parameters. The walker count is kept\n", - "below the default to keep runtime close to the DREAM tutorial while\n", - "retaining good convergence diagnostics for this five-parameter example." + "The `burn` setting is auto-resolved when left unset. Here we override\n", + "`steps` with a smaller value to keep the tutorial fast, and the\n", + "effective burn-in is recomputed automatically." ] }, { @@ -737,14 +735,6 @@ "source": [ "project.display.posterior.predictive(expt_name='hrpt', x_min=92, x_max=93)" ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "60", - "metadata": {}, - "outputs": [], - "source": [] } ], "metadata": { diff --git a/docs/docs/tutorials/ed-25.py b/docs/docs/tutorials/ed-25.py index 58de766a7..3a66d1950 100644 --- a/docs/docs/tutorials/ed-25.py +++ b/docs/docs/tutorials/ed-25.py @@ -347,5 +347,3 @@ # %% project.display.posterior.predictive(expt_name='hrpt', x_min=92, x_max=93) - -# %% From 5b51d6def2383552c619d27ede9eace0a1f32708 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 22:16:54 +0200 Subject: [PATCH 49/65] Add emcee phase 2 verification tests --- docs/dev/plans/emcee-minimizer.md | 2 +- tests/integration/fitting/test_emcee.py | 115 ++++++++++++++++++ .../analysis/minimizers/test_emcee.py | 11 +- .../minimizers/test_emcee_defaults.py | 20 +++ .../easydiffraction/analysis/test_analysis.py | 24 +++- .../io/test_results_sidecar.py | 26 ++++ 6 files changed, 192 insertions(+), 6 deletions(-) create mode 100644 tests/integration/fitting/test_emcee.py create mode 100644 tests/unit/easydiffraction/analysis/minimizers/test_emcee_defaults.py diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index ccd37cc7e..1f7a9785d 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -705,7 +705,7 @@ Mark `[x]` as each step lands. Each command captures its log with a zsh-safe exit-code variable as required by `AGENTS.md` → **Workflow**. -- [ ] **P2.1 — Add unit + integration tests.** +- [x] **P2.1 — Add unit + integration tests.** - `tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py`: category-class descriptor defaults; `_native_key_map` override; pairing with `BayesianFitResult`; swap behavior; resume diff --git a/tests/integration/fitting/test_emcee.py b/tests/integration/fitting/test_emcee.py new file mode 100644 index 000000000..e7aa41652 --- /dev/null +++ b/tests/integration/fitting/test_emcee.py @@ -0,0 +1,115 @@ +# SPDX-FileCopyrightText: 2026 EasyScience contributors +# SPDX-License-Identifier: BSD-3-Clause +"""Integration checks for emcee Bayesian sampling.""" + +from __future__ import annotations + +from dataclasses import dataclass + +import numpy as np +import pytest + +from easydiffraction.utils.enums import VerbosityEnum + + +@dataclass +class ToyParameter: + """Minimal parameter object accepted by the Bayesian engines.""" + + unique_name: str + value: float + fit_min: float + fit_max: float + uncertainty: float | None = None + + @property + def name(self) -> str: + """Return the display name used in posterior summaries.""" + return self.unique_name + + @property + def _minimizer_uid(self) -> str: + """Return the BUMPS parameter identifier.""" + return self.unique_name + + def _set_value_from_minimizer(self, value: float) -> None: + """Store a value committed by the minimizer.""" + self.value = value + + +def _toy_parameters() -> list[ToyParameter]: + return [ + ToyParameter(unique_name='x', value=0.0, fit_min=-4.0, fit_max=4.0), + ToyParameter(unique_name='y', value=0.0, fit_min=-4.0, fit_max=4.0), + ] + + +def _array_residuals(values: np.ndarray) -> np.ndarray: + target = np.asarray([1.2, -0.7], dtype=float) + sigma = np.asarray([0.25, 0.35], dtype=float) + return (np.asarray(values, dtype=float) - target) / sigma + + +def _mapping_residuals(values: dict[str, object]) -> np.ndarray: + return _array_residuals(np.asarray([values['x'], values['y']], dtype=float)) + + +def _posterior_medians(results: object) -> np.ndarray: + return np.asarray( + [summary.median for summary in results.posterior_parameter_summaries], + dtype=float, + ) + + +@pytest.mark.parametrize('proposal_moves', ['de']) +def test_emcee_resume_matches_small_dream_posterior(tmp_path, proposal_moves): + from easydiffraction.analysis.minimizers.bumps_dream import BumpsDreamMinimizer + from easydiffraction.analysis.minimizers.emcee import EmceeMinimizer + + dream = BumpsDreamMinimizer() + dream.steps = 80 + dream.burn = 20 + dream.thin = 1 + dream.pop = 4 + dream.parallel = 1 + dream_results = dream.fit( + _toy_parameters(), + _array_residuals, + verbosity=VerbosityEnum.SILENT, + random_seed=123, + ) + + emcee = EmceeMinimizer() + emcee.nsteps = 80 + emcee.nburn = 20 + emcee.thin = 1 + emcee.nwalkers = 16 + emcee.parallel_workers = 1 + emcee.proposal_moves = proposal_moves + emcee._sidecar_path = tmp_path / 'analysis' / 'results.h5' + emcee_results = emcee.fit( + _toy_parameters(), + _mapping_residuals, + verbosity=VerbosityEnum.SILENT, + random_seed=123, + ) + resumed_results = emcee.fit( + _toy_parameters(), + _mapping_residuals, + verbosity=VerbosityEnum.SILENT, + random_seed=123, + resume=True, + extra_steps=20, + ) + + assert dream_results.success is True + assert emcee_results.success is True + assert resumed_results.success is True + assert resumed_results.posterior_samples is not None + assert resumed_results.posterior_samples.parameter_samples.shape[1:] == (16, 2) + assert resumed_results.sampler_settings['total_steps'] == 100 + np.testing.assert_allclose( + _posterior_medians(resumed_results), + _posterior_medians(dream_results), + atol=0.35, + ) diff --git a/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py b/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py index a89f0f262..02ddb0cc9 100644 --- a/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py +++ b/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py @@ -388,10 +388,17 @@ def test_emcee_sampler_settings_record_sampling_and_total_steps(): n_parameters=2, ) - assert settings['steps'] == 100 - assert settings['burn'] == 20 + assert settings['nsteps'] == 100 + assert settings['nburn'] == 20 + assert settings['nwalkers'] == minimizer.nwalkers + assert settings['parallel_workers'] == minimizer.parallel_workers + assert settings['initialization_method'] == 'ball' + assert settings['proposal_moves'] == 'de' assert settings['total_steps'] == 121 assert settings['samples'] == 100 * minimizer.nwalkers * 2 + assert 'steps' not in settings + assert 'burn' not in settings + assert 'pop' not in settings def test_sample_with_progress_iterates_sampler_and_reports_each_state(): diff --git a/tests/unit/easydiffraction/analysis/minimizers/test_emcee_defaults.py b/tests/unit/easydiffraction/analysis/minimizers/test_emcee_defaults.py new file mode 100644 index 000000000..1e50a0bb1 --- /dev/null +++ b/tests/unit/easydiffraction/analysis/minimizers/test_emcee_defaults.py @@ -0,0 +1,20 @@ +# SPDX-FileCopyrightText: 2026 EasyScience contributors +# SPDX-License-Identifier: BSD-3-Clause +"""Tests for shared emcee minimizer defaults.""" + +from __future__ import annotations + + +def test_emcee_defaults_match_category_and_engine_imports(): + from easydiffraction.analysis.categories.minimizer import emcee as category_defaults + from easydiffraction.analysis.minimizers import emcee as engine_defaults + from easydiffraction.analysis.minimizers import emcee_defaults + + assert category_defaults.DEFAULT_SAMPLING_STEPS == emcee_defaults.DEFAULT_NSTEPS + assert category_defaults.DEFAULT_BURN_IN_STEPS == emcee_defaults.DEFAULT_NBURN + assert category_defaults.DEFAULT_POPULATION_SIZE == emcee_defaults.DEFAULT_NWALKERS + assert category_defaults.DEFAULT_PROPOSAL_MOVES == emcee_defaults.DEFAULT_PROPOSAL_MOVES + assert engine_defaults.DEFAULT_NSTEPS == emcee_defaults.DEFAULT_NSTEPS + assert engine_defaults.DEFAULT_NBURN == emcee_defaults.DEFAULT_NBURN + assert engine_defaults.DEFAULT_NWALKERS == emcee_defaults.DEFAULT_NWALKERS + assert engine_defaults.DEFAULT_PROPOSAL_MOVES == emcee_defaults.DEFAULT_PROPOSAL_MOVES diff --git a/tests/unit/easydiffraction/analysis/test_analysis.py b/tests/unit/easydiffraction/analysis/test_analysis.py index 0b83fdd38..30d9978a3 100644 --- a/tests/unit/easydiffraction/analysis/test_analysis.py +++ b/tests/unit/easydiffraction/analysis/test_analysis.py @@ -201,14 +201,32 @@ def test_restored_bayesian_sampler_settings_reconstruct_sample_count(): n_parameters=5, ) - assert settings['steps'] == 10000 - assert settings['burn'] == 2000 + assert settings['nsteps'] == 10000 + assert settings['nburn'] == 2000 assert settings['thin'] == 1 - assert settings['pop'] == 16 + assert settings['nwalkers'] == 16 + assert settings['parallel_workers'] == 0 + assert settings['initialization_method'] == 'ball' + assert settings['proposal_moves'] == 'de' assert settings['samples'] == 800000 assert settings['random_seed'] == 123 +def test_emcee_fit_requires_saved_project_for_new_and_resume_runs(): + import pytest + + from easydiffraction.analysis.analysis import Analysis + + analysis = Analysis(project=_make_project_with_names([])) + analysis.minimizer.type = 'emcee' + + with pytest.raises(ValueError, match='emcee requires a saved project'): + analysis.fit() + + with pytest.raises(ValueError, match='emcee requires a saved project'): + analysis.fit(resume=True) + + def test_restored_bayesian_reduced_chi_square_recovers_from_log_posterior(monkeypatch): from easydiffraction.analysis.analysis import Analysis from easydiffraction.analysis.categories.fit_result.bayesian import BayesianFitResult diff --git a/tests/unit/easydiffraction/io/test_results_sidecar.py b/tests/unit/easydiffraction/io/test_results_sidecar.py index 75dbe571e..ae95e9532 100644 --- a/tests/unit/easydiffraction/io/test_results_sidecar.py +++ b/tests/unit/easydiffraction/io/test_results_sidecar.py @@ -164,6 +164,32 @@ def test_write_analysis_results_sidecar_truncates_stale_payloads(tmp_path): assert 'hrpt' in handle['predictive'] +def test_write_analysis_results_sidecar_preserves_emcee_chain_group(tmp_path): + from easydiffraction.analysis.minimizers.emcee import EMCEE_CHAIN_GROUP + from easydiffraction.io import results_sidecar as results_sidecar_mod + + analysis_dir = Path(tmp_path) / 'analysis' + analysis = _analysis_with_sidecar_payload() + results_sidecar_mod.write_analysis_results_sidecar( + analysis=analysis, + analysis_dir=analysis_dir, + ) + + import h5py + + with h5py.File(analysis_dir / 'results.h5', 'a') as handle: + chain = handle.require_group(EMCEE_CHAIN_GROUP) + chain.attrs['iteration'] = 7 + + results_sidecar_mod.write_analysis_results_sidecar( + analysis=analysis, + analysis_dir=analysis_dir, + ) + + with h5py.File(analysis_dir / 'results.h5', 'r') as handle: + assert handle[EMCEE_CHAIN_GROUP].attrs['iteration'] == 7 + + def test_should_use_sidecar_compares_to_fit_result_kind_enum(): """`_should_use_sidecar` must read from `FitResultKindEnum`, not a literal.""" from easydiffraction.analysis.enums import FitResultKindEnum From 71e9ad5240bb7879bf9c6afedad8337b8084fab7 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 22:17:24 +0200 Subject: [PATCH 50/65] Apply pixi run fix auto-fixes --- .../analysis/fit_helpers/_diagnostics.py | 69 +++++++++---------- .../analysis/fit_helpers/bayesian.py | 15 ++-- .../analysis/fit_helpers/reporting.py | 4 +- src/easydiffraction/io/results_sidecar.py | 8 ++- src/easydiffraction/utils/logging.py | 34 +++++---- src/easydiffraction/utils/utils.py | 20 +++--- 6 files changed, 77 insertions(+), 73 deletions(-) diff --git a/src/easydiffraction/analysis/fit_helpers/_diagnostics.py b/src/easydiffraction/analysis/fit_helpers/_diagnostics.py index 8696992d5..5908ce1d9 100644 --- a/src/easydiffraction/analysis/fit_helpers/_diagnostics.py +++ b/src/easydiffraction/analysis/fit_helpers/_diagnostics.py @@ -3,18 +3,18 @@ """ MCMC convergence diagnostics computed in pure NumPy + SciPy. -The two diagnostics this module produces — split-chain Gelman–Rubin -R̂ and bulk effective sample size (ESS) — are the only Bayesian -diagnostics EasyDiffraction reports. The implementations follow the -standard formulas described in Vehtari, Gelman, Simpson, Carpenter -and Bürkner (2019), *Rank-normalization, folding, and localization: -An improved R̂ for assessing convergence of MCMC* -(https://arxiv.org/abs/1903.08008), Stan's reference manual, and -Geyer (1992), *Practical Markov chain Monte Carlo*. - -Inputs use the project's preserved layout: a 2-D NumPy array of -shape ``(n_draws, n_chains)`` per parameter, never an ArviZ -``InferenceData`` object. +The two diagnostics this module produces — split-chain Gelman–Rubin R̂ +and bulk effective sample size (ESS) — are the only Bayesian diagnostics +EasyDiffraction reports. The implementations follow the standard +formulas described in Vehtari, Gelman, Simpson, Carpenter and Bürkner +(2019), *Rank-normalization, folding, and localization: An improved R̂ +for assessing convergence of MCMC* (https://arxiv.org/abs/1903.08008), +Stan's reference manual, and Geyer (1992), *Practical Markov chain Monte +Carlo*. + +Inputs use the project's preserved layout: a 2-D NumPy array of shape +``(n_draws, n_chains)`` per parameter, never an ArviZ ``InferenceData`` +object. """ from __future__ import annotations @@ -29,22 +29,22 @@ def compute_r_hat(samples: np.ndarray) -> float: """ Split-chain Gelman–Rubin R̂ for one parameter. - Each chain is split in half (the standard "split R̂" variant); - the within-chain (W) and between-chain (B) variances are - computed on the doubled chain set, and R̂ is returned as - ``sqrt(V̂ / W)`` where ``V̂ = ((n-1)/n) · W + B/n``. + Each chain is split in half (the standard "split R̂" variant); the + within-chain (W) and between-chain (B) variances are computed on the + doubled chain set, and R̂ is returned as ``sqrt(V̂ / W)`` where ``V̂ + = ((n-1)/n) · W + B/n``. Parameters ---------- samples : np.ndarray - Posterior samples for one parameter with shape - ``(n_draws, n_chains)``. + Posterior samples for one parameter with shape ``(n_draws, + n_chains)``. Returns ------- float - R̂ value. ``nan`` when there are fewer than 4 draws, fewer - than 2 chains, or zero within-chain variance. + R̂ value. ``nan`` when there are fewer than 4 draws, fewer than + 2 chains, or zero within-chain variance. Raises ------ @@ -83,25 +83,24 @@ def compute_ess_bulk(samples: np.ndarray) -> float: Bulk effective sample size for one parameter. Samples are rank-normalized across all chain/draw pairs (so the - diagnostic is robust to heavy-tailed marginals); the - autocorrelation function is then averaged across chains and - summed with Geyer's initial positive sequence: pairs of - consecutive lags are added to the running variance estimate - until a pair first becomes non-positive (Geyer 1992; Vehtari - et al. 2019 §3.1). + diagnostic is robust to heavy-tailed marginals); the autocorrelation + function is then averaged across chains and summed with Geyer's + initial positive sequence: pairs of consecutive lags are added to + the running variance estimate until a pair first becomes + non-positive (Geyer 1992; Vehtari et al. 2019 §3.1). Parameters ---------- samples : np.ndarray - Posterior samples for one parameter with shape - ``(n_draws, n_chains)``. + Posterior samples for one parameter with shape ``(n_draws, + n_chains)``. Returns ------- float - Effective sample size in the bulk of the posterior. ``nan`` - when there are fewer than 4 draws, no chains, zero - variance, or the autocorrelation sum is non-positive. + Effective sample size in the bulk of the posterior. ``nan`` when + there are fewer than 4 draws, no chains, zero variance, or the + autocorrelation sum is non-positive. Raises ------ @@ -147,10 +146,10 @@ def _autocorr_fft(series: np.ndarray) -> np.ndarray: """ Return the normalized autocorrelation function via FFT. - Pads to the next power of two so the FFT is well-conditioned - for arbitrary chain lengths. The returned ACF has the same - length as the input series and starts at ``rho[0] = 1`` when - the input has non-zero variance. + Pads to the next power of two so the FFT is well-conditioned for + arbitrary chain lengths. The returned ACF has the same length as the + input series and starts at ``rho[0] = 1`` when the input has + non-zero variance. Parameters ---------- diff --git a/src/easydiffraction/analysis/fit_helpers/bayesian.py b/src/easydiffraction/analysis/fit_helpers/bayesian.py index 42ce7d770..cd6d86ca5 100644 --- a/src/easydiffraction/analysis/fit_helpers/bayesian.py +++ b/src/easydiffraction/analysis/fit_helpers/bayesian.py @@ -119,7 +119,8 @@ def flattened(self) -> np.ndarray: def validate_shapes(self) -> tuple[int, int, int]: """ - Validate stored sample shapes and return ``(n_draws, n_chains, n_parameters)``. + Validate stored sample shapes and return ``(n_draws, n_chains, + n_parameters)``. Returns ------- @@ -129,8 +130,8 @@ def validate_shapes(self) -> tuple[int, int, int]: Raises ------ ValueError - If the sample array is not 3-D, the parameter axis does - not match ``parameter_names``, or ``log_posterior`` (when + If the sample array is not 3-D, the parameter axis does not + match ``parameter_names``, or ``log_posterior`` (when present) does not match the first two sample axes. """ posterior_array = np.asarray(self.parameter_samples, dtype=float) @@ -553,10 +554,10 @@ def _bayesian_overall_status( """ Return ``'success'`` or ``'failed'`` for the Bayesian run. - Bayesian success requires both the sampler to have completed and - the convergence diagnostics to have passed. Anything else is - rendered as ``failed`` in the overall row; the per-metric - convergence rows below carry the detail. + Bayesian success requires both the sampler to have completed and the + convergence diagnostics to have passed. Anything else is rendered as + ``failed`` in the overall row; the per-metric convergence rows below + carry the detail. """ if not success or not sampler_completed: return 'failed' diff --git a/src/easydiffraction/analysis/fit_helpers/reporting.py b/src/easydiffraction/analysis/fit_helpers/reporting.py index 37a933c2d..bd442ae63 100644 --- a/src/easydiffraction/analysis/fit_helpers/reporting.py +++ b/src/easydiffraction/analysis/fit_helpers/reporting.py @@ -183,7 +183,9 @@ def _build_fit_results_rows( return rows def _print_table_notes(self) -> None: - """Print color-coded warnings below the refined parameters table.""" + """ + Print color-coded warnings below the refined parameters table. + """ notes: list[str] = [] if any(getattr(p, '_outside_physical_limits', False) for p in self.parameters): notes.append( diff --git a/src/easydiffraction/io/results_sidecar.py b/src/easydiffraction/io/results_sidecar.py index b570a26b5..920bc947b 100644 --- a/src/easydiffraction/io/results_sidecar.py +++ b/src/easydiffraction/io/results_sidecar.py @@ -90,13 +90,17 @@ def _create_dataset(handle: object, path: str, data: np.ndarray) -> None: def _delete_group_if_present(handle: object, group_name: str) -> None: - """Delete one top-level group from an open HDF5 file when present.""" + """ + Delete one top-level group from an open HDF5 file when present. + """ if group_name in handle: del handle[group_name] def _delete_canonical_groups(handle: object) -> None: - """Delete EasyDiffraction-owned top-level groups before append writes.""" + """ + Delete EasyDiffraction-owned top-level groups before append writes. + """ for group_name in _CANONICAL_GROUPS: _delete_group_if_present(handle, group_name) diff --git a/src/easydiffraction/utils/logging.py b/src/easydiffraction/utils/logging.py index 2a6f851c5..d1c18ffe0 100644 --- a/src/easydiffraction/utils/logging.py +++ b/src/easydiffraction/utils/logging.py @@ -650,12 +650,11 @@ def _rich_markup_to_inline_html(markup: str) -> str: """ Translate a narrow subset of Rich markup to inline HTML. - Handles ``[red]…[/red]`` (→ a red ````) and silently - strips ``[dim]`` / ``[/dim]`` tags (the surrounding container - already conveys dimness via CSS opacity). Other Rich markup - passes through unescaped — callers must only emit markup from - this allow-list when calling ``ConsolePrinter.small`` in - Jupyter. + Handles ``[red]…[/red]`` (→ a red ````) and silently strips + ``[dim]`` / ``[/dim]`` tags (the surrounding container already + conveys dimness via CSS opacity). Other Rich markup passes through + unescaped — callers must only emit markup from this allow-list when + calling ``ConsolePrinter.small`` in Jupyter. Parameters ---------- @@ -665,8 +664,8 @@ def _rich_markup_to_inline_html(markup: str) -> str: Returns ------- str - HTML-escaped string with the allow-listed Rich tags - translated to inline ```` styles. + HTML-escaped string with the allow-listed Rich tags translated + to inline ```` styles. """ escaped = html.escape(markup) without_dim = _RICH_DIM_MARKUP_PATTERN.sub('', escaped) @@ -743,21 +742,20 @@ def small(cls, *lines: str) -> None: """ Print one or more lines as dim, smaller supplementary text. - Intended for table footnote glossaries and inline warning - notes that should read as subordinate to the table or - block they sit beneath. In Jupyter the lines render inside a - single ````-style HTML element so the font size - matches Jupyter's ``.dataframe`` table-cell text. In a - terminal the lines render with Rich's ``dim`` style. Rich - ``[red]…[/red]`` markup inside ``lines`` is preserved in - both renderers. + Intended for table footnote glossaries and inline warning notes + that should read as subordinate to the table or block they sit + beneath. In Jupyter the lines render inside a single + ````-style HTML element so the font size matches + Jupyter's ``.dataframe`` table-cell text. In a terminal the + lines render with Rich's ``dim`` style. Rich ``[red]…[/red]`` + markup inside ``lines`` is preserved in both renderers. Parameters ---------- *lines : str Pre-formatted display lines. Each may contain Rich - ``[red]…[/red]`` markup; other Rich markup is rendered - in the terminal and stripped in the HTML output. + ``[red]…[/red]`` markup; other Rich markup is rendered in + the terminal and stripped in the HTML output. """ if not lines: return diff --git a/src/easydiffraction/utils/utils.py b/src/easydiffraction/utils/utils.py index d98e3ac64..334d88dd9 100644 --- a/src/easydiffraction/utils/utils.py +++ b/src/easydiffraction/utils/utils.py @@ -34,12 +34,12 @@ def display_path(path: pathlib.Path | str) -> str: Format a filesystem path for user-facing display. Returns the path relative to the current working directory so - messages stay compact and avoid forced line breaks. Paths - outside the cwd subtree use ``..`` segments to walk up to a - common ancestor (e.g. ``../sibling/data.cif``) rather than - falling back to an absolute path. The absolute path is only - used when no relative form is possible — on Windows that - happens when the path is on a different drive from the cwd. + messages stay compact and avoid forced line breaks. Paths outside + the cwd subtree use ``..`` segments to walk up to a common ancestor + (e.g. ``../sibling/data.cif``) rather than falling back to an + absolute path. The absolute path is only used when no relative form + is possible — on Windows that happens when the path is on a + different drive from the cwd. Parameters ---------- @@ -86,14 +86,14 @@ def print_table_footnote(entries: list[tuple[str, str]]) -> None: Print a glossary block below a fit-results-style table. Each entry renders as a left-aligned ``• header = description`` - bullet line. The block uses :meth:`ConsolePrinter.small` so it - shows as dim, smaller supplementary text — in Jupyter the font - size matches the table-cell text. + bullet line. The block uses :meth:`ConsolePrinter.small` so it shows + as dim, smaller supplementary text — in Jupyter the font size + matches the table-cell text. Parameters ---------- entries : list[tuple[str, str]] - Each tuple is `(column header, one-line description)`. + Each tuple is ``(column header, one-line description)``. """ if not entries: return From 6a21ceac87ad563542e6cd52016dbb19d19ead1e Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 22:18:26 +0200 Subject: [PATCH 51/65] Apply remaining pixi run fix auto-fixes --- src/easydiffraction/analysis/analysis.py | 13 ++--- .../categories/fit_parameters/default.py | 5 +- .../analysis/categories/minimizer/emcee.py | 17 ++++-- .../analysis/fit_helpers/reporting.py | 3 +- .../analysis/minimizers/emcee.py | 3 +- src/easydiffraction/display/progress.py | 4 +- src/easydiffraction/utils/utils.py | 12 ++--- .../fitting/test_bayesian_helper_support.py | 52 ++++++++++++------- .../categories/fit_result/test_lsq.py | 5 +- .../analysis/fit_helpers/test__diagnostics.py | 5 +- .../analysis/fit_helpers/test_tracking.py | 8 ++- .../analysis/minimizers/test_emcee.py | 14 +++-- .../easydiffraction/analysis/test_analysis.py | 3 +- 13 files changed, 69 insertions(+), 75 deletions(-) diff --git a/src/easydiffraction/analysis/analysis.py b/src/easydiffraction/analysis/analysis.py index b45a9f9ac..6b286b46a 100644 --- a/src/easydiffraction/analysis/analysis.py +++ b/src/easydiffraction/analysis/analysis.py @@ -676,8 +676,8 @@ def _restored_predictive_summaries(self) -> dict[str, PosteriorPredictiveSummary experiment_name, x_axis_name, include_draws=True, - ) - ] = summary + ) + ] = summary return restored_predictive @staticmethod @@ -703,10 +703,7 @@ def _restored_bayesian_converged( ess_bulk = cls._finite_float(min_ess_bulk) if r_hat is None or ess_bulk is None: return False - return ( - r_hat <= R_HAT_CONVERGENCE_THRESHOLD - and ess_bulk >= ESS_BULK_CONVERGENCE_THRESHOLD - ) + return r_hat <= R_HAT_CONVERGENCE_THRESHOLD and ess_bulk >= ESS_BULK_CONVERGENCE_THRESHOLD def _restored_bayesian_convergence_diagnostics( self, @@ -1985,9 +1982,7 @@ def _store_posterior_fit_projection(self, results: BayesianFitResults) -> None: self.fit_result._set_best_log_posterior(results.best_log_posterior) self.fit_result._set_credible_interval_inner(credible_interval_inner) self.fit_result._set_credible_interval_outer(credible_interval_outer) - self.fit_result._set_resolved_random_seed( - self._bayesian_result_random_seed(results) - ) + self.fit_result._set_resolved_random_seed(self._bayesian_result_random_seed(results)) self.fit_result._set_gelman_rubin_max(convergence.get('max_r_hat')) self.fit_result._set_effective_sample_size_min(convergence.get('min_ess_bulk')) self.fit_result._set_acceptance_rate_mean(convergence.get('acceptance_rate_mean')) diff --git a/src/easydiffraction/analysis/categories/fit_parameters/default.py b/src/easydiffraction/analysis/categories/fit_parameters/default.py index ab0dbc1fb..9db1ae05a 100644 --- a/src/easydiffraction/analysis/categories/fit_parameters/default.py +++ b/src/easydiffraction/analysis/categories/fit_parameters/default.py @@ -367,10 +367,7 @@ def _include_posterior_cif_descriptors(self) -> bool: def _include_uncertainty_multiplier_cif_descriptor(self) -> bool: """Return whether CIF output includes the bounds multiplier.""" - return any( - item.fit_bounds_uncertainty_multiplier.value is not None - for item in self - ) + return any(item.fit_bounds_uncertainty_multiplier.value is not None for item in self) def _cif_loop_parameters(self, item: FitParameterItem) -> list[object]: """Return CIF loop descriptors for the current fit kind.""" diff --git a/src/easydiffraction/analysis/categories/minimizer/emcee.py b/src/easydiffraction/analysis/categories/minimizer/emcee.py index af07f0cd4..6fc13eee5 100644 --- a/src/easydiffraction/analysis/categories/minimizer/emcee.py +++ b/src/easydiffraction/analysis/categories/minimizer/emcee.py @@ -9,12 +9,20 @@ from easydiffraction.analysis.categories.minimizer.bayesian_base import BayesianMinimizerBase from easydiffraction.analysis.categories.minimizer.factory import MinimizerCategoryFactory from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_INITIALIZATION_METHOD -from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_NBURN as DEFAULT_BURN_IN_STEPS -from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_NSTEPS as DEFAULT_SAMPLING_STEPS -from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_NWALKERS as DEFAULT_POPULATION_SIZE +from easydiffraction.analysis.minimizers.emcee_defaults import ( + DEFAULT_NBURN as DEFAULT_BURN_IN_STEPS, +) +from easydiffraction.analysis.minimizers.emcee_defaults import ( + DEFAULT_NSTEPS as DEFAULT_SAMPLING_STEPS, +) +from easydiffraction.analysis.minimizers.emcee_defaults import ( + DEFAULT_NWALKERS as DEFAULT_POPULATION_SIZE, +) from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_PARALLEL_WORKERS from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_PROPOSAL_MOVES -from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_THIN as DEFAULT_THINNING_INTERVAL +from easydiffraction.analysis.minimizers.emcee_defaults import ( + DEFAULT_THIN as DEFAULT_THINNING_INTERVAL, +) from easydiffraction.analysis.minimizers.emcee_defaults import SUPPORTED_PROPOSAL_MOVES from easydiffraction.analysis.minimizers.enums import InitializationMethodEnum from easydiffraction.analysis.minimizers.enums import MinimizerTypeEnum @@ -24,6 +32,7 @@ from easydiffraction.core.variable import StringDescriptor from easydiffraction.io.cif.handler import CifHandler + @MinimizerCategoryFactory.register class EmceeMinimizer(BayesianMinimizerBase): """Persisted settings for the emcee minimizer.""" diff --git a/src/easydiffraction/analysis/fit_helpers/reporting.py b/src/easydiffraction/analysis/fit_helpers/reporting.py index bd442ae63..6f84d7cac 100644 --- a/src/easydiffraction/analysis/fit_helpers/reporting.py +++ b/src/easydiffraction/analysis/fit_helpers/reporting.py @@ -194,8 +194,7 @@ def _print_table_notes(self) -> None: ) if any(_is_uncertainty_large(p) for p in self.parameters): notes.append( - '⚠️ [red]Red s.u.:[/red] exceeds the refined value ' - '(consider adding constraints)' + '⚠️ [red]Red s.u.:[/red] exceeds the refined value (consider adding constraints)' ) if notes: console.small(*notes) diff --git a/src/easydiffraction/analysis/minimizers/emcee.py b/src/easydiffraction/analysis/minimizers/emcee.py index a6c81c4eb..d12e64530 100644 --- a/src/easydiffraction/analysis/minimizers/emcee.py +++ b/src/easydiffraction/analysis/minimizers/emcee.py @@ -841,8 +841,7 @@ def _validate_resume( backend_shape = getattr(backend, 'shape', None) if backend_shape != (self.nwalkers, n_parameters): msg = ( - 'Existing emcee chain shape does not match current parameters; ' - 'start a fresh run.' + 'Existing emcee chain shape does not match current parameters; start a fresh run.' ) raise ValueError(msg) diff --git a/src/easydiffraction/display/progress.py b/src/easydiffraction/display/progress.py index 010a76fd5..1a9a73bc4 100644 --- a/src/easydiffraction/display/progress.py +++ b/src/easydiffraction/display/progress.py @@ -747,9 +747,7 @@ def _current_kernel_id() -> str: return '' with suppress(Exception): - return NotebookFitStopControl._kernel_id_from_connection_file( - get_connection_file() - ) + return NotebookFitStopControl._kernel_id_from_connection_file(get_connection_file()) return '' @staticmethod diff --git a/src/easydiffraction/utils/utils.py b/src/easydiffraction/utils/utils.py index 334d88dd9..2af1f53b1 100644 --- a/src/easydiffraction/utils/utils.py +++ b/src/easydiffraction/utils/utils.py @@ -350,13 +350,11 @@ def download_data( return str(project_dir) if not overwrite: console.print( - f"✅ Data #{id} already present at '{display_path(file_path)}'. " - 'Keeping existing.' + f"✅ Data #{id} already present at '{display_path(file_path)}'. Keeping existing." ) return str(file_path) log.debug( - f"Data #{id} already present at '{display_path(file_path)}', " - 'but will be overwritten.' + f"Data #{id} already present at '{display_path(file_path)}', but will be overwritten." ) file_path.unlink() @@ -376,9 +374,7 @@ def download_data( if is_project_archive: project_dir = extract_project_from_zip(file_path, destination=extraction_dir) file_path.unlink() - console.print( - f"✅ Data #{id} downloaded and extracted to '{display_path(project_dir)}'" - ) + console.print(f"✅ Data #{id} downloaded and extracted to '{display_path(project_dir)}'") return str(project_dir) console.print(f"✅ Data #{id} downloaded to '{display_path(file_path)}'") @@ -718,7 +714,7 @@ def download_all_tutorials( resolved_destination = resolve_artifact_path(destination) console.print( - f"✅ Downloaded {len(downloaded_paths)} tutorials to " + f'✅ Downloaded {len(downloaded_paths)} tutorials to ' f"'{display_path(resolved_destination)}'" ) return downloaded_paths diff --git a/tests/integration/fitting/test_bayesian_helper_support.py b/tests/integration/fitting/test_bayesian_helper_support.py index 1c3d67169..c8028d3b7 100644 --- a/tests/integration/fitting/test_bayesian_helper_support.py +++ b/tests/integration/fitting/test_bayesian_helper_support.py @@ -219,26 +219,38 @@ def test_bayesian_format_helpers_cover_edge_cases(): # Two-state overall-status helper: 'success' only when sampler # completed AND convergence passed. - assert _bayesian_overall_status( - success=False, - sampler_completed=False, - convergence_diagnostics={}, - ) == 'failed' - assert _bayesian_overall_status( - success=True, - sampler_completed=False, - convergence_diagnostics={'converged': False}, - ) == 'failed' - assert _bayesian_overall_status( - success=True, - sampler_completed=True, - convergence_diagnostics={'converged': True}, - ) == 'success' - assert _bayesian_overall_status( - success=True, - sampler_completed=False, - convergence_diagnostics={}, - ) == 'failed' + assert ( + _bayesian_overall_status( + success=False, + sampler_completed=False, + convergence_diagnostics={}, + ) + == 'failed' + ) + assert ( + _bayesian_overall_status( + success=True, + sampler_completed=False, + convergence_diagnostics={'converged': False}, + ) + == 'failed' + ) + assert ( + _bayesian_overall_status( + success=True, + sampler_completed=True, + convergence_diagnostics={'converged': True}, + ) + == 'success' + ) + assert ( + _bayesian_overall_status( + success=True, + sampler_completed=False, + convergence_diagnostics={}, + ) + == 'failed' + ) metrics = _calculate_fit_quality_metrics( y_obs=[10.0, 20.0], diff --git a/tests/unit/easydiffraction/analysis/categories/fit_result/test_lsq.py b/tests/unit/easydiffraction/analysis/categories/fit_result/test_lsq.py index 52cd075e2..d9e1c5e85 100644 --- a/tests/unit/easydiffraction/analysis/categories/fit_result/test_lsq.py +++ b/tests/unit/easydiffraction/analysis/categories/fit_result/test_lsq.py @@ -82,7 +82,4 @@ def test_least_squares_fit_result_keeps_distinct_exit_reason(): cif_text = fit_result.as_cif assert '_fit_result.message "Fit failed."' in cif_text - assert ( - '_fit_result.exit_reason "maximum number of evaluations reached"' - in cif_text - ) + assert '_fit_result.exit_reason "maximum number of evaluations reached"' in cif_text diff --git a/tests/unit/easydiffraction/analysis/fit_helpers/test__diagnostics.py b/tests/unit/easydiffraction/analysis/fit_helpers/test__diagnostics.py index aade1b4eb..5503009d3 100644 --- a/tests/unit/easydiffraction/analysis/fit_helpers/test__diagnostics.py +++ b/tests/unit/easydiffraction/analysis/fit_helpers/test__diagnostics.py @@ -53,10 +53,7 @@ def test_r_hat_close_to_one_for_well_mixed_independent_chains(): def test_r_hat_above_one_when_chains_disagree(): rng = np.random.default_rng(1) - chains_with_offsets = ( - rng.standard_normal((1000, 4)) - + np.array([-2.0, -1.0, 1.0, 2.0]) - ) + chains_with_offsets = rng.standard_normal((1000, 4)) + np.array([-2.0, -1.0, 1.0, 2.0]) r_hat = compute_r_hat(chains_with_offsets) assert math.isfinite(r_hat) diff --git a/tests/unit/easydiffraction/analysis/fit_helpers/test_tracking.py b/tests/unit/easydiffraction/analysis/fit_helpers/test_tracking.py index dc7af2447..59c5c22bd 100644 --- a/tests/unit/easydiffraction/analysis/fit_helpers/test_tracking.py +++ b/tests/unit/easydiffraction/analysis/fit_helpers/test_tracking.py @@ -160,8 +160,8 @@ def update(self, value: SimpleNamespace) -> None: pass assert 'Stop fitting' in html_updates[0] - assert "api/kernels/" in javascript_outputs[0] - assert "Interrupt sent..." in javascript_outputs[0] + assert 'api/kernels/' in javascript_outputs[0] + assert 'Interrupt sent...' in javascript_outputs[0] assert html_updates[-1] == '' @@ -200,8 +200,6 @@ def update(self, value: SimpleNamespace) -> None: def test_notebook_fit_stop_control_extracts_kernel_id_from_connection_file(): from easydiffraction.display.progress import NotebookFitStopControl - kernel_id = NotebookFitStopControl._kernel_id_from_connection_file( - 'kernel-abc-123.json' - ) + kernel_id = NotebookFitStopControl._kernel_id_from_connection_file('kernel-abc-123.json') assert kernel_id == 'abc-123' diff --git a/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py b/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py index 02ddb0cc9..46c0e2dd2 100644 --- a/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py +++ b/tests/unit/easydiffraction/analysis/minimizers/test_emcee.py @@ -39,14 +39,12 @@ def sample( skip_initial_state_check: bool, progress: bool, ) -> object: - self.calls.append( - { - 'initial_state': initial_state, - 'iterations': iterations, - 'skip_initial_state_check': skip_initial_state_check, - 'progress': progress, - } - ) + self.calls.append({ + 'initial_state': initial_state, + 'iterations': iterations, + 'skip_initial_state_check': skip_initial_state_check, + 'progress': progress, + }) for index in range(iterations): yield SimpleNamespace(log_prob=np.array([float(index)], dtype=float)) diff --git a/tests/unit/easydiffraction/analysis/test_analysis.py b/tests/unit/easydiffraction/analysis/test_analysis.py index 30d9978a3..9b124c667 100644 --- a/tests/unit/easydiffraction/analysis/test_analysis.py +++ b/tests/unit/easydiffraction/analysis/test_analysis.py @@ -103,8 +103,7 @@ def test_minimizer_selector_swap_warns_for_different_defaults(monkeypatch): removed_warning = next(w for w in warnings if 'removes these settings' in w) added_warning = next(w for w in warnings if 'adds these settings' in w) assert removed_warning == ( - 'Switching minimizer type removes these settings:\n' - '• max_iterations' + 'Switching minimizer type removes these settings:\n• max_iterations' ) assert added_warning.splitlines() == [ 'Switching minimizer type adds these settings with defaults:', From f8b745ea95a396e28a6f8bd0f491f49886b23c9a Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 22:37:49 +0200 Subject: [PATCH 52/65] Complete emcee static checks --- docs/dev/package-structure/full.md | 14 +- docs/dev/package-structure/short.md | 4 + docs/dev/plans/emcee-minimizer.md | 2 +- docs/docs/tutorials/ed-26.py | 18 ++- src/easydiffraction/analysis/analysis.py | 86 +++++------ .../analysis/fit_helpers/_diagnostics.py | 47 +++--- .../analysis/fit_helpers/bayesian.py | 142 ++++++++++++------ .../analysis/fit_helpers/reporting.py | 2 +- src/easydiffraction/analysis/fitting.py | 58 ++++--- .../analysis/minimizers/base.py | 50 +++--- .../analysis/minimizers/emcee.py | 20 ++- src/easydiffraction/utils/logging.py | 5 +- .../fitting/test_bayesian_dream.py | 4 +- .../fitting/test_bayesian_tracker_and_base.py | 11 +- tests/integration/fitting/test_emcee.py | 9 +- .../analysis/minimizers/test_base.py | 3 +- .../easydiffraction/analysis/test_analysis.py | 15 +- .../easydiffraction/analysis/test_fitting.py | 2 +- .../analysis/test_sequential.py | 8 +- 19 files changed, 292 insertions(+), 208 deletions(-) diff --git a/docs/dev/package-structure/full.md b/docs/dev/package-structure/full.md index 1f0663352..f39fef485 100644 --- a/docs/dev/package-structure/full.md +++ b/docs/dev/package-structure/full.md @@ -87,6 +87,8 @@ │ │ │ │ └── 🏷️ class BumpsLmMinimizer │ │ │ ├── 📄 dfols.py │ │ │ │ └── 🏷️ class DfolsMinimizer +│ │ │ ├── 📄 emcee.py +│ │ │ │ └── 🏷️ class EmceeMinimizer │ │ │ ├── 📄 factory.py │ │ │ │ └── 🏷️ class MinimizerCategoryFactory │ │ │ ├── 📄 lmfit.py @@ -113,6 +115,7 @@ │ │ └── 📄 __init__.py │ ├── 📁 fit_helpers │ │ ├── 📄 __init__.py +│ │ ├── 📄 _diagnostics.py │ │ ├── 📄 bayesian.py │ │ │ ├── 🏷️ class PosteriorPredictiveSummary │ │ │ ├── 🏷️ class PosteriorSamples @@ -126,6 +129,7 @@ │ ├── 📁 minimizers │ │ ├── 📄 __init__.py │ │ ├── 📄 base.py +│ │ │ ├── 🏷️ class MinimizerFitOptions │ │ │ └── 🏷️ class MinimizerBase │ │ ├── 📄 bumps.py │ │ │ ├── 🏷️ class _BumpsEvaluationLimitError @@ -145,6 +149,12 @@ │ │ │ └── 🏷️ class BumpsLmMinimizer │ │ ├── 📄 dfols.py │ │ │ └── 🏷️ class DfolsMinimizer +│ │ ├── 📄 emcee.py +│ │ │ ├── 🏷️ class _EmceePoolContext +│ │ │ ├── 🏷️ class _EmceeLogProbability +│ │ │ ├── 🏷️ class _EmceeProgressReporter +│ │ │ └── 🏷️ class EmceeMinimizer +│ │ ├── 📄 emcee_defaults.py │ │ ├── 📄 enums.py │ │ │ ├── 🏷️ class MinimizerTypeEnum │ │ │ ├── 🏷️ class InitializationMethodEnum @@ -168,6 +178,7 @@ │ │ ├── 🏷️ class FitResultKindEnum │ │ └── 🏷️ class FitCorrelationSourceEnum │ ├── 📄 fitting.py +│ │ ├── 🏷️ class FitterFitOptions │ │ └── 🏷️ class Fitter │ └── 📄 sequential.py │ ├── 🏷️ class SequentialFitExtractRule @@ -476,7 +487,8 @@ │ ├── 📄 progress.py │ │ ├── 🏷️ class _TerminalLiveHandle │ │ ├── 🏷️ class ActivityIndicator -│ │ └── 🏷️ class _ActivityIndicatorContext +│ │ ├── 🏷️ class _ActivityIndicatorContext +│ │ └── 🏷️ class NotebookFitStopControl │ ├── 📄 tables.py │ │ ├── 🏷️ class TableEngineEnum │ │ ├── 🏷️ class TableRenderer diff --git a/docs/dev/package-structure/short.md b/docs/dev/package-structure/short.md index e0a529800..25e3c1acd 100644 --- a/docs/dev/package-structure/short.md +++ b/docs/dev/package-structure/short.md @@ -52,6 +52,7 @@ │ │ │ ├── 📄 bumps_dream.py │ │ │ ├── 📄 bumps_lm.py │ │ │ ├── 📄 dfols.py +│ │ │ ├── 📄 emcee.py │ │ │ ├── 📄 factory.py │ │ │ ├── 📄 lmfit.py │ │ │ ├── 📄 lmfit_least_squares.py @@ -68,6 +69,7 @@ │ │ └── 📄 __init__.py │ ├── 📁 fit_helpers │ │ ├── 📄 __init__.py +│ │ ├── 📄 _diagnostics.py │ │ ├── 📄 bayesian.py │ │ ├── 📄 metrics.py │ │ ├── 📄 reporting.py @@ -81,6 +83,8 @@ │ │ ├── 📄 bumps_dream.py │ │ ├── 📄 bumps_lm.py │ │ ├── 📄 dfols.py +│ │ ├── 📄 emcee.py +│ │ ├── 📄 emcee_defaults.py │ │ ├── 📄 enums.py │ │ ├── 📄 factory.py │ │ ├── 📄 lmfit.py diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index 1f7a9785d..c8b90ad00 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -735,7 +735,7 @@ required by `AGENTS.md` → **Workflow**. exit $test_structure_check_exit_code ``` -- [ ] **P2.2 — Auto-fixes and static checks.** +- [x] **P2.2 — Auto-fixes and static checks.** ``` pixi run fix > /tmp/easydiffraction-fix.log 2>&1; \ diff --git a/docs/docs/tutorials/ed-26.py b/docs/docs/tutorials/ed-26.py index 42dc88803..2d72379df 100644 --- a/docs/docs/tutorials/ed-26.py +++ b/docs/docs/tutorials/ed-26.py @@ -2,14 +2,18 @@ # # Bayesian Analysis Resume (`emcee`): LBCO, HRPT # # This tutorial shows how to reopen the Bayesian project created previously, -# inspect the saved fit results and then run more sampling steps to extend the existing chain. -# Resuming only works with EMCEE because the current BUMPS-DREAM implementation does not support -# saving and resuming its state. +# inspect the saved fit results and then run more sampling steps to +# extend the existing chain. Resuming only works with EMCEE because the +# current BUMPS-DREAM implementation does not support saving and +# resuming its state. # # This workflow is useful when: # - the initial sampling run has not yet converged and more steps are needed, -# - the initial sampling run has converged but more steps are desired for better posterior resolution, -# - the initial sampling run has converged but the posterior plots have not yet been inspected and the user wants to see the plots before deciding whether to run more steps. +# - the initial sampling run has converged but more steps are desired +# for better posterior resolution, +# - the initial sampling run has converged but the posterior plots have +# not yet been inspected and the user wants to see the plots before +# deciding whether to run more steps. # # The workflow uses the same La0.5Ba0.5CoO3 powder diffraction example # as the DREAM Bayesian tutorial: @@ -98,7 +102,9 @@ # ## Resume emcee Sampling # # Resume from the saved backend and append 100 more emcee steps to the -# existing chain. We use only 100 steps here to keep the tutorial fast, but in practice you would typically run more steps to ensure convergence and better posterior resolution. +# existing chain. We use only 100 steps here to keep the tutorial fast, +# but in practice you would typically run more steps to ensure +# convergence and better posterior resolution. # %% project.analysis.fit(resume=True, extra_steps=100) diff --git a/src/easydiffraction/analysis/analysis.py b/src/easydiffraction/analysis/analysis.py index 6b286b46a..004fc6f9a 100644 --- a/src/easydiffraction/analysis/analysis.py +++ b/src/easydiffraction/analysis/analysis.py @@ -38,6 +38,7 @@ from easydiffraction.analysis.fit_helpers.bayesian import posterior_predictive_cache_key from easydiffraction.analysis.fit_helpers.reporting import FitResults from easydiffraction.analysis.fitting import Fitter +from easydiffraction.analysis.fitting import FitterFitOptions from easydiffraction.analysis.minimizers.emcee import EMCEE_CHAIN_GROUP from easydiffraction.analysis.minimizers.enums import MinimizerTypeEnum from easydiffraction.core.category_owner import CategoryOwner @@ -2113,10 +2114,7 @@ def _run_single( verb, structures, experiments, - use_physical_limits=False, - random_seed=None, - resume=resume, - extra_steps=extra_steps, + fit_options=FitterFitOptions(resume=resume, extra_steps=extra_steps), ) if self.project.info.path is not None: @@ -2142,10 +2140,7 @@ def _run_joint( verb, structures, experiments, - use_physical_limits=False, - random_seed=None, - resume=resume, - extra_steps=extra_steps, + fit_options=FitterFitOptions(resume=resume, extra_steps=extra_steps), ) if self.project.info.path is not None: @@ -2194,10 +2189,7 @@ def _fit_joint( structures: object, experiments: object, *, - use_physical_limits: bool, - random_seed: int | None, - resume: bool = False, - extra_steps: int | None = None, + fit_options: FitterFitOptions, ) -> None: """ Run joint fitting across all experiments with weights. @@ -2210,16 +2202,15 @@ def _fit_joint( Project structures collection. experiments : object Project experiments collection. - use_physical_limits : bool - Whether to use physical limits as fit bounds. - random_seed : int | None - Optional random seed passed to stochastic minimizers. - resume : bool, default=False - Whether to resume a sampler state. - extra_steps : int | None, default=None - Additional sampler steps for resume-capable minimizers. + fit_options : FitterFitOptions + Execution options controlling limits, randomness and resume. + + Raises + ------ + ValueError + If resume is requested for joint fitting. """ - if resume: + if fit_options.resume: msg = 'Resume is supported in single fit mode only.' raise ValueError(msg) @@ -2242,10 +2233,12 @@ def _fit_joint( weights=weights_array, analysis=self, verbosity=verb, - use_physical_limits=use_physical_limits, - random_seed=self._resolved_fit_random_seed(random_seed), - resume=resume, - extra_steps=extra_steps, + options=FitterFitOptions( + use_physical_limits=fit_options.use_physical_limits, + random_seed=self._resolved_fit_random_seed(fit_options.random_seed), + resume=fit_options.resume, + extra_steps=fit_options.extra_steps, + ), ) # After fitting, get the results @@ -2257,10 +2250,7 @@ def _fit_single( structures: object, experiments: object, *, - use_physical_limits: bool, - random_seed: int | None, - resume: bool = False, - extra_steps: int | None = None, + fit_options: FitterFitOptions, ) -> None: """ Run single-mode fitting for each experiment independently. @@ -2273,18 +2263,18 @@ def _fit_single( Project structures collection. experiments : object Project experiments collection. - use_physical_limits : bool - Whether to use physical limits as fit bounds. - random_seed : int | None - Optional random seed passed to stochastic minimizers. - resume : bool, default=False - Whether to resume a sampler state. - extra_steps : int | None, default=None - Additional sampler steps for resume-capable minimizers. + fit_options : FitterFitOptions + Execution options controlling limits, randomness and resume. + + Raises + ------ + ValueError + If resume is requested for more than one single-fit + experiment. """ mode = FitModeEnum.SINGLE expt_names = experiments.names - if resume and len(expt_names) != 1: + if fit_options.resume and len(expt_names) != 1: msg = 'Resume is supported for one single-fit experiment at a time.' raise ValueError(msg) @@ -2297,10 +2287,7 @@ def _fit_single( verb, structures, experiments, - use_physical_limits=use_physical_limits, - random_seed=random_seed, - resume=resume, - extra_steps=extra_steps, + fit_options=fit_options, short_state=(short_rows, short_display_handle), ) finally: @@ -2317,10 +2304,7 @@ def _fit_single_experiments( structures: object, experiments: object, *, - use_physical_limits: bool, - random_seed: int | None, - resume: bool, - extra_steps: int | None, + fit_options: FitterFitOptions, short_state: tuple[list[list[str]], object], ) -> None: """Run the per-experiment loop for single-fit mode.""" @@ -2338,10 +2322,12 @@ def _fit_single_experiments( [experiment], analysis=self, verbosity=verb, - use_physical_limits=use_physical_limits, - random_seed=self._resolved_fit_random_seed(random_seed), - resume=resume, - extra_steps=extra_steps, + options=FitterFitOptions( + use_physical_limits=fit_options.use_physical_limits, + random_seed=self._resolved_fit_random_seed(fit_options.random_seed), + resume=fit_options.resume, + extra_steps=fit_options.extra_steps, + ), ) results = self.fitter.results diff --git a/src/easydiffraction/analysis/fit_helpers/_diagnostics.py b/src/easydiffraction/analysis/fit_helpers/_diagnostics.py index 5908ce1d9..a50fb5c5d 100644 --- a/src/easydiffraction/analysis/fit_helpers/_diagnostics.py +++ b/src/easydiffraction/analysis/fit_helpers/_diagnostics.py @@ -3,14 +3,14 @@ """ MCMC convergence diagnostics computed in pure NumPy + SciPy. -The two diagnostics this module produces — split-chain Gelman–Rubin R̂ -and bulk effective sample size (ESS) — are the only Bayesian diagnostics -EasyDiffraction reports. The implementations follow the standard -formulas described in Vehtari, Gelman, Simpson, Carpenter and Bürkner -(2019), *Rank-normalization, folding, and localization: An improved R̂ -for assessing convergence of MCMC* (https://arxiv.org/abs/1903.08008), -Stan's reference manual, and Geyer (1992), *Practical Markov chain Monte -Carlo*. +The two diagnostics this module produces - split-chain Gelman-Rubin +R-hat and bulk effective sample size (ESS) - are the only Bayesian +diagnostics EasyDiffraction reports. The implementations follow the +standard formulas described in Vehtari, Gelman, Simpson, Carpenter and +Buerkner (2019), *Rank-normalization, folding, and localization: An +improved R-hat for assessing convergence of MCMC* +(https://arxiv.org/abs/1903.08008), Stan's reference manual, and Geyer +(1992), *Practical Markov chain Monte Carlo*. Inputs use the project's preserved layout: a 2-D NumPy array of shape ``(n_draws, n_chains)`` per parameter, never an ArviZ ``InferenceData`` @@ -23,16 +23,19 @@ from scipy import stats _MIN_DRAWS = 4 +_SAMPLE_NDIM = 2 +_MIN_RHAT_CHAINS = 2 +_MIN_ESS_CHAINS = 1 def compute_r_hat(samples: np.ndarray) -> float: """ - Split-chain Gelman–Rubin R̂ for one parameter. + Split-chain Gelman-Rubin R-hat for one parameter. - Each chain is split in half (the standard "split R̂" variant); the - within-chain (W) and between-chain (B) variances are computed on the - doubled chain set, and R̂ is returned as ``sqrt(V̂ / W)`` where ``V̂ - = ((n-1)/n) · W + B/n``. + Each chain is split in half (the standard "split R-hat" variant); + the within-chain (W) and between-chain (B) variances are computed on + the doubled chain set, and R-hat is returned as ``sqrt(Vhat / W)`` + where ``Vhat = ((n-1)/n) * W + B/n``. Parameters ---------- @@ -43,22 +46,22 @@ def compute_r_hat(samples: np.ndarray) -> float: Returns ------- float - R̂ value. ``nan`` when there are fewer than 4 draws, fewer than - 2 chains, or zero within-chain variance. + R-hat value. ``nan`` when there are too few draws or chains, or + zero within-chain variance. Raises ------ ValueError If ``samples`` is not 2-D. """ - if samples.ndim != 2: + if samples.ndim != _SAMPLE_NDIM: msg = 'samples must have shape (n_draws, n_chains)' raise ValueError(msg) n_draws, n_chains = samples.shape - if n_draws < _MIN_DRAWS or n_chains < 2: + if n_draws < _MIN_DRAWS or n_chains < _MIN_RHAT_CHAINS: return float('nan') - # Split each chain in half. With odd n_draws, drop the middle sample. + # Split each chain in half. For odd draws, drop the middle sample. half = n_draws // 2 splits = np.concatenate( [samples[:half, :], samples[-half:, :]], @@ -87,7 +90,7 @@ def compute_ess_bulk(samples: np.ndarray) -> float: function is then averaged across chains and summed with Geyer's initial positive sequence: pairs of consecutive lags are added to the running variance estimate until a pair first becomes - non-positive (Geyer 1992; Vehtari et al. 2019 §3.1). + non-positive (Geyer 1992; Vehtari et al. 2019 section 3.1). Parameters ---------- @@ -107,16 +110,16 @@ def compute_ess_bulk(samples: np.ndarray) -> float: ValueError If ``samples`` is not 2-D. """ - if samples.ndim != 2: + if samples.ndim != _SAMPLE_NDIM: msg = 'samples must have shape (n_draws, n_chains)' raise ValueError(msg) n_draws, n_chains = samples.shape - if n_draws < _MIN_DRAWS or n_chains < 1: + if n_draws < _MIN_DRAWS or n_chains < _MIN_ESS_CHAINS: return float('nan') total = n_draws * n_chains - # Rank-normalize across all samples → standard normal scores. + # Rank-normalize across all samples into standard normal scores. ranks = stats.rankdata(samples.ravel()).reshape(samples.shape) z = stats.norm.ppf((ranks - 0.5) / total) diff --git a/src/easydiffraction/analysis/fit_helpers/bayesian.py b/src/easydiffraction/analysis/fit_helpers/bayesian.py index cd6d86ca5..bc50ba7a7 100644 --- a/src/easydiffraction/analysis/fit_helpers/bayesian.py +++ b/src/easydiffraction/analysis/fit_helpers/bayesian.py @@ -119,8 +119,7 @@ def flattened(self) -> np.ndarray: def validate_shapes(self) -> tuple[int, int, int]: """ - Validate stored sample shapes and return ``(n_draws, n_chains, - n_parameters)``. + Validate stored sample shapes. Returns ------- @@ -312,47 +311,9 @@ def _build_fit_results_rows(self, metrics: dict[str, float | None]) -> list[list ) rows: list[list[str]] = [] - sampler_label = self.minimizer_type or self.sampler_name - if sampler_label: - rows.append(['🧪 Sampler', str(sampler_label)]) - rows.append([_overall_status_row_label(overall_status), overall_status]) - if self.message: - rows.append(['💬 Engine message', self.message]) - if self.fitting_time is not None: - rows.append(['⏱️ Fitting time (seconds)', f'{self.fitting_time:.2f}']) - if self.reduced_chi_square is not None: - rows.append(['📏 Goodness-of-fit (reduced χ²)', f'{self.reduced_chi_square:.2f}']) - rf = metrics.get('rf') - rf2 = metrics.get('rf2') - wr = metrics.get('wr') - br = metrics.get('br') - if rf is not None: - rows.append(['📏 R-factor (Rf, %)', f'{rf:.2f}']) - if rf2 is not None: - rows.append(['📏 R-factor squared (Rf², %)', f'{rf2:.2f}']) - if wr is not None: - rows.append(['📏 Weighted R-factor (wR, %)', f'{wr:.2f}']) - if br is not None: - rows.append(['📏 Bragg R-factor (BR, %)', f'{br:.2f}']) - if self.best_log_posterior is not None: - rows.append(['📉 Best log-posterior', f'{self.best_log_posterior:.2f}']) - - diagnostics = self.convergence_diagnostics or {} - converged = diagnostics.get('converged') - if converged is not None: - rows.append(['📊 Convergence status', 'passed' if converged else 'failed']) - max_r_hat = diagnostics.get('max_r_hat') - if max_r_hat is not None: - rows.append(['📊 Max r-hat', f'{max_r_hat:.3f}']) - min_ess_bulk = diagnostics.get('min_ess_bulk') - if min_ess_bulk is not None: - rows.append(['📊 Min ess bulk', f'{min_ess_bulk:.1f}']) - n_draws = diagnostics.get('n_draws') - if n_draws is not None: - rows.append(['📊 Draws per chain', str(n_draws)]) - n_chains = diagnostics.get('n_chains') - if n_chains is not None: - rows.append(['📊 Chains', str(n_chains)]) + _append_bayesian_identity_rows(results=self, rows=rows, overall_status=overall_status) + _append_fit_quality_rows(results=self, rows=rows, metrics=metrics) + _append_convergence_rows(rows=rows, diagnostics=self.convergence_diagnostics or {}) return rows def _print_table_notes(self) -> None: @@ -365,6 +326,93 @@ def _print_table_notes(self) -> None: console.small(*notes) +def _append_bayesian_identity_rows( + *, + results: BayesianFitResults, + rows: list[list[str]], + overall_status: str, +) -> None: + """Append sampler identity and status rows.""" + sampler_label = results.minimizer_type or results.sampler_name + if sampler_label: + rows.append(['🧪 Sampler', str(sampler_label)]) + rows.append([_overall_status_row_label(overall_status), overall_status]) + if results.message: + rows.append(['💬 Engine message', results.message]) + + +def _append_fit_quality_rows( + *, + results: BayesianFitResults, + rows: list[list[str]], + metrics: dict[str, float | None], +) -> None: + """Append fit-quality and best-posterior rows.""" + if results.fitting_time is not None: + rows.append(['⏱️ Fitting time (seconds)', f'{results.fitting_time:.2f}']) + if results.reduced_chi_square is not None: + rows.append([ + '📏 Goodness-of-fit (reduced χ²)', + f'{results.reduced_chi_square:.2f}', + ]) + for key, label in ( + ('rf', '📏 R-factor (Rf, %)'), + ('rf2', '📏 R-factor squared (Rf², %)'), + ('wr', '📏 Weighted R-factor (wR, %)'), + ('br', '📏 Bragg R-factor (BR, %)'), + ): + value = metrics.get(key) + if value is not None: + rows.append([label, f'{value:.2f}']) + if results.best_log_posterior is not None: + rows.append(['📉 Best log-posterior', f'{results.best_log_posterior:.2f}']) + + +def _append_convergence_rows( + *, + rows: list[list[str]], + diagnostics: dict[str, object], +) -> None: + """Append Bayesian convergence rows.""" + converged = diagnostics.get('converged') + if converged is not None: + rows.append(['📊 Convergence status', 'passed' if converged else 'failed']) + _append_optional_row(rows=rows, label='📊 Max r-hat', value=diagnostics.get('max_r_hat')) + _append_optional_row( + rows=rows, + label='📊 Min ess bulk', + value=diagnostics.get('min_ess_bulk'), + ) + _append_optional_row( + rows=rows, + label='📊 Draws per chain', + value=diagnostics.get('n_draws'), + precision=None, + ) + _append_optional_row( + rows=rows, + label='📊 Chains', + value=diagnostics.get('n_chains'), + precision=None, + ) + + +def _append_optional_row( + *, + rows: list[list[str]], + label: str, + value: object, + precision: int | None = 3, +) -> None: + """Append a formatted row when ``value`` is present.""" + if value is None: + return + if precision is None: + rows.append([label, str(value)]) + return + rows.append([label, f'{float(value):.{precision}f}']) + + def compute_convergence_diagnostics(posterior_samples: PosteriorSamples) -> dict[str, object]: """ Compute convergence diagnostics from posterior samples. @@ -570,15 +618,15 @@ def _bayesian_overall_status( _COMMITTED_PARAMETERS_FOOTNOTE: list[tuple[str, str]] = [ ('start', 'parameter value before sampling'), ('value', 'estimate written back to the project (best posterior sample)'), - ('s.u.', 'standard uncertainty (1σ), the posterior standard deviation'), + ('s.u.', 'standard uncertainty (one sigma), posterior standard deviation'), ('change', 'relative change from start, in %; ↑ = increase, ↓ = decrease'), ] _POSTERIOR_DISTRIBUTION_FOOTNOTE: list[tuple[str, str]] = [ ('median', '50th percentile of the marginal posterior'), - ('95% CI', '95% credible interval (2.5%–97.5%, asymmetric)'), - ('r-hat', 'Gelman–Rubin diagnostic R̂ (good convergence: r-hat ≤ 1.01)'), - ('ess bulk', 'bulk effective sample size (typically ≥ 400)'), + ('95% CI', '95% credible interval (2.5%-97.5%, asymmetric)'), + ('r-hat', 'Gelman-Rubin diagnostic (good convergence: r-hat <= 1.01)'), + ('ess bulk', 'bulk effective sample size (typically >= 400)'), ] diff --git a/src/easydiffraction/analysis/fit_helpers/reporting.py b/src/easydiffraction/analysis/fit_helpers/reporting.py index 6f84d7cac..40f898d86 100644 --- a/src/easydiffraction/analysis/fit_helpers/reporting.py +++ b/src/easydiffraction/analysis/fit_helpers/reporting.py @@ -203,7 +203,7 @@ def _print_table_notes(self) -> None: _REFINED_PARAMETERS_FOOTNOTE: list[tuple[str, str]] = [ ('start', 'parameter value before refinement'), ('value', 'refined value from least-squares minimization'), - ('s.u.', 'standard uncertainty (1σ), from the covariance matrix'), + ('s.u.', 'standard uncertainty (one sigma), from the covariance matrix'), ('change', 'relative change from start, in %; ↑ = increase, ↓ = decrease'), ] diff --git a/src/easydiffraction/analysis/fitting.py b/src/easydiffraction/analysis/fitting.py index 90587853d..7dd8c731d 100644 --- a/src/easydiffraction/analysis/fitting.py +++ b/src/easydiffraction/analysis/fitting.py @@ -3,12 +3,14 @@ from __future__ import annotations +from dataclasses import dataclass from typing import TYPE_CHECKING from typing import Any import numpy as np from easydiffraction.analysis.fit_helpers.metrics import get_reliability_inputs +from easydiffraction.analysis.minimizers.base import MinimizerFitOptions from easydiffraction.analysis.minimizers.enums import MinimizerTypeEnum from easydiffraction.analysis.minimizers.factory import MinimizerFactory from easydiffraction.core.variable import Parameter @@ -21,6 +23,26 @@ from easydiffraction.datablocks.structure.collection import Structures +@dataclass(frozen=True, slots=True) +class FitterFitOptions: + """Execution options for one fitter run.""" + + use_physical_limits: bool = False + random_seed: int | None = None + resume: bool = False + extra_steps: int | None = None + + def as_minimizer_options(self) -> MinimizerFitOptions: + """Return equivalent minimizer options for this fitter run.""" + return MinimizerFitOptions( + finalize_tracking=False, + use_physical_limits=self.use_physical_limits, + random_seed=self.random_seed, + resume=self.resume, + extra_steps=self.extra_steps, + ) + + def _resolve_fit_result_message(results: FitResults) -> str: """Return a normalized fit-result message.""" if results.message: @@ -146,10 +168,7 @@ def fit( analysis: object = None, verbosity: VerbosityEnum = VerbosityEnum.FULL, *, - use_physical_limits: bool = False, - random_seed: int | None = None, - resume: bool = False, - extra_steps: int | None = None, + options: FitterFitOptions | None = None, ) -> None: """ Run the fitting process. @@ -172,17 +191,16 @@ def fit( fitting. verbosity : VerbosityEnum, default=VerbosityEnum.FULL Console output verbosity. - use_physical_limits : bool, default=False - When ``True``, fall back to physical limits from the value - spec for parameters whose ``fit_min``/``fit_max`` are - unbounded. - random_seed : int | None, default=None - Optional random seed passed to stochastic minimizers. - resume : bool, default=False - Whether to resume a sampler state. - extra_steps : int | None, default=None - Additional sampler steps for resume-capable minimizers. + options : FitterFitOptions | None, default=None + Execution options controlling limits, randomness and resume. + + Raises + ------ + ValueError + If resume is requested without the same free parameter set + used by the saved emcee chain. """ + fit_options = options or FitterFitOptions() # Enforce symmetry constraints (e.g. ADP) before collecting # free parameters so that components fixed by site symmetry are # excluded from the minimizer's parameter set. @@ -193,7 +211,7 @@ def fit( params = self._collect_fit_parameters(structures, experiments) if not params: - if resume: + if fit_options.resume: msg = 'Resume requires the same free parameters used by the saved emcee chain.' raise ValueError(msg) if analysis is not None: @@ -203,9 +221,9 @@ def fit( print('⚠️ No parameters selected for fitting.') return - if analysis is not None and not resume: + if analysis is not None and not fit_options.resume: analysis._capture_fit_parameter_state(params) - if analysis is not None and resume: + if analysis is not None and fit_options.resume: self._validate_resume_parameter_set(params=params, analysis=analysis) for param in params: @@ -228,11 +246,7 @@ def fit( params, objective_function, verbosity=verbosity, - finalize_tracking=False, - use_physical_limits=use_physical_limits, - random_seed=random_seed, - resume=resume, - extra_steps=extra_steps, + options=fit_options.as_minimizer_options(), ) self._postprocess_fit_results( analysis=analysis, diff --git a/src/easydiffraction/analysis/minimizers/base.py b/src/easydiffraction/analysis/minimizers/base.py index a5ebb7020..9be83b6ad 100644 --- a/src/easydiffraction/analysis/minimizers/base.py +++ b/src/easydiffraction/analysis/minimizers/base.py @@ -4,6 +4,7 @@ from abc import ABC from abc import abstractmethod from collections.abc import Callable +from dataclasses import dataclass from typing import Any import numpy as np @@ -16,6 +17,17 @@ BOUNDARY_PROXIMITY_FRACTION = 0.01 +@dataclass(frozen=True, slots=True) +class MinimizerFitOptions: + """Execution options for one minimizer run.""" + + finalize_tracking: bool = True + use_physical_limits: bool = False + random_seed: int | None = None + resume: bool = False + extra_steps: int | None = None + + class MinimizerBase(ABC): """ Abstract base for concrete minimizers. @@ -354,11 +366,7 @@ def fit( objective_function: Callable[..., object], verbosity: VerbosityEnum = VerbosityEnum.FULL, *, - finalize_tracking: bool = True, - use_physical_limits: bool = False, - random_seed: int | None = None, - resume: bool = False, - extra_steps: int | None = None, + options: MinimizerFitOptions | None = None, ) -> FitResults: """ Run the full minimization workflow. @@ -372,35 +380,31 @@ def fit( arguments. verbosity : VerbosityEnum, default=VerbosityEnum.FULL Console output verbosity. - finalize_tracking : bool, default=True - Whether to stop and finalize live tracking before returning. - use_physical_limits : bool, default=False - When ``True``, fall back to physical limits from the value - spec for parameters whose ``fit_min``/``fit_max`` are - unbounded. - random_seed : int | None, default=None - Optional random seed passed to stochastic minimizers. - resume : bool, default=False - Whether to resume an existing sampler state. Unsupported by - the base minimizer implementation. - extra_steps : int | None, default=None - Additional sampler steps for resume-capable minimizers. + options : MinimizerFitOptions | None, default=None + Execution options controlling limits, randomness, resume, + and tracker finalization. Returns ------- FitResults FitResults with success flag, best chi2 and timing. + + Raises + ------ + NotImplementedError + If resume is requested for a minimizer that does not support + it. """ - del extra_steps - if resume: + fit_options = options or MinimizerFitOptions() + if fit_options.resume: minimizer_name = self.name or self.__class__.__name__ msg = f"Minimizer '{minimizer_name}' does not support resume." raise NotImplementedError(msg) - if use_physical_limits: + if fit_options.use_physical_limits: self._apply_physical_limits(parameters) - resolved_random_seed = self._resolve_random_seed(random_seed) + resolved_random_seed = self._resolve_random_seed(fit_options.random_seed) minimizer_name = self.name or 'Unnamed Minimizer' if self.method is not None and f'({self.method})' not in minimizer_name: @@ -415,7 +419,7 @@ def fit( raw_result = self._run_solver(objective_function, **solver_args) return self._finalize_fit(parameters, raw_result) finally: - if finalize_tracking: + if fit_options.finalize_tracking: self._stop_tracking() def _objective_function( diff --git a/src/easydiffraction/analysis/minimizers/emcee.py b/src/easydiffraction/analysis/minimizers/emcee.py index d12e64530..51c129fb9 100644 --- a/src/easydiffraction/analysis/minimizers/emcee.py +++ b/src/easydiffraction/analysis/minimizers/emcee.py @@ -23,6 +23,7 @@ from easydiffraction.analysis.fit_helpers.metrics import calculate_reduced_chi_square from easydiffraction.analysis.fit_helpers.tracking import SamplerProgressUpdate from easydiffraction.analysis.minimizers.base import MinimizerBase +from easydiffraction.analysis.minimizers.base import MinimizerFitOptions from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_INITIALIZATION_METHOD from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_METHOD from easydiffraction.analysis.minimizers.emcee_defaults import DEFAULT_NBURN @@ -367,37 +368,34 @@ def proposal_moves(self) -> str: def proposal_moves(self, value: str) -> None: self._proposal_moves = self._validated_proposal_moves(value) - def fit( # noqa: PLR0913 + def fit( self, parameters: list[object], objective_function: Callable[..., object], verbosity: VerbosityEnum = VerbosityEnum.FULL, *, - finalize_tracking: bool = True, - use_physical_limits: bool = False, - random_seed: int | None = None, - resume: bool = False, - extra_steps: int | None = None, + options: MinimizerFitOptions | None = None, ) -> BayesianFitResults: """ Run emcee sampling and return Bayesian fit results. """ - if use_physical_limits: + fit_options = options or MinimizerFitOptions() + if fit_options.use_physical_limits: self._apply_physical_limits(parameters) - resolved_random_seed = self._resolve_random_seed(random_seed) + resolved_random_seed = self._resolve_random_seed(fit_options.random_seed) minimizer_name = self.name or 'emcee' self._start_tracking(minimizer_name, verbosity=verbosity) try: solver_args = self._prepare_solver_args(parameters) solver_args['random_seed'] = resolved_random_seed - solver_args['resume'] = resume - solver_args['extra_steps'] = extra_steps + solver_args['resume'] = fit_options.resume + solver_args['extra_steps'] = fit_options.extra_steps raw_result = self._run_solver(objective_function, **solver_args) return self._finalize_fit(parameters, raw_result) finally: - if finalize_tracking: + if fit_options.finalize_tracking: self._stop_tracking() @staticmethod diff --git a/src/easydiffraction/utils/logging.py b/src/easydiffraction/utils/logging.py index d1c18ffe0..a185205fb 100644 --- a/src/easydiffraction/utils/logging.py +++ b/src/easydiffraction/utils/logging.py @@ -765,6 +765,9 @@ def small(cls, *lines: str) -> None: from IPython.display import display # noqa: PLC0415 body = '
'.join(_rich_markup_to_inline_html(line) for line in lines) + except ImportError: # pragma: no cover + pass + else: display( HTML( '
np.ndarray: @pytest.mark.parametrize('proposal_moves', ['de']) def test_emcee_resume_matches_small_dream_posterior(tmp_path, proposal_moves): + from easydiffraction.analysis.minimizers.base import MinimizerFitOptions from easydiffraction.analysis.minimizers.bumps_dream import BumpsDreamMinimizer from easydiffraction.analysis.minimizers.emcee import EmceeMinimizer @@ -76,7 +77,7 @@ def test_emcee_resume_matches_small_dream_posterior(tmp_path, proposal_moves): _toy_parameters(), _array_residuals, verbosity=VerbosityEnum.SILENT, - random_seed=123, + options=MinimizerFitOptions(random_seed=123), ) emcee = EmceeMinimizer() @@ -91,15 +92,13 @@ def test_emcee_resume_matches_small_dream_posterior(tmp_path, proposal_moves): _toy_parameters(), _mapping_residuals, verbosity=VerbosityEnum.SILENT, - random_seed=123, + options=MinimizerFitOptions(random_seed=123), ) resumed_results = emcee.fit( _toy_parameters(), _mapping_residuals, verbosity=VerbosityEnum.SILENT, - random_seed=123, - resume=True, - extra_steps=20, + options=MinimizerFitOptions(random_seed=123, resume=True, extra_steps=20), ) assert dream_results.success is True diff --git a/tests/unit/easydiffraction/analysis/minimizers/test_base.py b/tests/unit/easydiffraction/analysis/minimizers/test_base.py index 4ee99f1d1..50a353d69 100644 --- a/tests/unit/easydiffraction/analysis/minimizers/test_base.py +++ b/tests/unit/easydiffraction/analysis/minimizers/test_base.py @@ -237,6 +237,7 @@ def _check_success(self, raw_result): def test_minimizer_base_stop_tracking_backfills_result_fitting_time(): from easydiffraction.analysis.minimizers.base import MinimizerBase + from easydiffraction.analysis.minimizers.base import MinimizerFitOptions class DummyResult: success = True @@ -280,7 +281,7 @@ def _compute_residuals( result = minimizer.fit( parameters=params, objective_function=objective, - finalize_tracking=False, + options=MinimizerFitOptions(finalize_tracking=False), ) assert result.fitting_time is None diff --git a/tests/unit/easydiffraction/analysis/test_analysis.py b/tests/unit/easydiffraction/analysis/test_analysis.py index 9b124c667..01f54ce0c 100644 --- a/tests/unit/easydiffraction/analysis/test_analysis.py +++ b/tests/unit/easydiffraction/analysis/test_analysis.py @@ -24,6 +24,7 @@ def names(self): class P: experiments = ExpCol(names) structures = object() + info = SimpleNamespace(path=None) _varname = 'proj' return P() @@ -474,6 +475,7 @@ def values(self): def test_fit_single_short_reuses_tracker_display_handle(monkeypatch): from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.fitting import FitterFitOptions from easydiffraction.utils.enums import VerbosityEnum class Handle: @@ -521,20 +523,14 @@ def fake_fit( *, analysis: object, verbosity: object, - use_physical_limits: bool, - random_seed: int | None, - resume: bool, - extra_steps: int | None, + options: FitterFitOptions, ) -> None: del ( structures, experiments, analysis, verbosity, - use_physical_limits, - random_seed, - resume, - extra_steps, + options, ) analysis_obj = fake_fit.analysis_obj analysis_obj.fitter.results = SimpleNamespace( @@ -567,8 +563,7 @@ def fake_update_short_table( VerbosityEnum.SHORT, project.structures, project.experiments, - use_physical_limits=False, - random_seed=None, + fit_options=FitterFitOptions(), ) assert tracker.display_handles == [handle, None] diff --git a/tests/unit/easydiffraction/analysis/test_fitting.py b/tests/unit/easydiffraction/analysis/test_fitting.py index 04b4f20ec..d5995f5ef 100644 --- a/tests/unit/easydiffraction/analysis/test_fitting.py +++ b/tests/unit/easydiffraction/analysis/test_fitting.py @@ -193,7 +193,7 @@ def _finalize_timing(self): verbosity=VerbosityEnum.FULL, ) - assert fitter.minimizer.fit_calls[0]['finalize_tracking'] is False + assert fitter.minimizer.fit_calls[0]['options'].finalize_tracking is False assert fitter.minimizer.stop_calls == 1 assert analysis_events == ['capture', 'store', 'finalize', ('time', 12.5), 'stop'] diff --git a/tests/unit/easydiffraction/analysis/test_sequential.py b/tests/unit/easydiffraction/analysis/test_sequential.py index 2be4c19f3..ac8932df7 100644 --- a/tests/unit/easydiffraction/analysis/test_sequential.py +++ b/tests/unit/easydiffraction/analysis/test_sequential.py @@ -576,6 +576,8 @@ def test_fit_sequential_non_silent_starts_indicator_with_progress_table( tmp_path, verbosity, ): + from easydiffraction.utils.utils import display_path + events = _run_non_silent_fit( monkeypatch, tmp_path, @@ -601,7 +603,11 @@ def test_fit_sequential_non_silent_starts_indicator_with_progress_table( assert events[8] == ('stop',) assert events[9:] == [ ('console_print', ('✅ Sequential fitting complete: 1 files processed.',), {}), - ('console_print', (f'📄 Results saved to:\n{tmp_path / "results.csv"}',), {}), + ( + 'console_print', + (f"📄 Results saved to '{display_path(tmp_path / 'results.csv')}'",), + {}, + ), ] From 963241f4f3e4c088375ccd272634551cc8acf8f8 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 22:38:44 +0200 Subject: [PATCH 53/65] Record emcee unit test verification --- docs/dev/plans/emcee-minimizer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index c8b90ad00..20635cd79 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -753,7 +753,7 @@ required by `AGENTS.md` → **Workflow**. exit $check_exit_code ``` -- [ ] **P2.3 — Unit tests.** +- [x] **P2.3 — Unit tests.** ``` pixi run unit-tests > /tmp/easydiffraction-unit-tests.log 2>&1; \ From 60116cae7caee129a393a575432732bb0556617c Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 22:48:03 +0200 Subject: [PATCH 54/65] Fix emcee result synchronization --- docs/dev/plans/emcee-minimizer.md | 2 +- .../analysis/minimizers/emcee.py | 17 ++++++++--------- tests/integration/fitting/test_emcee.py | 10 +++++++++- 3 files changed, 18 insertions(+), 11 deletions(-) diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index 20635cd79..3c1d77223 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -762,7 +762,7 @@ required by `AGENTS.md` → **Workflow**. exit $unit_tests_exit_code ``` -- [ ] **P2.4 — Integration tests.** +- [x] **P2.4 — Integration tests.** ``` pixi run integration-tests > /tmp/easydiffraction-integration-tests.log 2>&1; \ diff --git a/src/easydiffraction/analysis/minimizers/emcee.py b/src/easydiffraction/analysis/minimizers/emcee.py index 51c129fb9..d3be60b65 100644 --- a/src/easydiffraction/analysis/minimizers/emcee.py +++ b/src/easydiffraction/analysis/minimizers/emcee.py @@ -1202,15 +1202,6 @@ def _sync_result_to_parameters( """ Sync proposed or best posterior values to live parameters. """ - if isinstance(raw_result, dict): - for parameter in parameters: - value = raw_result.get(parameter.unique_name) - if value is None: - value = raw_result.get(getattr(parameter, '_minimizer_uid', '')) - if value is not None: - parameter._set_value_from_minimizer(float(value)) - return - if hasattr(raw_result, 'x'): if getattr(raw_result, 'success', False): values = raw_result.x @@ -1218,6 +1209,14 @@ def _sync_result_to_parameters( else: values = getattr(raw_result, 'starting_values', raw_result.x) uncertainties = getattr(raw_result, 'starting_uncertainties', None) + elif isinstance(raw_result, dict): + for parameter in parameters: + value = raw_result.get(parameter.unique_name) + if value is None: + value = raw_result.get(getattr(parameter, '_minimizer_uid', '')) + if value is not None: + parameter._set_value_from_minimizer(float(value)) + return else: values = raw_result uncertainties = None diff --git a/tests/integration/fitting/test_emcee.py b/tests/integration/fitting/test_emcee.py index 005673e28..3f33739d8 100644 --- a/tests/integration/fitting/test_emcee.py +++ b/tests/integration/fitting/test_emcee.py @@ -36,6 +36,14 @@ def _set_value_from_minimizer(self, value: float) -> None: """Store a value committed by the minimizer.""" self.value = value + def _physical_lower_bound(self) -> float: + """Return the lower physical limit for warning checks.""" + return -np.inf + + def _physical_upper_bound(self) -> float: + """Return the upper physical limit for warning checks.""" + return np.inf + def _toy_parameters() -> list[ToyParameter]: return [ @@ -106,7 +114,7 @@ def test_emcee_resume_matches_small_dream_posterior(tmp_path, proposal_moves): assert resumed_results.success is True assert resumed_results.posterior_samples is not None assert resumed_results.posterior_samples.parameter_samples.shape[1:] == (16, 2) - assert resumed_results.sampler_settings['total_steps'] == 100 + assert resumed_results.sampler_settings['total_steps'] == 121 np.testing.assert_allclose( _posterior_medians(resumed_results), _posterior_medians(dream_results), From d9fd22068b0a52cfc7212b4a265cf0a092a8c59f Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 23:04:28 +0200 Subject: [PATCH 55/65] Add live elapsed time to benchmark runs --- tools/benchmark_tutorials.py | 147 +++++++++++++++++++++++++++++------ 1 file changed, 123 insertions(+), 24 deletions(-) diff --git a/tools/benchmark_tutorials.py b/tools/benchmark_tutorials.py index e28d77e98..87dd380d1 100644 --- a/tools/benchmark_tutorials.py +++ b/tools/benchmark_tutorials.py @@ -8,6 +8,7 @@ import platform import subprocess # noqa: S404 import sys +import tempfile import time from dataclasses import dataclass from datetime import datetime @@ -21,6 +22,12 @@ CHECKPOINT_DIR_NAME = '.ipynb_checkpoints' CSV_HEADER = ['tutorial_name', 'elapsed_seconds', 'status', 'return_code'] +# Layout for the live single-line table. The name column is sized +# from the longest tutorial name encountered. +STATUS_COLUMN_WIDTH = 10 +TIME_COLUMN_WIDTH = 10 # includes trailing 's' +PROGRESS_POLL_SECONDS = 0.2 + @dataclass(frozen=True) class TutorialBenchmarkResult: @@ -75,37 +82,118 @@ def _matches_requested_patterns( return any(rel_path.match(pattern) or script_path.name == pattern for pattern in patterns) +def _format_progress_line( + *, + index: int, + total: int, + name: str, + name_width: int, + status: str, + elapsed_seconds: float, +) -> str: + """Format one tutorial row for the live progress table.""" + index_width = len(str(total)) + counter = f'[{index:>{index_width}}/{total}]' + time_field_width = TIME_COLUMN_WIDTH - 1 # reserve one column for trailing 's' + return ( + f'{counter} ' + f'{name:<{name_width}} ' + f'{status:<{STATUS_COLUMN_WIDTH}} ' + f'{elapsed_seconds:>{time_field_width}.1f}s' + ) + + +def _emit_progress_line(*, line: str, final: bool, is_tty: bool) -> None: + """ + Render one progress line. + + On a TTY the same line is rewritten in place via carriage return so + the elapsed-seconds field updates while the tutorial is running; + only the final write terminates with a newline. When stdout is not + a TTY (CI, log file), only the final row is printed so the log + stays one-line-per-tutorial without carriage-return noise. + """ + if is_tty: + terminator = '\n' if final else '' + sys.stdout.write('\r' + line + terminator) + sys.stdout.flush() + elif final: + print(line) + + def _run_tutorial( script_path: Path, tutorial_dir: Path, env: dict[str, str], + *, + index: int, + total: int, + name_width: int, + is_tty: bool, ) -> TutorialBenchmarkResult: + """Run one tutorial with a live single-line progress indicator.""" tutorial_name = _relative_display_path(script_path, tutorial_dir) start_time = time.perf_counter() - result = subprocess.run( # noqa: S603 - [sys.executable, str(script_path)], - cwd=str(ROOT), - env=env, - capture_output=True, - text=True, - encoding='utf-8', - ) - elapsed_seconds = time.perf_counter() - start_time - status = 'ok' if result.returncode == 0 else 'failed' - if result.returncode == 0: - print(f' OK {elapsed_seconds:.1f}s') - else: - print(f' FAILED {elapsed_seconds:.1f}s', file=sys.stderr) - details = ((result.stdout or '') + (result.stderr or '')).strip() - if details: - print(details, file=sys.stderr) + # Combined stdout+stderr land in a temp file so the OS pipe buffer + # cannot fill up and stall the subprocess while we poll. We only + # read the contents back if the tutorial fails. + with tempfile.TemporaryFile(mode='wb+') as out_file: + proc = subprocess.Popen( # noqa: S603 + [sys.executable, str(script_path)], + cwd=str(ROOT), + env=env, + stdout=out_file, + stderr=subprocess.STDOUT, + ) + + last_displayed_second = -1 + while proc.poll() is None: + elapsed = time.perf_counter() - start_time + current_second = int(elapsed) + if current_second != last_displayed_second: + _emit_progress_line( + line=_format_progress_line( + index=index, + total=total, + name=tutorial_name, + name_width=name_width, + status='Running...', + elapsed_seconds=elapsed, + ), + final=False, + is_tty=is_tty, + ) + last_displayed_second = current_second + time.sleep(PROGRESS_POLL_SECONDS) + + elapsed_seconds = time.perf_counter() - start_time + success = proc.returncode == 0 + status_word = 'OK' if success else 'FAILED' + _emit_progress_line( + line=_format_progress_line( + index=index, + total=total, + name=tutorial_name, + name_width=name_width, + status=status_word, + elapsed_seconds=elapsed_seconds, + ), + final=True, + is_tty=is_tty, + ) + + if not success: + out_file.seek(0) + details = out_file.read().decode('utf-8', errors='replace').strip() + if details: + print(details, file=sys.stderr) return TutorialBenchmarkResult( tutorial_name=tutorial_name, elapsed_seconds=elapsed_seconds, - status=status, - return_code=result.returncode, + status='ok' if success else 'failed', + return_code=proc.returncode, ) @@ -191,11 +279,22 @@ def main() -> int: _write_csv_header(output_path) env = _build_env() + is_tty = sys.stdout.isatty() + name_width = max( + len(_relative_display_path(path, tutorial_dir)) for path in tutorials + ) + results: list[TutorialBenchmarkResult] = [] for index, tutorial_path in enumerate(tutorials, start=1): - tutorial_name = _relative_display_path(tutorial_path, tutorial_dir) - print(f'[{index:2}/{len(tutorials)}] Running {tutorial_name}') - result = _run_tutorial(tutorial_path, tutorial_dir, env) + result = _run_tutorial( + tutorial_path, + tutorial_dir, + env, + index=index, + total=len(tutorials), + name_width=name_width, + is_tty=is_tty, + ) results.append(result) _append_result(output_path, result) @@ -203,7 +302,7 @@ def main() -> int: failure_count = sum(result.status == 'failed' for result in results) print(f'Wrote benchmark results to {_relative_display_path(output_path, ROOT)}') - print(f'Total elapsed time: {total_elapsed:.3f}s') + print(f'Total elapsed time: {total_elapsed:.1f}s') if failure_count: print(f'Failed tutorials: {failure_count}', file=sys.stderr) @@ -213,4 +312,4 @@ def main() -> int: if __name__ == '__main__': - raise SystemExit(main()) \ No newline at end of file + raise SystemExit(main()) From 3a36a3e61877a3f66e3d9c0176c00594b6402ef4 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 23:12:18 +0200 Subject: [PATCH 56/65] Isolate sequential tutorial project path --- docs/dev/plans/emcee-minimizer.md | 2 +- docs/docs/tutorials/ed-13.ipynb | 2 +- docs/docs/tutorials/ed-17.ipynb | 2 +- docs/docs/tutorials/ed-17.py | 2 +- docs/docs/tutorials/ed-26.ipynb | 18 ++++++++++++------ 5 files changed, 16 insertions(+), 10 deletions(-) diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index 3c1d77223..3008105bc 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -771,7 +771,7 @@ required by `AGENTS.md` → **Workflow**. exit $integration_tests_exit_code ``` -- [ ] **P2.5 — Script tests.** +- [x] **P2.5 — Script tests.** ``` pixi run script-tests > /tmp/easydiffraction-script-tests.log 2>&1; \ diff --git a/docs/docs/tutorials/ed-13.ipynb b/docs/docs/tutorials/ed-13.ipynb index 36b56afea..a99270afa 100644 --- a/docs/docs/tutorials/ed-13.ipynb +++ b/docs/docs/tutorials/ed-13.ipynb @@ -2657,7 +2657,7 @@ ], "metadata": { "jupytext": { - "cell_metadata_filter": "tags,title,-all", + "cell_metadata_filter": "title,tags,-all", "main_language": "python", "notebook_metadata_filter": "-all" } diff --git a/docs/docs/tutorials/ed-17.ipynb b/docs/docs/tutorials/ed-17.ipynb index 89f0488ca..8d7d9d2be 100644 --- a/docs/docs/tutorials/ed-17.ipynb +++ b/docs/docs/tutorials/ed-17.ipynb @@ -90,7 +90,7 @@ "metadata": {}, "outputs": [], "source": [ - "project.save_as(dir_path='projects/cosio_d20')" + "project.save_as(dir_path='projects/cosio_d20_scan')" ] }, { diff --git a/docs/docs/tutorials/ed-17.py b/docs/docs/tutorials/ed-17.py index 3f8f9b1ed..0622b2310 100644 --- a/docs/docs/tutorials/ed-17.py +++ b/docs/docs/tutorials/ed-17.py @@ -29,7 +29,7 @@ # results can be written to `analysis/results.csv`. # %% -project.save_as(dir_path='projects/cosio_d20') +project.save_as(dir_path='projects/cosio_d20_scan') # %% [markdown] # ## Step 2: Define Crystal Structure diff --git a/docs/docs/tutorials/ed-26.ipynb b/docs/docs/tutorials/ed-26.ipynb index 1740dfcdb..7e14253de 100644 --- a/docs/docs/tutorials/ed-26.ipynb +++ b/docs/docs/tutorials/ed-26.ipynb @@ -27,14 +27,18 @@ "# Bayesian Analysis Resume (`emcee`): LBCO, HRPT\n", "\n", "This tutorial shows how to reopen the Bayesian project created previously,\n", - "inspect the saved fit results and then run more sampling steps to extend the existing chain.\n", - "Resuming only works with EMCEE because the current BUMPS-DREAM implementation does not support\n", - "saving and resuming its state.\n", + "inspect the saved fit results and then run more sampling steps to\n", + "extend the existing chain. Resuming only works with EMCEE because the\n", + "current BUMPS-DREAM implementation does not support saving and\n", + "resuming its state.\n", "\n", "This workflow is useful when:\n", "- the initial sampling run has not yet converged and more steps are needed,\n", - "- the initial sampling run has converged but more steps are desired for better posterior resolution,\n", - "- the initial sampling run has converged but the posterior plots have not yet been inspected and the user wants to see the plots before deciding whether to run more steps.\n", + "- the initial sampling run has converged but more steps are desired\n", + " for better posterior resolution,\n", + "- the initial sampling run has converged but the posterior plots have\n", + " not yet been inspected and the user wants to see the plots before\n", + " deciding whether to run more steps.\n", "\n", "The workflow uses the same La0.5Ba0.5CoO3 powder diffraction example\n", "as the DREAM Bayesian tutorial:\n", @@ -231,7 +235,9 @@ "## Resume emcee Sampling\n", "\n", "Resume from the saved backend and append 100 more emcee steps to the\n", - "existing chain. We use only 100 steps here to keep the tutorial fast, but in practice you would typically run more steps to ensure convergence and better posterior resolution." + "existing chain. We use only 100 steps here to keep the tutorial fast,\n", + "but in practice you would typically run more steps to ensure\n", + "convergence and better posterior resolution." ] }, { From 36b4243767e241053c6f6e372fe0def4646bbf79 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 23:34:34 +0200 Subject: [PATCH 57/65] Document emcee fit options refinement --- docs/dev/plans/emcee-minimizer.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index 3008105bc..909b93717 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -143,6 +143,13 @@ reconciliation), stop and ask before proceeding. This mirrors the existing `BumpsDreamMinimizer` split between `categories/minimizer/bumps_dream.py` and `minimizers/bumps_dream.py`. +8. Phase 2 resolved the `PLR0913` signature-width check by replacing + the internal explicit execution keyword set on `MinimizerBase.fit` + and `Fitter.fit` with `MinimizerFitOptions` and `FitterFitOptions` + parameter objects. `Analysis.fit(resume=True, extra_steps=N)` + remains the user-facing API. This is a deliberate refinement of P1.5 + that follows `AGENTS.md`'s refactor guidance instead of raising + thresholds or suppressing the lint rule. ## Open questions From 82e20a26afc681a15f728714239edc0cc464cac9 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 23:34:57 +0200 Subject: [PATCH 58/65] Record emcee Phase 2 bug-fix note --- docs/dev/plans/emcee-minimizer.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index 909b93717..c0399807d 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -787,6 +787,14 @@ required by `AGENTS.md` → **Workflow**. exit $script_tests_exit_code ``` +## Phase 2 review notes + +- Phase 2 bug fixes found by verification should be committed separately + from checklist updates. The result-synchronization fix is retained, + but a future equivalent commit should describe the behavior change + directly, for example: reorder result-sync branches so + `OptimizeResult` takes the `.x` path. + ## Suggested Pull Request **Title:** Add emcee Bayesian sampler with resumable runs From 61b9ab4e6ff13db82f593b760e8c8cd44e4e01b9 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 23:35:16 +0200 Subject: [PATCH 59/65] Document emcee benchmark tooling note --- docs/dev/plans/emcee-minimizer.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index c0399807d..9ed503067 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -825,3 +825,6 @@ is straightforward. New tutorials walk through a short run (`ed-25`) and reopening the saved project to resume for additional steps (`ed-26`). + +Phase 2 also includes a small benchmark-runner elapsed-time display +improvement that landed alongside the emcee verification work. From c30488587c67a9202e9c6c08c3938223c56bb511 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 23:37:27 +0200 Subject: [PATCH 60/65] Expand emcee minimizer category tests --- .../categories/minimizer/test_emcee.py | 105 ++++++++++++++++++ 1 file changed, 105 insertions(+) diff --git a/tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py b/tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py index 1b38d65aa..5965657fc 100644 --- a/tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py +++ b/tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py @@ -4,6 +4,24 @@ from __future__ import annotations +from collections.abc import Iterator +from types import SimpleNamespace + + +class _ExperimentCollection: + @property + def names(self) -> list[str]: + return [] + + +def _make_project() -> object: + return SimpleNamespace( + experiments=_ExperimentCollection(), + structures=object(), + info=SimpleNamespace(path=None), + _varname='proj', + ) + def test_emcee_minimizer_category_defaults_to_max_parallel_workers(): from easydiffraction.analysis.categories.minimizer.emcee import ( @@ -35,3 +53,90 @@ def test_emcee_minimizer_category_defaults_to_de_without_thinning(): assert minimizer.thinning_interval.value == 1 assert minimizer._native_kwargs()['proposal_moves'] == 'de' assert minimizer._native_kwargs()['thin'] == 1 + + +def test_emcee_minimizer_category_maps_native_kwargs(): + from easydiffraction.analysis.categories.minimizer.emcee import ( + DEFAULT_BURN_IN_STEPS, + ) + from easydiffraction.analysis.categories.minimizer.emcee import ( + DEFAULT_INITIALIZATION_METHOD, + ) + from easydiffraction.analysis.categories.minimizer.emcee import ( + DEFAULT_PARALLEL_WORKERS, + ) + from easydiffraction.analysis.categories.minimizer.emcee import ( + DEFAULT_POPULATION_SIZE, + ) + from easydiffraction.analysis.categories.minimizer.emcee import ( + DEFAULT_PROPOSAL_MOVES, + ) + from easydiffraction.analysis.categories.minimizer.emcee import ( + DEFAULT_SAMPLING_STEPS, + ) + from easydiffraction.analysis.categories.minimizer.emcee import ( + DEFAULT_THINNING_INTERVAL, + ) + from easydiffraction.analysis.categories.minimizer.emcee import EmceeMinimizer + + native_kwargs = EmceeMinimizer()._native_kwargs() + + assert native_kwargs == { + 'nsteps': DEFAULT_SAMPLING_STEPS, + 'nburn': DEFAULT_BURN_IN_STEPS, + 'thin': DEFAULT_THINNING_INTERVAL, + 'nwalkers': DEFAULT_POPULATION_SIZE, + 'parallel_workers': DEFAULT_PARALLEL_WORKERS, + 'initialization_method': DEFAULT_INITIALIZATION_METHOD.value, + 'random_seed': None, + 'proposal_moves': DEFAULT_PROPOSAL_MOVES, + } + assert 'steps' not in native_kwargs + assert 'burn' not in native_kwargs + assert 'pop' not in native_kwargs + assert 'init' not in native_kwargs + + +def test_emcee_minimizer_swap_pairs_bayesian_fit_result(): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.categories.fit_result.bayesian import BayesianFitResult + + analysis = Analysis(project=_make_project()) + + analysis.minimizer.type = 'emcee' + + assert isinstance(analysis.fit_result, BayesianFitResult) + assert analysis.fit_result._parent is analysis + + +def test_emcee_resume_parameter_set_mismatch_raises_before_sampler(): + import pytest + + from easydiffraction.analysis.fitting import Fitter + from easydiffraction.analysis.fitting import FitterFitOptions + + class Structures: + def __init__(self) -> None: + self.free_parameters = [SimpleNamespace(unique_name='current.param')] + + def __iter__(self) -> Iterator[object]: + return iter([self]) + + def _update_categories(self) -> None: + pass + + analysis = SimpleNamespace( + fit_parameters=[ + SimpleNamespace( + param_unique_name=SimpleNamespace(value='saved.param'), + ), + ], + ) + + with pytest.raises(ValueError, match='Resume parameter set differs'): + Fitter('emcee').fit( + structures=Structures(), + experiments=[], + analysis=analysis, + options=FitterFitOptions(resume=True), + ) From 59083e6c9bf5835519e59a726a223b16fe5a9c9c Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Mon, 25 May 2026 23:37:53 +0200 Subject: [PATCH 61/65] Document emcee tutorial verification note --- docs/dev/plans/emcee-minimizer.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index 9ed503067..8e64652cb 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -794,6 +794,10 @@ required by `AGENTS.md` → **Workflow**. but a future equivalent commit should describe the behavior change directly, for example: reorder result-sync branches so `OptimizeResult` takes the `.x` path. +- Script-test fixes should also be split from checklist-only commits + when practical. The `ed-17` save-path change is retained because it + avoids a project-path collision with `ed-5`; notebook metadata + reordering came from the required `notebook-prepare` regeneration. ## Suggested Pull Request From 7adca53ae2003cadecd9693b5bdb7d5944387ab3 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Tue, 26 May 2026 00:00:09 +0200 Subject: [PATCH 62/65] Improve benchmark tutorial progress table --- docs/dev/plans/emcee-minimizer.md | 14 +-- .../categories/minimizer/test_emcee.py | 5 +- tests/unit/test_benchmark_tutorials.py | 14 ++- tools/benchmark_tutorials.py | 89 ++++++++++++++----- 4 files changed, 85 insertions(+), 37 deletions(-) diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md index 8e64652cb..7bd2bfa8a 100644 --- a/docs/dev/plans/emcee-minimizer.md +++ b/docs/dev/plans/emcee-minimizer.md @@ -143,13 +143,13 @@ reconciliation), stop and ask before proceeding. This mirrors the existing `BumpsDreamMinimizer` split between `categories/minimizer/bumps_dream.py` and `minimizers/bumps_dream.py`. -8. Phase 2 resolved the `PLR0913` signature-width check by replacing - the internal explicit execution keyword set on `MinimizerBase.fit` - and `Fitter.fit` with `MinimizerFitOptions` and `FitterFitOptions` - parameter objects. `Analysis.fit(resume=True, extra_steps=N)` - remains the user-facing API. This is a deliberate refinement of P1.5 - that follows `AGENTS.md`'s refactor guidance instead of raising - thresholds or suppressing the lint rule. +8. Phase 2 resolved the `PLR0913` signature-width check by replacing the + internal explicit execution keyword set on `MinimizerBase.fit` and + `Fitter.fit` with `MinimizerFitOptions` and `FitterFitOptions` + parameter objects. `Analysis.fit(resume=True, extra_steps=N)` remains + the user-facing API. This is a deliberate refinement of P1.5 that + follows `AGENTS.md`'s refactor guidance instead of raising thresholds + or suppressing the lint rule. ## Open questions diff --git a/tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py b/tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py index 5965657fc..9f4e904b0 100644 --- a/tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py +++ b/tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py @@ -4,8 +4,11 @@ from __future__ import annotations -from collections.abc import Iterator from types import SimpleNamespace +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from collections.abc import Iterator class _ExperimentCollection: diff --git a/tests/unit/test_benchmark_tutorials.py b/tests/unit/test_benchmark_tutorials.py index c0b390ef7..9688bddda 100644 --- a/tests/unit/test_benchmark_tutorials.py +++ b/tests/unit/test_benchmark_tutorials.py @@ -36,7 +36,6 @@ def test_append_result_writes_one_row(tmp_path): tutorial_name='ed-21.py', elapsed_seconds=12.3456, status='ok', - return_code=0, ), ) @@ -48,7 +47,6 @@ def test_append_result_writes_one_row(tmp_path): 'tutorial_name': 'ed-21.py', 'elapsed_seconds': '12.346', 'status': 'ok', - 'return_code': '0', } ] @@ -80,8 +78,13 @@ def fake_run_tutorial( script_path: Path, tutorial_dir_path: Path, env: dict[str, str], + *, + index: int, + total: int, + name_width: int, + is_tty: bool, ) -> MUT.TutorialBenchmarkResult: - del tutorial_dir_path, env + del tutorial_dir_path, env, index, total, name_width, is_tty if script_path == first_tutorial: with output_path.open(encoding='utf-8', newline='') as handle: rows = list(csv.reader(handle)) @@ -90,7 +93,6 @@ def fake_run_tutorial( tutorial_name='ed-01.py', elapsed_seconds=1.0, status='ok', - return_code=0, ) with output_path.open(encoding='utf-8', newline='') as handle: @@ -100,14 +102,12 @@ def fake_run_tutorial( 'tutorial_name': 'ed-01.py', 'elapsed_seconds': '1.000', 'status': 'ok', - 'return_code': '0', } ] return MUT.TutorialBenchmarkResult( tutorial_name='ed-02.py', elapsed_seconds=2.0, status='ok', - return_code=0, ) monkeypatch.setattr(MUT, '_run_tutorial', fake_run_tutorial) @@ -122,12 +122,10 @@ def fake_run_tutorial( 'tutorial_name': 'ed-01.py', 'elapsed_seconds': '1.000', 'status': 'ok', - 'return_code': '0', }, { 'tutorial_name': 'ed-02.py', 'elapsed_seconds': '2.000', 'status': 'ok', - 'return_code': '0', }, ] diff --git a/tools/benchmark_tutorials.py b/tools/benchmark_tutorials.py index 87dd380d1..a43932b4a 100644 --- a/tools/benchmark_tutorials.py +++ b/tools/benchmark_tutorials.py @@ -6,6 +6,7 @@ import csv import os import platform +import re import subprocess # noqa: S404 import sys import tempfile @@ -15,12 +16,14 @@ from pathlib import Path from pathlib import PurePosixPath +SECONDS_PER_MINUTE = 60 + ROOT = Path(__file__).resolve().parents[1] SRC_ROOT = ROOT / 'src' DEFAULT_TUTORIAL_DIR = ROOT / 'docs' / 'docs' / 'tutorials' DEFAULT_OUTPUT_DIR = ROOT / 'docs' / 'dev' / 'benchmarking' CHECKPOINT_DIR_NAME = '.ipynb_checkpoints' -CSV_HEADER = ['tutorial_name', 'elapsed_seconds', 'status', 'return_code'] +CSV_HEADER = ['tutorial_name', 'elapsed_seconds', 'status'] # Layout for the live single-line table. The name column is sized # from the longest tutorial name encountered. @@ -36,7 +39,6 @@ class TutorialBenchmarkResult: tutorial_name: str elapsed_seconds: float status: str - return_code: int def _relative_display_path(path: Path, start_path: Path) -> str: @@ -62,14 +64,58 @@ def _build_env() -> dict[str, str]: return env +def _natural_sort_key(path: Path) -> list[object]: + """ + Return a sort key for natural ordering. + + Splits the path string into alternating text and integer tokens + so ``ed-1.py``, ``ed-2.py``, ``ed-10.py`` sort numerically + instead of lexicographically. + """ + return [ + int(token) if token.isdigit() else token.lower() + for token in re.split(r'(\d+)', str(path)) + ] + + def _discover_tutorials(tutorial_dir: Path) -> list[Path]: return [ path - for path in sorted(tutorial_dir.rglob('*.py')) + for path in sorted(tutorial_dir.rglob('*.py'), key=_natural_sort_key) if CHECKPOINT_DIR_NAME not in path.parts ] +def _format_elapsed(seconds: float) -> str: + """ + Format elapsed time for table display. + + Rounds to the nearest whole second. Values that round to less + than one minute render as ``Xs`` (e.g. ``20s``); values that + round to a minute or more render as ``Xm Ys`` (e.g. ``1m 18s``, + ``20m 1s``). + """ + total_seconds = int(round(seconds)) + if total_seconds < SECONDS_PER_MINUTE: + return f'{total_seconds}s' + minutes, remaining_seconds = divmod(total_seconds, SECONDS_PER_MINUTE) + return f'{minutes}m {remaining_seconds:>2}s' + + +def _format_counter(index: int | None, total: int) -> str: + """ + Format the ``[n/N]`` counter cell for the progress table. + + Returns a blank string of the same width when *index* is + ``None`` so the total-time summary row aligns with the data + rows. + """ + width = len(str(total)) + if index is None: + return ' ' * (width * 2 + 3) # length of '[n/N]' + return f'[{index:>{width}}/{total}]' + + def _matches_requested_patterns( script_path: Path, tutorial_dir: Path, @@ -84,22 +130,18 @@ def _matches_requested_patterns( def _format_progress_line( *, - index: int, - total: int, + counter: str, name: str, name_width: int, status: str, - elapsed_seconds: float, + elapsed_text: str, ) -> str: - """Format one tutorial row for the live progress table.""" - index_width = len(str(total)) - counter = f'[{index:>{index_width}}/{total}]' - time_field_width = TIME_COLUMN_WIDTH - 1 # reserve one column for trailing 's' + """Format one row for the live progress table.""" return ( f'{counter} ' f'{name:<{name_width}} ' f'{status:<{STATUS_COLUMN_WIDTH}} ' - f'{elapsed_seconds:>{time_field_width}.1f}s' + f'{elapsed_text:>{TIME_COLUMN_WIDTH}}' ) @@ -147,6 +189,7 @@ def _run_tutorial( stderr=subprocess.STDOUT, ) + counter = _format_counter(index, total) last_displayed_second = -1 while proc.poll() is None: elapsed = time.perf_counter() - start_time @@ -154,12 +197,11 @@ def _run_tutorial( if current_second != last_displayed_second: _emit_progress_line( line=_format_progress_line( - index=index, - total=total, + counter=counter, name=tutorial_name, name_width=name_width, status='Running...', - elapsed_seconds=elapsed, + elapsed_text=_format_elapsed(elapsed), ), final=False, is_tty=is_tty, @@ -172,12 +214,11 @@ def _run_tutorial( status_word = 'OK' if success else 'FAILED' _emit_progress_line( line=_format_progress_line( - index=index, - total=total, + counter=counter, name=tutorial_name, name_width=name_width, status=status_word, - elapsed_seconds=elapsed_seconds, + elapsed_text=_format_elapsed(elapsed_seconds), ), final=True, is_tty=is_tty, @@ -193,7 +234,6 @@ def _run_tutorial( tutorial_name=tutorial_name, elapsed_seconds=elapsed_seconds, status='ok' if success else 'failed', - return_code=proc.returncode, ) @@ -224,7 +264,6 @@ def _append_result(output_path: Path, result: TutorialBenchmarkResult) -> None: result.tutorial_name, f'{result.elapsed_seconds:.3f}', result.status, - result.return_code, ] ) @@ -301,8 +340,16 @@ def main() -> int: total_elapsed = sum(result.elapsed_seconds for result in results) failure_count = sum(result.status == 'failed' for result in results) - print(f'Wrote benchmark results to {_relative_display_path(output_path, ROOT)}') - print(f'Total elapsed time: {total_elapsed:.1f}s') + print( + _format_progress_line( + counter=_format_counter(None, len(tutorials)), + name='Total', + name_width=name_width, + status='', + elapsed_text=_format_elapsed(total_elapsed), + ) + ) + print(f'Save results to: {_relative_display_path(output_path, ROOT)}') if failure_count: print(f'Failed tutorials: {failure_count}', file=sys.stderr) From d8d5733a782442c655afa035f3f048ec9f37b5a6 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Tue, 26 May 2026 00:06:02 +0200 Subject: [PATCH 63/65] Update tutorial benchmarks for emcee minimizer --- ...darwin-arm64_py314_tutorial-benchmarks.csv | 48 +++++++++---------- ...darwin-arm64_py314_tutorial-benchmarks.csv | 48 +++++++++---------- ...darwin-arm64_py314_tutorial-benchmarks.csv | 48 +++++++++---------- ...darwin-arm64_py314_tutorial-benchmarks.csv | 26 ++++++++++ 4 files changed, 98 insertions(+), 72 deletions(-) create mode 100644 docs/dev/benchmarking/20260525-231754_darwin-arm64_py314_tutorial-benchmarks.csv diff --git a/docs/dev/benchmarking/20260519-103251_darwin-arm64_py314_tutorial-benchmarks.csv b/docs/dev/benchmarking/20260519-103251_darwin-arm64_py314_tutorial-benchmarks.csv index edb8e3f63..24cedfe9c 100644 --- a/docs/dev/benchmarking/20260519-103251_darwin-arm64_py314_tutorial-benchmarks.csv +++ b/docs/dev/benchmarking/20260519-103251_darwin-arm64_py314_tutorial-benchmarks.csv @@ -1,24 +1,24 @@ -tutorial_name,elapsed_seconds,status,return_code -ed-1.py,13.789,ok,0 -ed-10.py,39.098,ok,0 -ed-11.py,10.343,ok,0 -ed-12.py,8.331,ok,0 -ed-13.py,21.745,ok,0 -ed-14.py,6.158,ok,0 -ed-15.py,240.151,ok,0 -ed-16.py,58.481,ok,0 -ed-17.py,152.214,ok,0 -ed-18.py,6.114,ok,0 -ed-2.py,18.322,ok,0 -ed-20.py,36.422,ok,0 -ed-21.py,197.610,ok,0 -ed-22.py,194.351,ok,0 -ed-23.py,6.060,ok,0 -ed-24.py,4.749,ok,0 -ed-3.py,19.050,ok,0 -ed-4.py,4.480,ok,0 -ed-5.py,36.605,ok,0 -ed-6.py,61.846,ok,0 -ed-7.py,115.147,ok,0 -ed-8.py,101.442,ok,0 -ed-9.py,9.214,ok,0 +tutorial_name,elapsed_seconds,status +ed-1.py,13.789,ok +ed-2.py,18.322,ok +ed-3.py,19.050,ok +ed-4.py,4.480,ok +ed-5.py,36.605,ok +ed-6.py,61.846,ok +ed-7.py,115.147,ok +ed-8.py,101.442,ok +ed-9.py,9.214,ok +ed-10.py,39.098,ok +ed-11.py,10.343,ok +ed-12.py,8.331,ok +ed-13.py,21.745,ok +ed-14.py,6.158,ok +ed-15.py,240.151,ok +ed-16.py,58.481,ok +ed-17.py,152.214,ok +ed-18.py,6.114,ok +ed-20.py,36.422,ok +ed-21.py,197.610,ok +ed-22.py,194.351,ok +ed-23.py,6.060,ok +ed-24.py,4.749,ok diff --git a/docs/dev/benchmarking/20260519-103500_darwin-arm64_py314_tutorial-benchmarks.csv b/docs/dev/benchmarking/20260519-103500_darwin-arm64_py314_tutorial-benchmarks.csv index 608944cb5..1658c45aa 100644 --- a/docs/dev/benchmarking/20260519-103500_darwin-arm64_py314_tutorial-benchmarks.csv +++ b/docs/dev/benchmarking/20260519-103500_darwin-arm64_py314_tutorial-benchmarks.csv @@ -1,24 +1,24 @@ -tutorial_name,elapsed_seconds,status,return_code -ed-1.py,13.979,ok,0 -ed-10.py,38.764,ok,0 -ed-11.py,10.606,ok,0 -ed-12.py,9.044,ok,0 -ed-13.py,23.157,ok,0 -ed-14.py,6.585,ok,0 -ed-15.py,258.188,ok,0 -ed-16.py,60.097,ok,0 -ed-17.py,69.418,ok,0 -ed-18.py,6.181,ok,0 -ed-2.py,18.844,ok,0 -ed-20.py,39.157,ok,0 -ed-21.py,96.730,ok,0 -ed-22.py,73.480,ok,0 -ed-23.py,5.984,ok,0 -ed-24.py,4.942,ok,0 -ed-3.py,20.782,ok,0 -ed-4.py,5.780,ok,0 -ed-5.py,37.716,ok,0 -ed-6.py,66.911,ok,0 -ed-7.py,119.645,ok,0 -ed-8.py,103.887,ok,0 -ed-9.py,8.891,ok,0 +tutorial_name,elapsed_seconds,status +ed-1.py,13.979,ok +ed-2.py,18.844,ok +ed-3.py,20.782,ok +ed-4.py,5.780,ok +ed-5.py,37.716,ok +ed-6.py,66.911,ok +ed-7.py,119.645,ok +ed-8.py,103.887,ok +ed-9.py,8.891,ok +ed-10.py,38.764,ok +ed-11.py,10.606,ok +ed-12.py,9.044,ok +ed-13.py,23.157,ok +ed-14.py,6.585,ok +ed-15.py,258.188,ok +ed-16.py,60.097,ok +ed-17.py,69.418,ok +ed-18.py,6.181,ok +ed-20.py,39.157,ok +ed-21.py,96.730,ok +ed-22.py,73.480,ok +ed-23.py,5.984,ok +ed-24.py,4.942,ok diff --git a/docs/dev/benchmarking/20260519-121524_darwin-arm64_py314_tutorial-benchmarks.csv b/docs/dev/benchmarking/20260519-121524_darwin-arm64_py314_tutorial-benchmarks.csv index ff8eda19a..11101c797 100644 --- a/docs/dev/benchmarking/20260519-121524_darwin-arm64_py314_tutorial-benchmarks.csv +++ b/docs/dev/benchmarking/20260519-121524_darwin-arm64_py314_tutorial-benchmarks.csv @@ -1,24 +1,24 @@ -tutorial_name,elapsed_seconds,status,return_code -ed-1.py,15.557,ok,0 -ed-10.py,40.860,ok,0 -ed-11.py,10.823,ok,0 -ed-12.py,8.861,ok,0 -ed-13.py,24.128,ok,0 -ed-14.py,6.722,ok,0 -ed-15.py,28.243,ok,0 -ed-16.py,59.218,ok,0 -ed-17.py,70.816,ok,0 -ed-18.py,6.944,ok,0 -ed-2.py,20.385,ok,0 -ed-20.py,39.513,ok,0 -ed-21.py,96.953,ok,0 -ed-22.py,75.390,ok,0 -ed-23.py,6.115,ok,0 -ed-24.py,5.159,ok,0 -ed-3.py,34.082,ok,0 -ed-4.py,8.215,ok,0 -ed-5.py,61.949,ok,0 -ed-6.py,83.857,ok,0 -ed-7.py,120.332,ok,0 -ed-8.py,103.831,ok,0 -ed-9.py,9.270,ok,0 +tutorial_name,elapsed_seconds,status +ed-1.py,15.557,ok +ed-2.py,20.385,ok +ed-3.py,34.082,ok +ed-4.py,8.215,ok +ed-5.py,61.949,ok +ed-6.py,83.857,ok +ed-7.py,120.332,ok +ed-8.py,103.831,ok +ed-9.py,9.270,ok +ed-10.py,40.860,ok +ed-11.py,10.823,ok +ed-12.py,8.861,ok +ed-13.py,24.128,ok +ed-14.py,6.722,ok +ed-15.py,28.243,ok +ed-16.py,59.218,ok +ed-17.py,70.816,ok +ed-18.py,6.944,ok +ed-20.py,39.513,ok +ed-21.py,96.953,ok +ed-22.py,75.390,ok +ed-23.py,6.115,ok +ed-24.py,5.159,ok diff --git a/docs/dev/benchmarking/20260525-231754_darwin-arm64_py314_tutorial-benchmarks.csv b/docs/dev/benchmarking/20260525-231754_darwin-arm64_py314_tutorial-benchmarks.csv new file mode 100644 index 000000000..0f0c8fab0 --- /dev/null +++ b/docs/dev/benchmarking/20260525-231754_darwin-arm64_py314_tutorial-benchmarks.csv @@ -0,0 +1,26 @@ +tutorial_name,elapsed_seconds,status +ed-1.py,20.204,ok +ed-2.py,25.471,ok +ed-3.py,25.688,ok +ed-4.py,5.507,ok +ed-5.py,53.209,ok +ed-6.py,83.848,ok +ed-7.py,155.200,ok +ed-8.py,160.673,ok +ed-9.py,10.812,ok +ed-10.py,48.118,ok +ed-11.py,13.865,ok +ed-12.py,12.846,ok +ed-13.py,33.593,ok +ed-14.py,8.157,ok +ed-15.py,36.489,ok +ed-16.py,77.915,ok +ed-17.py,98.729,ok +ed-18.py,10.579,ok +ed-20.py,49.750,ok +ed-21.py,95.431,ok +ed-22.py,63.560,ok +ed-23.py,28.869,ok +ed-24.py,6.105,ok +ed-25.py,39.325,ok +ed-26.py,37.368,ok From 52917b8903d7aa8b9a1ae82ecb79047d60b9ce9d Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Tue, 26 May 2026 00:10:53 +0200 Subject: [PATCH 64/65] Clean up --- .../minimizer-category-consolidation.md | 43 +- .../switchable-category-owned-selectors.md | 29 +- docs/dev/plans/emcee-minimizer.md | 834 ------------------ .../dev/plans/minimizer-input-output-split.md | 673 -------------- 4 files changed, 45 insertions(+), 1534 deletions(-) delete mode 100644 docs/dev/plans/emcee-minimizer.md delete mode 100644 docs/dev/plans/minimizer-input-output-split.md diff --git a/docs/dev/adrs/accepted/minimizer-category-consolidation.md b/docs/dev/adrs/accepted/minimizer-category-consolidation.md index 7d0df4e1f..552b7b1dd 100644 --- a/docs/dev/adrs/accepted/minimizer-category-consolidation.md +++ b/docs/dev/adrs/accepted/minimizer-category-consolidation.md @@ -304,6 +304,11 @@ swap warnings. ### 9. Example CIF layouts +The fit-result outputs in these examples live under `_fit_result.*` +per +[`minimizer-input-output-split.md`](minimizer-input-output-split.md); +`_minimizer.*` carries only user-writable settings. + `bumps (lm)`: ``` @@ -313,10 +318,12 @@ _fitting.mode_type joint _fitting.minimizer_type 'bumps (lm)' _minimizer.max_iterations 200 -_minimizer.runtime_seconds 12.34 -_minimizer.iterations_performed 87 -_minimizer.exit_reason converged -_minimizer.reduced_chi2 1.42 + +_fit_result.result_kind deterministic +_fit_result.fitting_time 12.34 +_fit_result.iterations 87 +_fit_result.exit_reason converged +_fit_result.reduced_chi_square 1.42 ``` `bumps (dream)`: @@ -334,12 +341,14 @@ _minimizer.population_size 4 _minimizer.parallel_workers 0 _minimizer.initialization_method latin_hypercube _minimizer.random_seed ? -_minimizer.runtime_seconds 124.7 -_minimizer.acceptance_rate_mean 0.27 -_minimizer.gelman_rubin_max 1.03 -_minimizer.effective_sample_size_min 482 -_minimizer.best_log_posterior -1234.56 -_minimizer.reduced_chi2 1.18 + +_fit_result.result_kind bayesian +_fit_result.fitting_time 124.7 +_fit_result.reduced_chi_square 1.18 +_fit_result.acceptance_rate_mean 0.27 +_fit_result.gelman_rubin_max 1.03 +_fit_result.effective_sample_size_min 482 +_fit_result.best_log_posterior -1234.56 ``` `emcee` (added by the follow-up plan): @@ -358,12 +367,14 @@ _minimizer.proposal_moves de _minimizer.parallel_workers 0 _minimizer.initialization_method ball _minimizer.random_seed 42 -_minimizer.runtime_seconds 87.3 -_minimizer.acceptance_rate_mean 0.31 -_minimizer.gelman_rubin_max 1.02 -_minimizer.effective_sample_size_min 612 -_minimizer.best_log_posterior -1237.89 -_minimizer.reduced_chi2 1.22 + +_fit_result.result_kind bayesian +_fit_result.fitting_time 87.3 +_fit_result.reduced_chi_square 1.22 +_fit_result.acceptance_rate_mean 0.31 +_fit_result.gelman_rubin_max 1.02 +_fit_result.effective_sample_size_min 612 +_fit_result.best_log_posterior -1237.89 ``` emcee's resumable chain state lives in the `/emcee_chain` group of the diff --git a/docs/dev/adrs/accepted/switchable-category-owned-selectors.md b/docs/dev/adrs/accepted/switchable-category-owned-selectors.md index 8d39fbfcd..9207992e5 100644 --- a/docs/dev/adrs/accepted/switchable-category-owned-selectors.md +++ b/docs/dev/adrs/accepted/switchable-category-owned-selectors.md @@ -981,16 +981,21 @@ _minimizer.population_size 4 _minimizer.parallel_workers 0 _minimizer.initialization_method latin_hypercube _minimizer.random_seed ? -_minimizer.runtime_seconds 124.7 -_minimizer.acceptance_rate_mean 0.27 -_minimizer.gelman_rubin_max 1.03 -_minimizer.effective_sample_size_min 482 -_minimizer.best_log_posterior -1234.56 -_minimizer.reduced_chi2 1.18 + +_fit_result.result_kind bayesian +_fit_result.fitting_time 124.7 +_fit_result.reduced_chi_square 1.18 +_fit_result.acceptance_rate_mean 0.27 +_fit_result.gelman_rubin_max 1.03 +_fit_result.effective_sample_size_min 482 +_fit_result.best_log_posterior -1234.56 ``` The `_fitting.*` block is gone (`_fitting.minimizer_type` → `_minimizer.type`; `_fitting.mode_type` → `_fitting_mode.type`). +Fit-result outputs live under `_fit_result.*` per +[`minimizer-input-output-split.md`](minimizer-input-output-split.md); +`_minimizer.*` carries only user-writable settings. ### `analysis.cif` (deterministic fit) @@ -1001,11 +1006,13 @@ _fitting_mode.type single _minimizer.type 'lmfit (leastsq)' _minimizer.max_iterations 1000 -_minimizer.objective_value 1532.4 -_minimizer.runtime_seconds 12.34 -_minimizer.iterations_performed 87 -_minimizer.exit_reason converged -_minimizer.reduced_chi2 1.42 + +_fit_result.result_kind deterministic +_fit_result.fitting_time 12.34 +_fit_result.iterations 87 +_fit_result.exit_reason converged +_fit_result.reduced_chi_square 1.42 +_fit_result.objective_value 1532.4 ``` `_minimizer.optimizer_name` and `_minimizer.method_name` are gone — they diff --git a/docs/dev/plans/emcee-minimizer.md b/docs/dev/plans/emcee-minimizer.md deleted file mode 100644 index 7bd2bfa8a..000000000 --- a/docs/dev/plans/emcee-minimizer.md +++ /dev/null @@ -1,834 +0,0 @@ -# Plan: Emcee Minimizer - -> This plan follows [`AGENTS.md`](../../../AGENTS.md). No deliberate -> exceptions. - -## Prerequisite - -This plan depends on three accepted ADRs, all merged on `develop`: - -- [`minimizer-category-consolidation`](../adrs/accepted/minimizer-category-consolidation.md) - — unified `minimizer` category with `BayesianMinimizerBase` and the - `BumpsDreamMinimizer` Bayesian precedent. -- [`switchable-category-owned-selectors`](../adrs/accepted/switchable-category-owned-selectors.md) - — `analysis.minimizer.type = 'X'` is the writable surface; no - owner-level `._type` shims. -- [`minimizer-input-output-split`](../adrs/accepted/minimizer-input-output-split.md) - — fit-filled outputs live on a paired `analysis.fit_result` instance - (`LeastSquaresFitResult` or `BayesianFitResult`), not on the - minimizer. - -emcee inherits the entire paired surface for free: -`BayesianMinimizerBase._fit_result_class = BayesianFitResult`, so -`Analysis._swap_minimizer` instantiates both `EmceeMinimizer` (inputs) -and `BayesianFitResult` (outputs) atomically. - -## ADR - -This plan implements the emcee follow-on described in §1, §5, §6 and §9 -of -[`docs/dev/adrs/accepted/minimizer-category-consolidation.md`](../adrs/accepted/minimizer-category-consolidation.md). -No new ADR is required. - -If implementation uncovers a design question not covered by the ADRs -(e.g. resume semantics on parameter-set mismatch, or -`InitializationMethodEnum` ↔ `DreamPopulationInitializationEnum` -reconciliation), stop and ask before proceeding. - -## Branch and PR - -- Branch: `emcee-minimizer`. Do not push unless asked. -- Each step in §"Implementation steps (Phase 1)" must be staged with - explicit paths and committed locally **before** moving to the next - step. See `AGENTS.md` → **Commits**. -- After P1.8, stop and wait for the user review gate before starting - Phase 2. - -## Decisions already made (from the accepted ADRs) - -1. emcee is a new concrete Bayesian minimizer registered under - `MinimizerTypeEnum.EMCEE = 'emcee'`. Selected via - `project.analysis.minimizer.type = 'emcee'`. -2. emcee inherits two paired surfaces from `BayesianMinimizerBase`: - - **Settings** (writable inputs): `sampling_steps`, `burn_in_steps`, - `thinning_interval`, `population_size`, `parallel_workers`, - `initialization_method`, `random_seed` — all declared by - `BayesianMinimizerBase.__init__`. emcee may override class-level - defaults (e.g. `sampling_steps=5000`, `population_size=32`) and - adds one emcee-specific input `proposal_moves`. - - **Outputs** (fit-filled, internal `_set_*`): - `acceptance_rate_mean`, `gelman_rubin_max`, - `effective_sample_size_min`, `best_log_posterior`, - `point_estimate_name`, `sampler_completed`, - `credible_interval_inner/outer` — already on `BayesianFitResult`, - not on the minimizer. emcee's projection writer calls - `self.fit_result._set_*` (not `self.minimizer._set_*`). -3. Verbose attribute names map to emcee native kwargs via - `EmceeMinimizer._native_key_map` (overrides the DREAM-style defaults - on `BayesianMinimizerBase._native_key_map`): - - | Verbose | emcee native engine attribute | - | ----------------------- | ------------------------------------------------------------------------------ | - | `sampling_steps` | `nsteps` | - | `burn_in_steps` | `nburn` | - | `thinning_interval` | `thin` | - | `population_size` | `nwalkers` | - | `parallel_workers` | `parallel_workers` (engine integer; **not** mapped directly to emcee's `pool`) | - | `initialization_method` | (custom — see §6) | - | `random_seed` | `random_seed` | - - **`parallel_workers` semantics.** The category persists an integer; - the engine class holds it under the same name. emcee's - `EnsembleSampler` takes a `pool=` argument that is a pool object - (anything with a `.map` method) or `None` for serial execution — not - an integer. The engine builds the actual pool around `run_mcmc`: - - | `parallel_workers` value | Engine behaviour | - | ------------------------ | -------------------------------------- | - | `1` | `pool=None` (serial); single process | - | `0` | `multiprocessing.Pool(os.cpu_count())` | - | `N > 1` | `multiprocessing.Pool(N)` | - - The pool is closed in a `finally:` block after `run_mcmc` returns. - `Analysis._sync_engine_from_minimizer_category` must therefore skip - `parallel_workers` from the native-attribute sync (it already skips - `random_seed` for the same "engine handles it" reason; add - `parallel_workers` to the `_engine_sync_skip_keys` frozenset - introduced in P1.5). - -4. Resume uses emcee's `HDFBackend` against the `/emcee_chain` group of - the same `analysis/results.h5` file used by the snapshot writer. No - separate sidecar file. A non-resume `fit()` follows the prerequisite - plan's lifecycle and **truncates** `results.h5` (after the standard - warning); resume opens it in append mode. - - **`Project.save()` must not delete `/emcee_chain`.** The current - `write_analysis_results_sidecar` - ([`results_sidecar.py:282`](../../../src/easydiffraction/io/results_sidecar.py)) - opens `results.h5` with mode `'w'` (full truncate). Every save after - a fit will erase the emcee chain unless this writer is modified. - After this plan, the writer opens the file in append mode (`'a'`) and - replaces only the EasyDiffraction-canonical groups (`/posterior`, - `/distribution_cache`, `/pair_cache`, `/predictive`) by deleting them - first if present, leaving any other top-level groups (currently only - `/emcee_chain`) untouched. - - **Non-resume truncate must still happen — and explicitly.** The - current `_warn_results_sidecar_overwrite` - ([`analysis.py:943-953`](../../../src/easydiffraction/analysis/analysis.py)) - / - [`warn_analysis_results_sidecar_overwrite`](../../../src/easydiffraction/io/results_sidecar.py) - only **warns**; it does not delete. With the writer switched to - append mode, a fresh non-resume emcee fit after an older emcee run - would otherwise leave the stale `/emcee_chain` group in `results.h5`. - This plan therefore adds a real preparation step that warns **and - removes** the file before the engine starts. See P1.5a; the resume - path bypasses it. - -5. `Analysis.fit()` gains an optional `resume=True, extra_steps=N` call - shape for minimizers that support incremental sampling. For other - minimizers, passing `resume=True` raises immediately. -6. emcee outputs translate to the existing `BayesianFitResults` runtime - shape (plural — note the singular `BayesianFitResult` is the - persistence category, not the runtime object) exactly as DREAM does — - same `PosteriorSamples`, `PosteriorParameterSummary`, etc. Plotting - and display code needs no specialization. -7. Two `EmceeMinimizer` classes coexist with the same DREAM precedent: - - `src/easydiffraction/analysis/categories/minimizer/emcee.py` — the - persisted **category** class (`BayesianMinimizerBase` subclass) - used for CIF persistence and the user-facing setter surface. - - `src/easydiffraction/analysis/minimizers/emcee.py` — the live - **engine** class registered with the engine `MinimizerFactory`. - Holds the `emcee.EnsembleSampler` instance and runs the sampler. - This mirrors the existing `BumpsDreamMinimizer` split between - `categories/minimizer/bumps_dream.py` and - `minimizers/bumps_dream.py`. -8. Phase 2 resolved the `PLR0913` signature-width check by replacing the - internal explicit execution keyword set on `MinimizerBase.fit` and - `Fitter.fit` with `MinimizerFitOptions` and `FitterFitOptions` - parameter objects. `Analysis.fit(resume=True, extra_steps=N)` remains - the user-facing API. This is a deliberate refinement of P1.5 that - follows `AGENTS.md`'s refactor guidance instead of raising thresholds - or suppressing the lint rule. - -## Open questions - -- **Resume after parameter-set change.** If the user fits, then edits - which parameters are free, then calls `fit(resume=True, ...)`, emcee's - `HDFBackend.shape` mismatches the current parameter count. Plan - default: detect mismatch in P1.5 and raise `ValueError` with a clear - "start a fresh run" message. -- **Resume after a non-emcee fit.** If the user runs DREAM, switches to - emcee, and calls `fit(resume=True, ...)`, the `/emcee_chain` group - will be missing. Plan default: raise `ValueError` pointing at the - prerequisite ADR's lifecycle rule ("a new fit overwrites the file"). -- **`InitializationMethodEnum` ↔ `DreamPopulationInitializationEnum` - reconciliation** (deferred from review-8 F6 of the consolidation - work). emcee uses different init methods (`ball`, `uniform`, `prior`); - DREAM exposes a broader engine-level enum with `EPS`, `COV`, `LHS`, - `RANDOM`. The persisted user-facing enum (`InitializationMethodEnum`) - is narrower (`LATIN_HYPERCUBE`, `BALL`, `UNIFORM`, `PRIOR`). P1.3 must - decide whether to narrow DREAM's engine enum to match, or accept the - asymmetry. Recommend: keep DREAM's broader engine enum (legacy DREAM - users may pass `EPS`/`COV`/`RANDOM` directly to the engine) but - document that only `LATIN_HYPERCUBE` is persistable for DREAM; emcee - accepts `BALL`/`UNIFORM`/`PRIOR` only. -- **Move-mix semantics.** emcee supports proposal-move mixtures (e.g. 70 - % stretch + 30 % differential evolution). The consolidation ADR §5 - exposes `proposal_moves` as a single string descriptor. Plan default: - limit `proposal_moves` to single-move strings for v1 (`stretch`, `de`, - `de_snooker`, `walk`). Mixtures deferred to a follow-on plan. Record - this in the descriptor's `description=` text. - -## Cleanup opportunities inherited from earlier work - -The input/output-split work left four cleanup items open when this plan -was written. They touch code this plan modifies, so fold them in -opportunistically while the surrounding code is already being edited; -the plan does not block on them. - -- **#100 — Collapse duplicate predictive-cache-key helpers.** - `Analysis._predictive_cache_key` - ([analysis.py:528](../../../src/easydiffraction/analysis/analysis.py)) - and `Plotter._posterior_predictive_key` - ([plotting.py:3823](../../../src/easydiffraction/display/plotting.py)) - build the identical string; keep one canonical helper. P1.6 may touch - the predictive plotting path while validating emcee posterior - emission. Resolved in P1.6. -- **#101 — Remove dead branch in `Analysis._fit_state_categories`.** - Both branches return the same list since the Bayesian categories were - absorbed. One-line fix at - [analysis.py:1184-1205](../../../src/easydiffraction/analysis/analysis.py). - Resolved in P1.6. -- **#102 — Drop compute-and-ignore `result_kind` validation.** - `_restore_persisted_fit_state` - ([serialize.py:590-606](../../../src/easydiffraction/io/cif/serialize.py)) - calls `FitResultKindEnum(result_kind_value)` for its side effect only. - Move the warning into the descriptor setter, or extract a validator - helper. Still open. -- **#103 — Make `_sync_engine_from_minimizer_category` skip-keys - declarative.** This plan adds `proposal_moves` as a second - engine-level "ambient" key (alongside `random_seed`). The current - magic-string skip at - [analysis.py:1138](../../../src/easydiffraction/analysis/analysis.py) - should become a class-level - `_engine_sync_skip_keys: ClassVar[frozenset[str]] = frozenset(...)` on - `MinimizerCategoryBase` before the second member lands. Recommend - addressing as part of P1.5. Resolved in P1.5. - -When the matching open-issue is fully resolved, move it to -[`closed.md`](../issues/closed.md) and update -[`adrs/index.md`](../adrs/index.md) if relevant. - -## Concrete files likely to change - -### Created - -- `src/easydiffraction/analysis/categories/minimizer/emcee.py` — - persisted **category** class `EmceeMinimizer(BayesianMinimizerBase)` - with class-level defaults, `_engine_metadata`, and the overridden - `_native_key_map`. -- `src/easydiffraction/analysis/minimizers/emcee.py` — live **engine** - class `EmceeMinimizer(BayesianMinimizerEngineBase or equivalent)` - registered with `MinimizerFactory`. Holds `emcee.EnsembleSampler` and - the `HDFBackend`. -- `tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py`. -- `tests/unit/easydiffraction/analysis/minimizers/test_emcee.py`. -- `docs/docs/tutorials/ed-25.py` and `docs/docs/tutorials/ed-26.py` - (emcee fresh-run tutorial and resume/reload tutorial). The next free - tutorial slot — `ed-23` is already the "Co2SiO4 Sequential Fit" - tutorial and `ed-24` is the "LBCO Bayesian Display" tutorial. Verify - `ed-25.py` is unused before creating it at P1.7 start; bump if a newer - slot is already occupied. - -### Phase 2 verification files - -- `tests/integration/fitting/test_emcee.py` (cross-check vs DREAM on a - shared toy fit; assert posterior medians agree to within tolerance). - -### Modified - -- `src/easydiffraction/analysis/minimizers/enums.py` — add - `MinimizerTypeEnum.EMCEE = 'emcee'`. -- `src/easydiffraction/analysis/categories/minimizer/__init__.py` — add - explicit `EmceeMinimizer` (category) import so registration fires. -- `src/easydiffraction/analysis/minimizers/__init__.py` (or the - factory's package init) — add explicit `EmceeMinimizer` (engine) - import so engine registration fires. -- `src/easydiffraction/analysis/categories/minimizer/bayesian_base.py` — - only if review-8 F6 reconciliation (open question above) calls for - narrowing the persisted enum surface; otherwise unchanged. -- `src/easydiffraction/analysis/analysis.py` — `fit()` signature gains - `resume: bool = False, extra_steps: int | None = None`; validation + - dispatch to engine. Wire `_engine_sync_skip_keys` (#103) before adding - `proposal_moves` to the ambient set. -- `src/easydiffraction/io/results_sidecar.py` — read path: when - `/emcee_chain` is present, expose a helper to construct an - `emcee.backends.HDFBackend(path, name='emcee_chain', read_only=True)` - for inspection/visualisation. -- `pyproject.toml` — add `emcee` as a direct runtime dependency and - refresh the lockfile via `pixi lock` (CI installs from the lockfile, - not from the manifest files alone). - -### Deleted - -- None. - -## Implementation steps (Phase 1) - -Mark `[x]` as each step lands. - -- [x] **P1.1 — Add emcee dependency and refresh the lockfile.** - - Add `emcee` to `pyproject.toml` (runtime dependencies, not just the - `doc` extra — the existing lockfile carries emcee only as - `extra == 'doc'` which CI does not install for runtime). - - Add the same dependency to `pixi.toml` (runtime feature). - - Run `pixi lock` to regenerate `pixi.lock` with `emcee` as a direct - runtime dependency. The refreshed `pixi.lock` is the artifact CI - consumes; `pixi install` is a local sanity check only. - - Stage `pyproject.toml`, `pixi.toml`, and `pixi.lock` together. - - Files modified by this step: `pyproject.toml`, `pixi.toml`, - `pixi.lock`. Commit: `Add emcee runtime dependency` - -- [x] **P1.2 — Register `MinimizerTypeEnum.EMCEE`.** Add the enum member - with value `'emcee'` to - `src/easydiffraction/analysis/minimizers/enums.py`. No other code - wiring yet. Commit: `Register emcee minimizer enum value` - -- [x] **P1.3 — Add `EmceeMinimizer` category class.** New file - `src/easydiffraction/analysis/categories/minimizer/emcee.py`. - `EmceeMinimizer(BayesianMinimizerBase)` declares: - - `type_info` with `tag=MinimizerTypeEnum.EMCEE` and a description. - - `_engine_metadata: ClassVar[dict[str, str]] = {'optimizer_name': 'emcee', 'method_name': 'de'}` - (matching the `BumpsDreamMinimizer` precedent for the - `_restore_fit_results_from_projection` lookup). - - `_native_key_map` override mapping the verbose names to emcee's - native kwargs (see §"Decisions already made" point 3). - - Class-level defaults for emcee-specific values: - `sampling_steps=5000`, `burn_in_steps=1000`, `thinning_interval=1`, - `population_size=32`, `parallel_workers=0`, `proposal_moves='de'`. - - `__init__` constructs descriptors via the inherited helpers - (`_sampling_steps_descriptor(default)`, etc. from - `BayesianMinimizerBase`) and adds a new `proposal_moves` descriptor - with a `MembershipValidator` over the single-move set (`stretch`, - `de`, `de_snooker`, `walk`). - - **Decide the `InitializationMethodEnum` reconciliation** (open - question above) before wiring. - `EmceeMinimizer._supported_initialization_methods` should list - `(BALL, UNIFORM, PRIOR)` regardless of the DREAM decision. - - Update `src/easydiffraction/analysis/categories/minimizer/__init__.py` - to import `EmceeMinimizer` (registration trigger via - `@MinimizerCategoryFactory.register`). - - The paired `BayesianFitResult` flows automatically because - `BayesianMinimizerBase._fit_result_class = BayesianFitResult`; no - wiring needed in this step. - - Commit: `Add EmceeMinimizer category class` - -- [x] **P1.4 — Add `EmceeMinimizer` engine class.** New file - `src/easydiffraction/analysis/minimizers/emcee.py`. The engine - class is registered with `MinimizerFactory` and holds the - `emcee.EnsembleSampler` plus an `HDFBackend` attribute. Mirror the - shape of - [`bumps_dream.py BumpsDreamMinimizer`](../../../src/easydiffraction/analysis/minimizers/bumps_dream.py) - — descriptor attributes (`burn`, `thin`, `pop`, `init`, …) that - `Analysis._sync_engine_from_minimizer_category` writes to from the - category's `_native_kwargs()`. The descriptor names on the engine - must match the keys returned by `EmceeMinimizer._native_kwargs()`, - which are: `nsteps`, `nburn`, `thin`, `nwalkers`, - `parallel_workers`, `random_seed`, plus `initialization_method` - and `proposal_moves` handled by custom hooks (see §3 for the - mapping table and the `parallel_workers` semantics — the engine - attribute is an integer; the actual emcee `pool` object is built - and torn down inside `EmceeMinimizer.fit`, not by the native-key - sync). - - **Engine-facing contract.** `EmceeMinimizer.fit` matches the - existing - [`MinimizerBase.fit`](../../../src/easydiffraction/analysis/minimizers/base.py) - contract — `Fitter.fit` (the layer that owns `structures`, - `experiments`, `weights`, and `analysis`) calls every engine - uniformly. The engine receives only `parameters` and the - already-built `objective_function`: - - ```python - MinimizerBase.fit( - parameters: list[Parameter], - objective_function: Callable[[dict[str, object]], np.ndarray], - verbosity: VerbosityEnum = VerbosityEnum.FULL, - *, - finalize_tracking: bool = True, - use_physical_limits: bool = False, - random_seed: int | None = None, - resume: bool = False, # added by P1.5 - extra_steps: int | None = None, # added by P1.5 - ) -> FitResults - ``` - - The base implementation raises `NotImplementedError` only when - `resume=True` (see P1.5). `EmceeMinimizer.fit` overrides with - the same signature and honours `resume` / `extra_steps`. - - **Sidecar path.** emcee's `HDFBackend` needs a file path, but - the engine signature deliberately does not carry one. The - engine reads it from a private attribute - `self._sidecar_path: Path | None` that `Fitter.fit` sets on - the engine before calling `engine.fit(...)`, derived from - `analysis.project.info.path / 'analysis' / 'results.h5'`. - Engines that do not need it ignore the attribute; the - attribute defaults to `None` and `EmceeMinimizer.fit` raises - `RuntimeError` if it is `None` when needed. - - **Residual-to-log-probability adapter.** emcee expects a - scalar log probability from a flat walker coordinate - (`np.ndarray` shape `(ndim,)`), but the `objective_function` - `Fitter.fit` passes us returns a residual array from an - `engine_params` dict. The engine class builds an adapter - `log_prob(theta)` that: - - 1. Maps `theta` (the walker vector) onto an `engine_params` - dict using a fixed ordered list of free-parameter unique - names captured **once** at sampler construction (from the - `parameters` argument to `fit`). - 2. Rejects values outside - `[parameter.fit_min, parameter.fit_max]` by returning - `-np.inf` immediately — does not call the calculator for - invalid proposals. - 3. Calls the **passed-in** `objective_function(engine_params)` - (do **not** call `Fitter._build_objective_function` from - the engine — that is a `Fitter` helper, not engine API). - 4. Returns Gaussian log likelihood - `-0.5 * np.sum(r**2)`. The current fit weights are already - folded into the residuals by the objective function; this - step does **not** re-weight. - 5. Returns `-np.inf` on calculator exceptions (rare; emcee - re-proposes). - 6. Treats the prior as flat over the box-bounded parameter - volume — no informative priors in v1; deferred to a - follow-on plan. - - The adapter must **not** mutate live `Parameter.value` state - on `-np.inf` returns. Implement by passing an - `engine_params` dict to `objective_function` (the existing - objective writes the values into live parameters internally; - that mutation only happens once `objective_function` is - invoked, so the bounds check above must guard every call). - - Engine method shape (sketch): - - ```python - class EmceeMinimizer(MinimizerBase): - name = MinimizerTypeEnum.EMCEE - method = 'de' - - # Set by Fitter.fit before this fit() call: - _sidecar_path: Path | None = None - - def fit( - self, - parameters: list[Parameter], - objective_function: Callable[..., object], - verbosity: VerbosityEnum = VerbosityEnum.FULL, - *, - finalize_tracking: bool = True, - use_physical_limits: bool = False, - random_seed: int | None = None, - resume: bool = False, - extra_steps: int | None = None, - ) -> FitResults: - if self._sidecar_path is None: - msg = ('emcee engine requires Fitter.fit to set ' - '_sidecar_path; was Analysis configured?') - raise RuntimeError(msg) - - free_param_names = [p.unique_name for p in parameters] - param_by_name = {p.unique_name: p for p in parameters} - - def log_prob(theta): - for name, value in zip(free_param_names, theta): - p = param_by_name[name] - if not (p.fit_min <= value <= p.fit_max): - return -np.inf - engine_params = dict(zip(free_param_names, theta)) - try: - r = objective_function(engine_params) - except Exception: - return -np.inf - return -0.5 * float(np.sum(np.asarray(r) ** 2)) - - backend = emcee.backends.HDFBackend( - self._sidecar_path, name='emcee_chain', - read_only=False, - ) - pool = self._build_pool(self.parallel_workers) - try: - sampler = emcee.EnsembleSampler( - nwalkers=self.nwalkers, - ndim=len(free_param_names), - log_prob_fn=log_prob, - pool=pool, - moves=self._resolve_moves(self.proposal_moves), - backend=backend, - ) - if resume: - self._validate_resume(backend, free_param_names) - sampler.run_mcmc( - None, nsteps=extra_steps, - skip_initial_state_check=True, - progress=True, - ) - else: - initial_state = self._initial_state( - parameters, self.nwalkers, - self.init, random_seed, - ) - sampler.run_mcmc( - initial_state, nsteps=self.nsteps, - progress=True, - ) - finally: - if pool is not None: - pool.close() - pool.join() - return self._build_results(sampler, parameters) - ``` - - Register with the engine `MinimizerFactory` and update - `src/easydiffraction/analysis/minimizers/__init__.py` (or the - relevant package init) to import the engine class. - - Commit: `Add EmceeMinimizer engine class` - -- [x] **P1.5 — Wire `fit(resume=True, extra_steps=N)` end-to-end.** The - current fit stack does not accept `resume` / `extra_steps` - anywhere. Every signature and call site listed below must be - updated in this step. Each item is one short edit; the step lands - as a single commit because the signatures must change in lockstep. - - **`Fitter` stays the layer that owns `structures`, `experiments`, - `weights`, parameter collection, and objective construction.** Engines - receive only `parameters` and `objective_function` (already-built) via - the existing `MinimizerBase.fit` shape. This plan adds `resume` and - `extra_steps` to the same shape. - - **Signatures (add - `resume: bool = False, extra_steps: int | None = None`):** - - `Analysis.fit` - ([analysis.py:929](../../../src/easydiffraction/analysis/analysis.py)). - User-facing entry point. - - `Analysis._run_single`, `Analysis._run_joint`, - `Analysis._prepare_fit_run`, `Analysis._fit_single`, - `Analysis._fit_joint` (every internal helper that takes the fit - through to `Fitter.fit`). - - `Fitter.fit` - ([fitting.py:140-150](../../../src/easydiffraction/analysis/fitting.py)) - — adds the keyword pair to its existing signature (`structures`, - `experiments`, `weights`, `analysis`, `verbosity`, - `use_physical_limits`, `random_seed`, **+ `resume`, - `extra_steps`**). Forwards the pair to `self.minimizer.fit(...)`. - - `MinimizerBase.fit` - ([base.py:351-360](../../../src/easydiffraction/analysis/minimizers/base.py)) - — adds the keyword pair to its existing engine-facing shape - (`parameters`, `objective_function`, `verbosity`, _keyword_: - `finalize_tracking`, `use_physical_limits`, `random_seed`, **+ - `resume`, `extra_steps`**). The base implementation handles - non-resume calls unchanged and raises - `NotImplementedError(f"Minimizer '{self.name}' does not support resume.")` - when `resume=True`. `EmceeMinimizer.fit` overrides with the same - signature and honours both args. - - **Sidecar-path plumbing.** `Fitter.fit` resolves the sidecar path from - `analysis.project.info.path` (when both are non-None) and sets - `self.minimizer._sidecar_path` on the engine before calling - `self.minimizer.fit(...)`. Engines that do not need it ignore the - attribute; `EmceeMinimizer.fit` reads it (and raises `RuntimeError` if - `None` as defence in depth — but normal users never hit that path - because of the upfront save-required guard in the next bullet). - - **Behaviour rules:** - - **Single mode only for v1.** `Analysis._run_joint` raises - `ValueError('Resume is supported in single fit mode only')` when - `resume=True`. Joint-mode resume is deferred; recorded explicitly in - §"Open questions". - - **Validate the active minimizer.** `Analysis.fit` raises - `ValueError` when `resume=True` and - `self.minimizer.type != MinimizerTypeEnum.EMCEE.value`. Match the - clear-error pattern used elsewhere. - - **Require a saved project for emcee.** Unlike DREAM (which keeps the - chain in memory), emcee's `HDFBackend` is the sampler's live chain - store — it needs a real file path. `Analysis.fit` raises - `ValueError` when - `self.minimizer.type == MinimizerTypeEnum.EMCEE.value` and - `self.project.info.path is None`, with a clear scientist-facing - message that points at the fix: - `"emcee requires a saved project; call project.save_as() before analysis.fit()."` - The check fires for both the initial run and `resume=True`. The - engine's `_sidecar_path is None` `RuntimeError` (per P1.4) remains - as defence-in-depth for direct engine calls outside the `Analysis` - flow but is not reachable from the user-facing path. - - **Validate `extra_steps`.** Require positive integer when - `resume=True`. Raise on `None`, `0`, or negative. - - **Bypass reset.** Skip the new - `prepare_analysis_results_sidecar_for_new_fit` helper (see P1.5a) - and `_clear_persisted_fit_state` when `resume=True` — both would - clobber the chain and the persisted fit state. - - **Defence in depth.** `MinimizerBase.fit`'s `NotImplementedError` on - `resume=True` only matters if an engine is called directly outside - the `Analysis` flow — the `Analysis.fit` guard above fires first in - normal use. - - **Issue #103 cleanup.** Introduce - `_engine_sync_skip_keys: ClassVar[frozenset[str]] = frozenset({'random_seed', 'parallel_workers'})` - on `MinimizerCategoryBase`, and update - `_sync_engine_from_minimizer_category` - ([analysis.py:1134-1146](../../../src/easydiffraction/analysis/analysis.py)) - to use it. `EmceeMinimizer` (category) overrides the frozenset to add - `'proposal_moves'` if the engine consumes that key differently from - the category attribute (sketch in P1.4). - - Commit: `Wire emcee resume through fit stack` - -- [x] **P1.5a — Make `results.h5` append-on-save and add an explicit - truncate-on-new-fit prep step.** Two coordinated changes that land - in a single commit because they jointly preserve the ADR lifecycle - (resume keeps `/emcee_chain`; new fit removes it). - - In - [`src/easydiffraction/io/results_sidecar.py`](../../../src/easydiffraction/io/results_sidecar.py): - - Replace `h5py.File(sidecar_path, 'w')` (line 282) with - `h5py.File(sidecar_path, 'a')`. Before writing each - EasyDiffraction-canonical group (`/posterior`, - `/distribution_cache`, `/pair_cache`, `/predictive`), delete that - group first if present so the writer's behaviour for those groups is - unchanged. - - Do **not** touch any other top-level group from the writer. - `/emcee_chain` survives every save. - - **Add a new helper** - `prepare_analysis_results_sidecar_for_new_fit(*, analysis_dir: Path) -> None` - that takes the same `analysis_dir` shape as - `warn_analysis_results_sidecar_overwrite`, **warns** when the file - exists (matching the current warning text), and then **removes** the - file entirely so a fresh fit starts from a clean slate. The old - `warn_analysis_results_sidecar_overwrite` becomes a thin wrapper - that delegates to the new helper, or is replaced outright by the new - helper at every call site. - - In - [`src/easydiffraction/analysis/analysis.py`](../../../src/easydiffraction/analysis/analysis.py): - - Replace `_warn_results_sidecar_overwrite` (lines 943-953) with a - call to the new `prepare_analysis_results_sidecar_for_new_fit` - helper. Same call sites in `_run_single` and `_run_joint`. - - **Bypass on resume.** The `resume=True` branch in - `Analysis._prepare_fit_run` (added by P1.5) must **not** call this - helper — that is the whole point of resume keeping the chain alive. - The bypass rule listed in P1.5 "Behaviour rules → Bypass reset" - therefore now also covers the - `prepare_analysis_results_sidecar_for_new_fit` call. - - Add focused unit tests in Phase 2 (P2.1) covering: - - **Append preserves `/emcee_chain`.** Write a sidecar payload, create - a stub `/emcee_chain` group on the same file, re-write the sidecar — - the `/emcee_chain` group must survive. - - **New-fit prep removes the file.** Create a sidecar with a stub - `/emcee_chain` group; call - `prepare_analysis_results_sidecar_for_new_fit`; assert the file is - gone (or empty) and the stale group is unreachable. - - **Resume bypass.** Set up an `Analysis` with a saved project, invoke - the resume code path (mocked engine), and assert - `prepare_analysis_results_sidecar_for_new_fit` was **not** called. - - Commit: `Append-on-save plus explicit truncate-on-new-fit prep` - -- [x] **P1.6 — Route emcee outputs into the existing fit_result and - sidecar pipeline.** Verify the existing - `_store_posterior_fit_projection` - ([analysis.py](../../../src/easydiffraction/analysis/analysis.py)) - writes to `self.fit_result._set_*` (the `BayesianFitResult` - instance auto-paired with the `EmceeMinimizer`) and that the - `/posterior`, `/distribution_cache`, `/pair_cache`, `/predictive` - groups in `results.h5` receive emcee output without modification. - Adjust only where emcee surfaces data differently from DREAM (e.g. - `EnsembleSampler.get_chain(flat=False, discard=burn, thin=thin)` - vs the DREAM extraction helper). Cache derivations (KDE, pair - grids) reuse the existing pipeline. - - Opportunistic cleanup: address issue #100 if the - predictive plotting path is being touched anyway. Collapse - `Analysis._predictive_cache_key` and - `Plotter._posterior_predictive_key` into one canonical helper. - - Commit: `Route emcee posterior through fit_result and sidecar` - -- [x] **P1.7 — Add emcee tutorials.** Verify first that - `docs/docs/tutorials/ed-25.py` is unused. `ed-23.py` is the - "Co2SiO4 Sequential Fit" tutorial and `ed-24.py` is the "LBCO - Bayesian Display" tutorial — do **not** overwrite either. If - `ed-25.py` already exists by the time this step runs, pick the - next free integer slot and adjust the file name + references below - to match. - - New notebook sources at `docs/docs/tutorials/ed-25.py` and - `docs/docs/tutorials/ed-26.py` covering: - - - `ed-25.py`: `project.analysis.minimizer.type = 'emcee'` - (post-switchable syntax), tutorial-sized sampler settings, - `project.analysis.fit()`, posterior plots, and `project.save()`. - - `ed-26.py`: reopening the saved project, displaying restored - Bayesian results, and - `project.analysis.fit(resume=True, extra_steps=500)` to continue the - chain. - - Update the docs navigation in the same step: - - Add an entry under "MCMC / Bayesian" (or the appropriate section) in - [`docs/docs/tutorials/index.md`](../../docs/tutorials/index.md) - pointing at `ed-25.ipynb` and `ed-26.ipynb`. - - Add a navigation entry under the matching section in - [`docs/mkdocs.yml`](../../../docs/mkdocs.yml). - - Run `pixi run notebook-prepare` to regenerate the `.ipynb`. - - Verification greps: - - ``` - test -f docs/docs/tutorials/ed-25.py - test -f docs/docs/tutorials/ed-26.py - git grep -nE '\banalysis\.minimizer_type\b|\bminimizer\.runtime_seconds\b|\bminimizer\.gelman_rubin_max\b' docs/docs/tutorials/ed-25.py docs/docs/tutorials/ed-26.py - git grep -n 'ed-2[56]' docs/docs/tutorials/index.md docs/mkdocs.yml - ``` - - The first two must be true; the third must be empty; the fourth must - return at least one hit for both tutorial files. - - Commit: `Introduce emcee minimizer tutorials` - -- [x] **P1.8 — Phase 1 review gate.** No code change. Stop and request - user review. After approval, proceed to Phase 2. - -## Verification (Phase 2) - -Each command captures its log with a zsh-safe exit-code variable as -required by `AGENTS.md` → **Workflow**. - -- [x] **P2.1 — Add unit + integration tests.** - - `tests/unit/easydiffraction/analysis/categories/minimizer/test_emcee.py`: - category-class descriptor defaults; `_native_key_map` override; - pairing with `BayesianFitResult`; swap behavior; resume - parameter-set-mismatch error path (no real sampler). - - `tests/unit/easydiffraction/analysis/minimizers/test_emcee.py`: - engine-class registration, descriptor defaults, native kwargs - plumbing. - - `tests/unit/easydiffraction/analysis/test_analysis.py` (or matching - coverage file): assert `Analysis.fit()` raises `ValueError` with a - save-prompt message when emcee is the active minimizer and - `project.info.path is None`, for both initial fits and - `resume=True`. - - `tests/unit/easydiffraction/io/test_results_sidecar.py`: the - save-after-resume invariant from P1.5a — write a sidecar payload, - then write a stub `/emcee_chain` group on the same file, then - re-write the sidecar; the `/emcee_chain` group must survive. - - `tests/integration/fitting/test_emcee.py`: end-to-end fit on a small - synthetic problem; resume; assert posterior medians agree with a - DREAM run within tolerance. - - Layout check: - - ``` - pixi run test-structure-check > /tmp/easydiffraction-test-structure-check.log 2>&1; \ - test_structure_check_exit_code=$?; \ - tail -n 200 /tmp/easydiffraction-test-structure-check.log; \ - exit $test_structure_check_exit_code - ``` - -- [x] **P2.2 — Auto-fixes and static checks.** - - ``` - pixi run fix > /tmp/easydiffraction-fix.log 2>&1; \ - fix_exit_code=$?; \ - tail -n 200 /tmp/easydiffraction-fix.log; \ - exit $fix_exit_code - ``` - - Then: - - ``` - pixi run check > /tmp/easydiffraction-check.log 2>&1; \ - check_exit_code=$?; \ - tail -n 200 /tmp/easydiffraction-check.log; \ - exit $check_exit_code - ``` - -- [x] **P2.3 — Unit tests.** - - ``` - pixi run unit-tests > /tmp/easydiffraction-unit-tests.log 2>&1; \ - unit_tests_exit_code=$?; \ - tail -n 200 /tmp/easydiffraction-unit-tests.log; \ - exit $unit_tests_exit_code - ``` - -- [x] **P2.4 — Integration tests.** - - ``` - pixi run integration-tests > /tmp/easydiffraction-integration-tests.log 2>&1; \ - integration_tests_exit_code=$?; \ - tail -n 200 /tmp/easydiffraction-integration-tests.log; \ - exit $integration_tests_exit_code - ``` - -- [x] **P2.5 — Script tests.** - - ``` - pixi run script-tests > /tmp/easydiffraction-script-tests.log 2>&1; \ - script_tests_exit_code=$?; \ - tail -n 200 /tmp/easydiffraction-script-tests.log; \ - exit $script_tests_exit_code - ``` - -## Phase 2 review notes - -- Phase 2 bug fixes found by verification should be committed separately - from checklist updates. The result-synchronization fix is retained, - but a future equivalent commit should describe the behavior change - directly, for example: reorder result-sync branches so - `OptimizeResult` takes the `.x` path. -- Script-test fixes should also be split from checklist-only commits - when practical. The `ed-17` save-path change is retained because it - avoids a project-path collision with `ed-5`; notebook metadata - reordering came from the required `notebook-prepare` regeneration. - -## Suggested Pull Request - -**Title:** Add emcee Bayesian sampler with resumable runs - -**Description (user-facing):** - -EasyDiffraction adds emcee — a widely-used affine-invariant MCMC sampler -— as a second Bayesian fitter. It is selected exactly like the existing -samplers, via the uniform switchable-category surface: - -```python -project.analysis.minimizer.type = 'emcee' -project.analysis.minimizer.sampling_steps = 5000 -project.analysis.fit() -``` - -Long runs can be **resumed** without starting over: - -```python -project.analysis.fit(resume=True, extra_steps=2000) -``` - -emcee's chain state lives inside the same `analysis/results.h5` file as -the other posterior data, so saving and reopening a project is a -single-file affair. Plots, parameter posteriors, and tables work the -same as for DREAM, so switching between samplers to cross-check results -is straightforward. - -New tutorials walk through a short run (`ed-25`) and reopening the saved -project to resume for additional steps (`ed-26`). - -Phase 2 also includes a small benchmark-runner elapsed-time display -improvement that landed alongside the emcee verification work. diff --git a/docs/dev/plans/minimizer-input-output-split.md b/docs/dev/plans/minimizer-input-output-split.md deleted file mode 100644 index e93345ad7..000000000 --- a/docs/dev/plans/minimizer-input-output-split.md +++ /dev/null @@ -1,673 +0,0 @@ -# Plan: Minimizer Input/Output Split - -> This plan follows [`AGENTS.md`](../../../AGENTS.md). No deliberate -> exceptions. - -## ADR - -Implements -[`docs/dev/adrs/suggestions/minimizer-input-output-split.md`](../adrs/suggestions/minimizer-input-output-split.md). -This plan promotes that ADR from Suggestion → Accepted during -implementation (step P1.16). - -Affected ADRs that this plan amends (per the ADR's §"ADRs amended"): - -- [`accepted/minimizer-category-consolidation.md`](../adrs/accepted/minimizer-category-consolidation.md) - — §1 becomes a partial rule; §"Alternatives Considered → D" records - the reversal. -- [`accepted/analysis-cif-fit-state.md`](../adrs/accepted/analysis-cif-fit-state.md) - — §"Minimizer fit projection" rewritten for the settings-only - `_minimizer.*` / outputs-on-`_fit_result.*` shape. -- [`accepted/runtime-fit-results.md`](../adrs/accepted/runtime-fit-results.md) - — closing paragraph references this ADR alongside the existing two. -- [`accepted/switchable-category-owned-selectors.md`](../adrs/accepted/switchable-category-owned-selectors.md) - — §1 gains the documented "fully-determined paired category" - exception. -- [`accepted/display-ux.md`](../adrs/accepted/display-ux.md) — - `project.display.fit.results()` prints a "Settings used" block. - -## Branch and PR - -- Branch: `minimizer-input-output-split` (continued from the branch the - ADR was drafted on). Do not push unless asked. -- Each step in §"Implementation steps (Phase 1)" must be staged with - explicit paths and committed locally **before** moving to the next - step. See `AGENTS.md` → **Commits**. -- After P1.17, stop and wait for the user review gate before starting - Phase 2. - -## Decisions already made (from the ADR) - -1. `analysis.minimizer` holds **writable user settings only**. -2. `analysis.fit_result` becomes a class hierarchy paired with - `analysis.minimizer`. `FitResultBase` carries common fields; - `LeastSquaresFitResult` and `BayesianFitResult` add family-specific - ones. -3. `fit_result` is **not a user-facing switchable category**. No - `fit_result.type`, no `fit_result.show_supported()`. The owner's - `_swap_minimizer` hook installs both the minimizer and the paired - `fit_result` atomically. -4. Pairing rule is encoded on the minimizer base classes: - `LeastSquaresMinimizerBase._fit_result_class = LeastSquaresFitResult`, - `BayesianMinimizerBase._fit_result_class = BayesianFitResult`. -5. `objective_value` (raw χ²) and `reduced_chi_square` are distinct - fields, both kept on `LeastSquaresFitResult`. -6. `credible_interval_inner` / `credible_interval_outer` stay on the - output side (`BayesianFitResult`) at the fixed `0.68` / `0.95` - values. User-configurable levels deferred to a follow-on ADR. -7. The display extension lives under `project.display.fit.results()`; no - new `Analysis`-level display method is added. -8. Beta posture: hard cutover, no shims, no deprecation warnings. - Tutorials and saved fixtures regenerate. - -## Open questions - -- **Existing `analysis.fit_result.from_cif` parameter ordering.** After - this split, the CIF restore for `_fit_result.*` must run **after** - `_minimizer.*` is read so the paired class is known before the result - descriptors load. The current `_restore_*` order in - [`serialize.py`](../../../src/easydiffraction/io/cif/serialize.py) - reads `_minimizer.*` first via `_swap_minimizer`, then iterates the - rest. P1.6 must ensure `_fit_result` is included in the iteration only - after the swap has installed the paired class. Confirm during P1.6 - implementation. -- **Posterior-summary code path that currently writes - `_set_credible_interval_*` on `minimizer`.** After P1.11, the setters - move to `BayesianFitResult`. The `_store_posterior_fit_projection` - method in - [`analysis.py`](../../../src/easydiffraction/analysis/analysis.py) - must be updated to call - `self._fit_result._set_credible_interval_inner(...)` instead of - `self.minimizer._set_*`. Verify the order of operations against the - test in - [`test_results_sidecar.py`](../../../tests/unit/easydiffraction/io/test_results_sidecar.py). - -## Concrete files likely to change - -### Created - -- `src/easydiffraction/analysis/categories/fit_result/base.py` — rename - existing `FitResult` class to `FitResultBase` (or extract a base). - Common output descriptors live here. -- `src/easydiffraction/analysis/categories/fit_result/lsq.py` — - `LeastSquaresFitResult` with LSQ-specific output descriptors. -- `src/easydiffraction/analysis/categories/fit_result/bayesian.py` — - `BayesianFitResult` with Bayesian-specific output descriptors, - including `credible_interval_inner` / `credible_interval_outer`. -- _(`src/easydiffraction/analysis/categories/fit_result/factory.py` - already exists; this plan extends it rather than creating it. See P1.4 - — the factory becomes a registration helper for the two new family - classes; the authoritative swap mechanism is the `_fit_result_class` - attribute on the paired minimizer base, not a factory lookup. The - factory is still useful for introspection / testing.)_ -- `tests/unit/easydiffraction/analysis/categories/fit_result/test_base.py` -- `tests/unit/easydiffraction/analysis/categories/fit_result/test_lsq.py` -- `tests/unit/easydiffraction/analysis/categories/fit_result/test_bayesian.py` -- `tests/unit/easydiffraction/analysis/categories/fit_result/test_factory.py` - -### Modified - -- `src/easydiffraction/analysis/categories/fit_result/__init__.py` — add - explicit imports for every new concrete class to trigger factory - registration. -- `src/easydiffraction/analysis/categories/fit_result/default.py` — - rewritten to import from the new family modules; the existing - `FitResult` is renamed to `FitResultBase` and absorbed. -- `src/easydiffraction/analysis/categories/minimizer/base.py` — add - `_fit_result_class: ClassVar[type]` declaration. -- `src/easydiffraction/analysis/categories/minimizer/lsq_base.py` — set - `_fit_result_class = LeastSquaresFitResult`; **remove** - `objective_name`, `objective_value`, `n_data_points`, `n_parameters`, - `n_free_parameters`, `degrees_of_freedom`, `covariance_available`, - `correlation_available`, `runtime_seconds`, `iterations_performed`, - `exit_reason` from descriptor declarations and from - `_result_descriptor_names`. `_setting_descriptor_names` stays - `('max_iterations',)`. -- `src/easydiffraction/analysis/categories/minimizer/bayesian_base.py` — - set `_fit_result_class = BayesianFitResult`; remove `runtime_seconds`, - `point_estimate_name`, `sampler_completed`, `credible_interval_inner`, - `credible_interval_outer`, `acceptance_rate_mean`, `gelman_rubin_max`, - `effective_sample_size_min`, `best_log_posterior` from descriptor - declarations and from `_result_descriptor_names`. - `_setting_descriptor_names` keeps the seven Bayesian inputs. -- `src/easydiffraction/analysis/analysis.py` — extend `_swap_minimizer` - to also instantiate the paired `fit_result` via the minimizer's - `_fit_result_class`; add `analysis.fit_result` property; route every - `self._minimizer._set_*` result-writer call in - `_store_least_squares_result_projection` / - `_store_posterior_fit_projection` / - `_restore_fit_results_from_projection` to `self._fit_result._set_*` - instead. -- `src/easydiffraction/io/cif/serialize.py` — emit/read `_fit_result.*` - from the paired class; remove the removed `_minimizer.*` output tags - from the serialise/deserialise paths. Update the legacy-tag rejection - message. -- `src/easydiffraction/project/display.py` — extend - `project.display.fit.results()` to print a "Settings used" block - populated from `analysis.minimizer.*` above the existing result - tables. -- All tutorials referencing `analysis.minimizer.` (e.g. - `runtime_seconds`, `gelman_rubin_max`, `objective_value`) → - `analysis.fit_result.`. List enumerated at P1.15 start - via `git grep`. -- Tests reading `analysis.minimizer.` → migrate to - `analysis.fit_result.`. P2.1 enumerates. - -### Deleted - -- None. The existing `fit_result/default.py` is rewritten in place; the - existing `FitResult` class becomes `FitResultBase`. - -## Implementation steps (Phase 1) - -Mark `[x]` as each step lands. - -- [x] **P1.1 — Rename `FitResult` to `FitResultBase`; add reset hooks; - update every import site.** In - `src/easydiffraction/analysis/categories/fit_result/default.py`, - rename the class. The factory `@register` decorator stays on the - renamed class so the default-tag lookup keeps working until P1.4 - extends the factory. - - Add two class-level hooks to `FitResultBase` matching the - `MinimizerCategoryBase` shape introduced by the consolidation - work - ([`minimizer/base.py:69-74`](../../../src/easydiffraction/analysis/categories/minimizer/base.py)): - - ```python - _result_descriptor_names: ClassVar[tuple[str, ...]] = ( - 'success', 'message', 'iterations', - 'fitting_time', 'reduced_chi_square', 'result_kind', - ) - - def _reset_result_descriptors(self) -> None: - """Reset fit-result descriptors to declared defaults.""" - for name in self._result_descriptor_names: - descriptor = getattr(self, name) - if isinstance(descriptor, GenericDescriptorBase): - descriptor.value = descriptor._value_spec.default_value() - ``` - - `LeastSquaresFitResult` (P1.2) and `BayesianFitResult` (P1.3) - then add their own field names to `_result_descriptor_names` - so the inherited helper resets every relevant descriptor. - - This must land in P1.1 because P1.6 retargets - `_clear_minimizer_result_projection` (renamed - `_clear_fit_result_projection`) to call - `self.fit_result._reset_result_descriptors()`, and that method - must exist on `FitResultBase` before the swap is wired. - - Update every package-level import that referenced the old - name. `git grep -nP '\bFitResult\b' src/ tests/` lists the - sites at plan time: - - - `src/easydiffraction/analysis/__init__.py` (line 14 today) - - `src/easydiffraction/analysis/categories/__init__.py` - (line 14 today) - - `src/easydiffraction/analysis/categories/fit_result/__init__.py` - - `src/easydiffraction/analysis/analysis.py` (import line 18; - type annotation on the `fit_result` property at line 432; - `self._fit_result = FitResult()` at line 483; same - construction at line 1208) - - All four `FitResult` import/annotation/construction sites in - `analysis.py` become `FitResultBase` after this step. The two - `self._fit_result = FitResult()` construction sites (init and - `_clear_persisted_fit_state`) become - `FitResultBase()` temporarily; P1.6 retargets them to the - paired class. - - Re-run `git grep -nP '\bFitResult\b' src/` at the end of this - step — every remaining hit must be the renamed class name or a - module path, not the old bare class. Tests are migrated by - P2.1. - - Commit: `Rename FitResult to FitResultBase, add reset hooks` - -- [x] **P1.2 — Add `LeastSquaresFitResult` class.** New file - `src/easydiffraction/analysis/categories/fit_result/lsq.py`. - `LeastSquaresFitResult(FitResultBase)` declares: `objective_name`, - `objective_value`, `n_data_points`, `n_parameters`, - `n_free_parameters`, `degrees_of_freedom`, `covariance_available`, - `correlation_available`, `exit_reason`. **All defaults are `None` - with `allow_none=True`**, matching the consolidation cleanup that - previously moved LSQ outputs off `0` / `false` / `''` so a pre-fit - CIF emits `?` rather than a value that looks like a degenerate - result. This applies to numeric, integer-like, string, and bool - fields alike; the descriptor helpers in - `LeastSquaresMinimizerBase` - ([`lsq_base.py`](../../../src/easydiffraction/analysis/categories/minimizer/lsq_base.py)) - that currently produce these descriptors are the model — they can - be lifted into `LeastSquaresFitResult` verbatim before being - removed from `lsq_base.py` at P1.9. Declare - `_expected_descriptor_names`, `_result_descriptor_names` for - parity with the minimizer hierarchy. Tests deferred to Phase 2. - Commit: `Add LeastSquaresFitResult class` - -- [x] **P1.3 — Add `BayesianFitResult` class.** New file - `src/easydiffraction/analysis/categories/fit_result/bayesian.py`. - `BayesianFitResult(FitResultBase)` declares: - `point_estimate_name`, `sampler_completed`, - `credible_interval_inner` (default `0.68`), - `credible_interval_outer` (default `0.95`), - `acceptance_rate_mean`, `gelman_rubin_max`, - `effective_sample_size_min`, `best_log_posterior`. Declare - `_expected_descriptor_names`, `_result_descriptor_names`. Tests - deferred to Phase 2. Commit: `Add BayesianFitResult class` - -- [x] **P1.4 — Register fit-result classes with the existing - `FitResultFactory`.** The factory already exists at - [`src/easydiffraction/analysis/categories/fit_result/factory.py`](../../../src/easydiffraction/analysis/categories/fit_result/factory.py) - and currently registers only the default common class. Update it - to also register `LeastSquaresFitResult` and `BayesianFitResult` - with their family tags. Update - `src/easydiffraction/analysis/categories/fit_result/__init__.py` - to explicitly import every concrete class (so registration fires - on package import, per the repo's standard pattern). - - **Authoritative mechanism:** `Analysis._swap_minimizer` - constructs the paired fit-result via the minimizer's - `_fit_result_class` attribute (P1.5), not via a factory - lookup. The factory is kept as a registration helper for - introspection and testing; do not add a public selector surface - (`type`, `show_supported`) since `fit_result` is internally - paired, per ADR §1. Commit: - `Register fit-result family classes with factory` - -- [x] **P1.5 — Declare `_fit_result_class` on minimizer bases.** In - `src/easydiffraction/analysis/categories/minimizer/lsq_base.py`, - add `_fit_result_class: ClassVar[type] = LeastSquaresFitResult`. - In - `src/easydiffraction/analysis/categories/minimizer/bayesian_base.py`, - add `_fit_result_class: ClassVar[type] = BayesianFitResult`. Add - the matching declaration to - `src/easydiffraction/analysis/categories/minimizer/base.py` with - `_fit_result_class: ClassVar[type] = FitResultBase` as a safety - fallback (no concrete minimizer instantiates the bare base, but - `_swap_minimizer` reads through this attribute). Commit: - `Declare paired _fit_result_class on minimizer bases` - -- [x] **P1.6 — Wire `Analysis._swap_minimizer` to install both - instances, and update every `_fit_result` reset path.** In - `src/easydiffraction/analysis/analysis.py`: - - `__init__` constructs the initial `_fit_result` from the default - minimizer's `_fit_result_class`: - `self._fit_result = self._minimizer._fit_result_class()`. The line - 483 `self._fit_result = FitResultBase()` (after P1.1) is replaced. - - `_replace_minimizer` constructs - `self._fit_result = new_minimizer._fit_result_class()` after the new - minimizer is created. The old `fit_result` is detached - (`_parent = None`) before being replaced. - - `_clear_persisted_fit_state` (line 1204 today) currently calls - `self._clear_minimizer_result_projection()` and then - `self._fit_result = FitResult()`. After P1.1 + the split, both lines - must change: - - `self._fit_result = self.minimizer._fit_result_class()` replaces - the bare `FitResultBase()` construction. This keeps the paired - class invariant whenever the persisted state is reset. - - `self._clear_minimizer_result_projection()` currently calls - `self.minimizer._reset_result_descriptors()`. After P1.9/P1.10 - remove the result descriptors from the minimizer, this method - becomes a no-op. **Retarget it to - `self.fit_result._reset_result_descriptors()`** and rename it to - `_clear_fit_result_projection`. Update the call sites (line 1204; - potentially others — `git grep` confirms). - - Add `analysis.fit_result` read-only property - (`return self._fit_result`). Type annotation: `FitResultBase` (the - family classes inherit from it). - - Wire `self._fit_result._parent = self` in - `_attach_category_parents`. Every `_fit_result` reassignment in the - methods above must also set `_parent` on the new instance. - - Verification at the end of this step: - - ``` - git grep -nE 'self\._fit_result\s*=' src/easydiffraction/analysis/analysis.py - ``` - - Every match must construct via `self.minimizer._fit_result_class()` - (or `new_minimizer._fit_result_class()` in `_replace_minimizer`), not - a bare class name. There must be no remaining - `self._fit_result = FitResultBase()` after this step. - - Commit: `Wire fit_result swap and reset paths to paired class` - -- [x] **P1.7 — Route LSQ result writers to `fit_result`.** In - `src/easydiffraction/analysis/analysis.py`, - `_store_least_squares_result_projection` currently writes to - `self.minimizer._set_objective_name(...)` etc. Reroute every such - call to `self.fit_result._set_*`. Same for - `_restore_fit_results_from_projection`'s LSQ branch (it reads - `self.minimizer.objective_name.value` etc. — change to - `self.fit_result..value`). Commit: - `Route LSQ result writers to fit_result` - -- [x] **P1.8 — Route Bayesian result writers to `fit_result`.** Same - treatment for `_store_posterior_fit_projection` and the Bayesian - branch of `_restore_fit_results_from_projection`. Includes the - `_set_credible_interval_*` calls — they now target - `self.fit_result._set_credible_interval_*`. Commit: - `Route Bayesian result writers to fit_result` - -- [x] **P1.9 — Remove output fields from LSQ minimizer base.** In - `src/easydiffraction/analysis/categories/minimizer/lsq_base.py`, - delete the descriptor declarations and properties for - `objective_name`, `objective_value`, `n_data_points`, - `n_parameters`, `n_free_parameters`, `degrees_of_freedom`, - `covariance_available`, `correlation_available`, - `runtime_seconds`, `iterations_performed`, `exit_reason`. Remove - these names from `_expected_descriptor_names` and - `_result_descriptor_names`. `_setting_descriptor_names` stays - `('max_iterations',)`. After this step, `_result_descriptor_names` - on `LeastSquaresMinimizerBase` is `()` and - `_reset_result_descriptors()` is a no-op on every LSQ minimizer — - confirming the P1.6 retarget of - `_clear_minimizer_result_projection` to operate on - `self.fit_result` is the correct call site. - - Note: `optimizer_name` and `method_name` were already removed - by the consolidation work (`_engine_metadata` dict replaces - them); this step is the bulk removal of the remaining LSQ - outputs. - - Commit: `Remove LSQ output descriptors from minimizer base` - -- [x] **P1.10 — Remove duplicate fields from Bayesian minimizer base.** - In - `src/easydiffraction/analysis/categories/minimizer/bayesian_base.py`, - delete the descriptor declarations and properties for - `runtime_seconds`, `point_estimate_name`, `sampler_completed`, - `credible_interval_inner`, `credible_interval_outer`, - `acceptance_rate_mean`, `gelman_rubin_max`, - `effective_sample_size_min`, `best_log_posterior`. Remove these - names from `_expected_descriptor_names` and - `_result_descriptor_names`. The `_setting_descriptor_names` tuple - keeps the seven Bayesian inputs. - - Commit: `Remove Bayesian output descriptors from minimizer base` - -- [x] **P1.11 — Update CIF emit/read for the split.** In - `src/easydiffraction/io/cif/serialize.py`: - - **No category-list reordering is performed in this step.** Neither - `Analysis._serializable_categories()` nor - `Analysis._fit_state_categories()` is restructured. `fit_result` stays - conditionally included by `_fit_state_categories()` only when - `self._has_persisted_fit_state()` is true — exactly as today. Pre-fit - projects continue to emit no `_fit_result.*` block. - - The only changes in this step are content updates inside the existing - emit/read flow: - - `_minimizer.*` emit/read continues to handle settings only (the - minimizer category's `from_cif` walks its remaining descriptors - after P1.9 / P1.10 removed the output descriptors). - - `_fit_result.*` emit/read picks up the new family-specific - descriptors automatically because P1.6 wires the paired class - (`LeastSquaresFitResult` or `BayesianFitResult`) onto - `self._fit_result`. The existing - `analysis.fit_result.from_cif(block)` call inside - `_restore_common_fit_state` - ([`serialize.py:590`](../../../src/easydiffraction/io/cif/serialize.py)) - reads `_fit_result.*` tags into the already-paired class — no - reordering, no new call. - - The read-side already restores `minimizer.type` first - ([`serialize.py:553-555`](../../../src/easydiffraction/io/cif/serialize.py)), - so the paired-class swap fires before `fit_result.from_cif` runs. No - code change is required here. - - Update the legacy-tag rejection message in - `_raise_for_legacy_analysis_tags` to include the now-removed - `_minimizer.` tags (e.g. `_minimizer.runtime_seconds`, - `_minimizer.gelman_rubin_max`) as legacy markers that should raise a - clear error rather than load silently. - - Commit: `Serialize fit outputs to _fit_result.* tags` - -- [x] **P1.12 — Confirm `_fit_state_categories` returns the paired - `fit_result`.** In `src/easydiffraction/analysis/analysis.py`, - `_fit_state_categories()` already returns - `[self.fit_parameters, self.fit_result, self.fit_parameter_correlations]` - when persisted fit state exists. After P1.6 wires the paired-class - construction, `self.fit_result` is automatically the paired - `LeastSquaresFitResult` / `BayesianFitResult` instance — no method - body change is needed. The dead branch in `_fit_state_categories` - (review-9 finding F4, open issue #101) can be cleaned up here - since this step is already reading the function. The plan does not - require the cleanup; if taken, mention "closes #101" in the commit - message. - - Commit: `Confirm fit_result paired instance flows through serializer` - -- [x] **P1.13 — Update `project.display.fit.results()` to add a - "Settings used" block.** In - `src/easydiffraction/project/display.py`, extend the existing - results-display method to print, above the current tables, a - one-section table titled "Settings used" populated from - `analysis.minimizer.*`. Use the same `render_table` machinery the - rest of the display facade uses. Commit: - `Add settings-used block to fit.results display` - -- [x] **P1.14 — Amend the five accepted ADRs listed in §"ADR".** For - each, apply the matching paragraph from the ADR's §"ADRs amended" - section: - - `minimizer-category-consolidation.md` — §1 partial-rule - qualification; §"Alternatives Considered → D" reversal record. - - `analysis-cif-fit-state.md` — §"Minimizer fit projection" rewrite. - - `runtime-fit-results.md` — closing-paragraph reference. - - `switchable-category-owned-selectors.md` — §1 paired-category - exception paragraph. - - `display-ux.md` — `project.display.fit.results()` settings-block - note. - - Update `docs/dev/adrs/index.md` to add the new ADR row under - "Accepted" (per P1.16 promotion). - - Commit: `Amend affected ADRs for minimizer input/output split` - -- [x] **P1.15 — Update tutorials.** `git grep` `docs/docs/tutorials/` - for `analysis.minimizer.` references and rewrite - each per the migration table below. The two **collapsed** rows - target existing common fields on `FitResultBase` (already written - by the existing common projection writer); they are not 1:1 - renames of the old setter/getter name. The other rows are - moved-but-keep-the-name relocations. - - | Old (removed at P1.9 / P1.10) | New | Notes | - | --- | --- | --- | - | `analysis.minimizer.runtime_seconds` | `analysis.fit_result.fitting_time` | Collapsed onto existing common field; setter remains `fit_result._set_fitting_time(...)` (already in `FitResultBase`). | - | `analysis.minimizer.iterations_performed` | `analysis.fit_result.iterations` | Collapsed onto existing common field; setter remains `fit_result._set_iterations(...)`. | - | `analysis.minimizer.objective_name` | `analysis.fit_result.objective_name` | Moved to `LeastSquaresFitResult`. | - | `analysis.minimizer.objective_value` | `analysis.fit_result.objective_value` | Moved to `LeastSquaresFitResult`. | - | `analysis.minimizer.n_data_points` | `analysis.fit_result.n_data_points` | Moved to `LeastSquaresFitResult`. | - | `analysis.minimizer.n_parameters` | `analysis.fit_result.n_parameters` | Moved to `LeastSquaresFitResult`. | - | `analysis.minimizer.n_free_parameters` | `analysis.fit_result.n_free_parameters` | Moved to `LeastSquaresFitResult`. | - | `analysis.minimizer.degrees_of_freedom` | `analysis.fit_result.degrees_of_freedom` | Moved to `LeastSquaresFitResult`. | - | `analysis.minimizer.covariance_available` | `analysis.fit_result.covariance_available` | Moved to `LeastSquaresFitResult`. | - | `analysis.minimizer.correlation_available` | `analysis.fit_result.correlation_available` | Moved to `LeastSquaresFitResult`. | - | `analysis.minimizer.exit_reason` | `analysis.fit_result.exit_reason` | Moved to `LeastSquaresFitResult`. | - | `analysis.minimizer.point_estimate_name` | `analysis.fit_result.point_estimate_name` | Moved to `BayesianFitResult`. | - | `analysis.minimizer.sampler_completed` | `analysis.fit_result.sampler_completed` | Moved to `BayesianFitResult`. | - | `analysis.minimizer.credible_interval_inner` | `analysis.fit_result.credible_interval_inner` | Moved to `BayesianFitResult`. | - | `analysis.minimizer.credible_interval_outer` | `analysis.fit_result.credible_interval_outer` | Moved to `BayesianFitResult`. | - | `analysis.minimizer.acceptance_rate_mean` | `analysis.fit_result.acceptance_rate_mean` | Moved to `BayesianFitResult`. | - | `analysis.minimizer.gelman_rubin_max` | `analysis.fit_result.gelman_rubin_max` | Moved to `BayesianFitResult`. | - | `analysis.minimizer.effective_sample_size_min` | `analysis.fit_result.effective_sample_size_min` | Moved to `BayesianFitResult`. | - | `analysis.minimizer.best_log_posterior` | `analysis.fit_result.best_log_posterior` | Moved to `BayesianFitResult`. | - - Run `pixi run notebook-prepare` to regenerate the `.ipynb` - files. - - Verification grep (must return empty against - `docs/docs/tutorials/`): - - ``` - git grep -nE 'analysis\.minimizer\.(runtime_seconds|iterations_performed|objective_value|objective_name|n_data_points|n_parameters|n_free_parameters|degrees_of_freedom|covariance_available|correlation_available|exit_reason|point_estimate_name|sampler_completed|credible_interval_inner|credible_interval_outer|acceptance_rate_mean|gelman_rubin_max|effective_sample_size_min|best_log_posterior)' docs/docs/tutorials/ - ``` - - Commit: `Update tutorials to read outputs from fit_result` - -- [x] **P1.16 — Promote ADR + update index.** - - `git mv docs/dev/adrs/suggestions/minimizer-input-output-split.md docs/dev/adrs/accepted/minimizer-input-output-split.md`. - Flip the Status header to `Accepted`. - - Move the seven `_reply-N.md` and seven `_review-N.md` siblings: keep - them next to the ADR if the project convention preserves history - under `accepted/`; delete them if the convention is to drop the - deliberation artefacts on promotion (the - `switchable-category-owned-selectors` precedent deleted them). Per - the precedent, delete on promotion. - - Update `docs/dev/adrs/index.md` — move the row for this ADR from - Suggestion → Accepted. - - Commit: `Promote minimizer-input-output-split ADR` - -- [x] **P1.17 — Phase 1 review gate.** No code change. Re-run the P1.15 - tutorial grep against `src/`, `docs/docs/tutorials/`, and - `tests/`. The `src/` and `docs/docs/tutorials/` scopes must return - empty. The `tests/` sweep is deferred to P2.1, which migrates the - tests. Then stop and request user review. After approval, proceed - to Phase 2. - -## Verification (Phase 2) - -Each command captures its log with a zsh-safe exit-code variable as -required by `AGENTS.md` → **Workflow**. - -- [x] **P2.1 — Migrate existing tests off the removed minimizer output - fields.** `git grep` `tests/` for the same patterns as P1.15. - Apply the same migration table from P1.15 — including the two - collapsed rows (`runtime_seconds` → `fitting_time`, - `iterations_performed` → `iterations`) where the setter name also - changes. Examples: - - - Reader rewrite: - `analysis.minimizer.gelman_rubin_max` → - `analysis.fit_result.gelman_rubin_max`. - - Reader rewrite with collapse: - `analysis.minimizer.runtime_seconds` → - `analysis.fit_result.fitting_time`. - - Setter rewrite (moved-but-kept name): - `analysis.minimizer._set_gelman_rubin_max(...)` → - `analysis.fit_result._set_gelman_rubin_max(...)`. - - Setter rewrite (collapsed name): - `analysis.minimizer._set_runtime_seconds(...)` → - `analysis.fit_result._set_fitting_time(...)`. - - Layout check: - - ``` - pixi run test-structure-check > /tmp/easydiffraction-test-structure-check.log 2>&1; \ - test_structure_check_exit_code=$?; \ - tail -n 200 /tmp/easydiffraction-test-structure-check.log; \ - exit $test_structure_check_exit_code - ``` - - Final stale-reference grep (all four must return empty): - - ``` - git grep -nE 'analysis\.minimizer\.(runtime_seconds|iterations_performed|objective_value|objective_name|n_data_points|n_parameters|n_free_parameters|degrees_of_freedom|covariance_available|correlation_available|exit_reason|point_estimate_name|sampler_completed|credible_interval_inner|credible_interval_outer|acceptance_rate_mean|gelman_rubin_max|effective_sample_size_min|best_log_posterior)' src/ docs/docs/tutorials/ tests/ - git grep -nE '_minimizer\.(runtime_seconds|iterations_performed|objective_value|point_estimate_name|sampler_completed|credible_interval_inner|credible_interval_outer|acceptance_rate_mean|gelman_rubin_max|effective_sample_size_min|best_log_posterior)' src/ docs/docs/tutorials/ tests/ - ``` - -- [x] **P2.2 — Add unit tests for new modules.** New tests under - `tests/unit/easydiffraction/analysis/categories/fit_result/`: - - `test_base.py` — `FitResultBase` defaults, - `_reset_result_descriptors`. - - `test_lsq.py` — `LeastSquaresFitResult` defaults; CIF round-trip of - LSQ outputs. - - `test_bayesian.py` — `BayesianFitResult` defaults including the - fixed credible interval levels; CIF round-trip. - - `test_factory.py` — pairing rule via - `LeastSquaresMinimizerBase._fit_result_class` and - `BayesianMinimizerBase._fit_result_class`. - - Layout check: - - ``` - pixi run test-structure-check > /tmp/easydiffraction-test-structure-check.log 2>&1; \ - test_structure_check_exit_code=$?; \ - tail -n 200 /tmp/easydiffraction-test-structure-check.log; \ - exit $test_structure_check_exit_code - ``` - -- [x] **P2.3 — Auto-fixes and static checks.** - - ``` - pixi run fix > /tmp/easydiffraction-fix.log 2>&1; \ - fix_exit_code=$?; \ - tail -n 200 /tmp/easydiffraction-fix.log; \ - exit $fix_exit_code - ``` - - Then: - - ``` - pixi run check > /tmp/easydiffraction-check.log 2>&1; \ - check_exit_code=$?; \ - tail -n 200 /tmp/easydiffraction-check.log; \ - exit $check_exit_code - ``` - - Iterate `pixi run check` until clean. Do not raise lint - thresholds — refactor instead. - -- [x] **P2.4 — Unit tests.** - - ``` - pixi run unit-tests > /tmp/easydiffraction-unit-tests.log 2>&1; \ - unit_tests_exit_code=$?; \ - tail -n 200 /tmp/easydiffraction-unit-tests.log; \ - exit $unit_tests_exit_code - ``` - -- [x] **P2.5 — Integration tests.** - - ``` - pixi run integration-tests > /tmp/easydiffraction-integration-tests.log 2>&1; \ - integration_tests_exit_code=$?; \ - tail -n 200 /tmp/easydiffraction-integration-tests.log; \ - exit $integration_tests_exit_code - ``` - -- [x] **P2.6 — Script tests.** - - ``` - pixi run script-tests > /tmp/easydiffraction-script-tests.log 2>&1; \ - script_tests_exit_code=$?; \ - tail -n 200 /tmp/easydiffraction-script-tests.log; \ - exit $script_tests_exit_code - ``` - - This regenerates `tmp/tutorials/projects/*` fixtures with the - new CIF layout (`_minimizer.*` settings-only, - `_fit_result.*` outputs). - -## Suggested Pull Request - -**Title:** Split minimizer settings from fit-result outputs - -**Description (user-facing):** - -`analysis.minimizer` now holds only the settings you can change — -`sampling_steps`, `max_iterations`, and the other input knobs. Once a -fit completes, every output the project records (wall time, χ², the -Bayesian diagnostics, the LSQ counters) lives on a paired -`analysis.fit_result`. The pairing happens automatically when you change -minimizer type, so users never see a separate result selector. - -`project.display.fit.results()` now prints a "Settings used" block above -the existing result tables, so the settings that produced the fit and -the outputs the fit produced are visible side-by-side without having to -open two namespaces. - -The CIF layout follows the same split: `_minimizer.*` holds settings -only, `_fit_result.*` holds outputs. Saved projects from the previous -layout do not load unchanged (the project is in beta; no legacy shims). -Tutorials and saved-fixture regeneration land in this PR. - -Incidental cleanup also bundled in this branch: the unused -`essdiffraction` development dependency is removed, and a handful of -unrelated functions are split into helpers to satisfy the project's -complexity thresholds during Phase 2 verification: -`singleton.ConstraintsHandler.apply_constraints`, -`analysis.sequential._fit_worker`, -`display.plotting._posterior_predictive_*`, and -`calculators.{crysfml,pdffit}._calculate_pattern`. From 57f63a78c7ae2d2d0f93179d6ccfaa7f9930b109 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Tue, 26 May 2026 00:17:13 +0200 Subject: [PATCH 65/65] Formatting --- docs/dev/adrs/accepted/minimizer-category-consolidation.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/dev/adrs/accepted/minimizer-category-consolidation.md b/docs/dev/adrs/accepted/minimizer-category-consolidation.md index 552b7b1dd..f7a1cdb30 100644 --- a/docs/dev/adrs/accepted/minimizer-category-consolidation.md +++ b/docs/dev/adrs/accepted/minimizer-category-consolidation.md @@ -304,8 +304,7 @@ swap warnings. ### 9. Example CIF layouts -The fit-result outputs in these examples live under `_fit_result.*` -per +The fit-result outputs in these examples live under `_fit_result.*` per [`minimizer-input-output-split.md`](minimizer-input-output-split.md); `_minimizer.*` carries only user-writable settings.