docs: migrate to Zensical and overhaul the documentation

.claude/skills/check-docs-prose/SKILL.md

@@ -0,0 +1,122 @@
+---
+name: check-docs-prose
+description: Check that the hand-written text of the microsim docs (docs/**/*.md) is still true of the code - defaults, required parameters, what each stage uses, what is and isn't modeled, numbers in captions, code examples. Use after changes to src/microsim, before a release, or when asked to audit the docs for stale or wrong statements.
+---
+
+# Check the docs' prose against the code
+
+The guide pages are written by hand, and microsim changes quickly. Statements
+like "Only `truth_space` and `sample` are required", "the default is 1 W/cm²",
+or "sCMOS pixel-to-pixel variation is not modeled" go stale silently. This skill
+is how to find them.
+
+Do not trust the docs, or your memory of the code: check every claim against the
+source, or by running it.
+
+## What the build already checks
+
+Run these first; fix what they find.
+
+```bash
+uv run pytest tests/test_docs.py        # runs every ```python block in the docs
+uv run pytest tests/test_js_physics.py  # JS chart physics (see check-js-physics)
+uv run --group docs python scripts/make_doc_figures.py   # figures, chart data
+uv run python scripts/make_notebooks.py
+uv run --group docs zensical build --strict --clean  # links and cross-references
+```
+
+- `make_doc_figures.py` takes minutes and needs network access (FPbase,
+  OpenOrganelle, Allen). It only writes gitignored files, except
+  `cosem_examples`, which is only regenerated when named. It is also the only
+  thing that runs `tutorials/two-color-cell.md` (through the `tutorials`
+  figure), and it prints values that captions quote (e.g. `first_simulation`).
+- `test_docs.py` never runs `two-color-cell.md` (its `SKIP` set). Without
+  network access it silently *drops* the pages in `NEEDS_INTERNET` rather than
+  skipping them. Check what ran with
+  `uv run pytest tests/test_docs.py --collect-only -q` (one item per page, plus
+  `README.md`), and run dropped pages another way.
+- Without `--clean`, the build can be incremental and miss problems.
+
+## What only reading can check
+
+Go page by page through `docs/` (`zensical.toml` `nav` lists them). For each
+statement of fact about microsim, find the code that makes it true, and cite it
+(`file:line`) in your report. Kinds of claims that go stale, and where they are:
+
+- **Code that isn't tested.** `` ```py `` blocks are fragments and are **not**
+  run, ```` ```json title=... ```` blocks are written out but never validated,
+  and shell commands in the docs (`about/contributing.md`,
+  `getting-started/installation.md`) are never run. Run the cheap,
+  self-contained ones in a scratch script (I/O, small spaces), validate JSON
+  examples with `ms.Simulation.model_validate_json`, and check
+  `make_doc_figures.py` arguments against its `@figure` function names.
+- **Defaults and required fields.** Tables of defaults
+  (`getting-started/first-simulation.md`, `guide/settings.md`,
+  `guide/detectors.md` "Default camera settings", `guide/channels.md`), and
+  sentences like "Only `truth_space` and `sample` are required"
+  (`guide/index.md`). Check the pydantic field definitions, including
+  `default_factory` and validators that fill values in.
+- **What each stage uses.** The stage diagrams (the mermaid blocks in
+  `index.md` and `guide/index.md`) and the stage table in `guide/index.md` list
+  which parameters feed which method. Check `Simulation.ground_truth`,
+  `filtered_emission_rates`, `optical_image`, and `digital_image`, and what they
+  call.
+- **What is and isn't modeled.** `guide/index.md` "What is (and isn't)
+  modeled", and every admonition that states a limitation or a workaround:
+
+  ```bash
+  grep -rn '^ *\(!!!\|???\) ' docs | grep -v 'docs/\(data\|notebooks\)'
+  ```
+
+  A new feature can make a "not modeled" line false; a refactor can make a
+  "this is how it works" line false. Check each one. Also check that each
+  setting the docs describe actually takes effect (e.g. that
+  `max_psf_radius_aus` changes the PSF size in the log line `Need PSF (nz,nx)`).
+- **Equations.** The formulas in `guide/photometry.md`, `guide/detectors.md`,
+  and `guide/modalities.md` should match the code step by step (normalizations,
+  units, which quantity is summed, the order of steps). The JS copies of the
+  same physics are covered by the `check-js-physics` skill.
+- **Library contents.** Tables of `optical_config.lib` filter sets,
+  `detectors.lib` presets, settings, extras (`getting-started/installation.md`),
+  and dataset or layer names. Compare with the modules and `pyproject.toml`.
+  Figure captions describe what `scripts/make_doc_figures.py` computed (e.g. the
+  shared `_cell_sim()`): check them against that script.
+- **Numbers in text and captions.** Figures and charts are regenerated on every
+  build, but numbers quoted in the text are not (e.g. "392 photons absorbed",
+  "about 6 photons", "71%"). Recompute each one from the current code, from the
+  output of `make_doc_figures.py`, or from the generated chart data in
+  `docs/data/generated/`.
+- **Names.** Every class, method, parameter, and keyword mentioned in prose or
+  `` ```py `` fragments must exist with that name and signature. The strict build
+  only checks `[name][microsim.x]` cross-references, not plain `` `code` ``.
+- **New API.** Public methods and settings added since the docs were written
+  (`git log` on `src/microsim`) should be mentioned where they belong (e.g. a
+  new `Simulation` method in `guide/index.md` and `api/index.md`).
+- **Links to issues.** Statements tied to a GitHub issue (e.g. #152): check
+  whether the issue is closed and the statement still holds.
+
+## Style
+
+The docs are plain and scientific. Check these with a search, then by reading:
+
+```bash
+grep -rn '—' docs --include='*.md' | grep -v 'docs/\(data\|notebooks\)'   # em dashes
+grep -rniE '\b(recommend|best for|ideal for|natural choice|should use|which [a-z]+ (when|to use)|beats|outperform)' \
+  docs --include='*.md' docs/javascripts/charts | grep -v 'docs/\(data\|notebooks\)'
+```
+
+- No em dashes, and no marketing language.
+- The docs describe what microsim computes; they don't advise readers on which
+  hardware to use (e.g. "which camera when", "the natural choice", "best for
+  low light"). Rephrase such lines as statements of what the model or the
+  physics does. Advice on using microsim itself (e.g. "Prefer Zarr") is fine.
+- Chart labels and captions in `docs/javascripts/charts/*.js` count too.
+
+## Reporting and fixing
+
+- Fix clear factual errors directly, in the docs' existing style, unless you
+  were asked for a report only.
+- If the code looks wrong rather than the docs (a doc describes intended behavior
+  the code doesn't implement), don't change either; report it.
+- Report what you checked, what you changed, and anything you could not verify,
+  with `file:line` references for both the docs and the code.

.claude/skills/check-js-physics/SKILL.md

@@ -0,0 +1,131 @@
+---
+name: check-js-physics
+description: Check that the JavaScript physics behind the docs' interactive charts still matches microsim's Python. Use after changing photometry, detector, or optical-config code in src/microsim, after editing docs/javascripts/physics or docs/javascripts/charts, or when asked to review the docs charts for drift.
+---
+
+# Check the docs' JavaScript physics against microsim
+
+The interactive charts in the docs (`docs/javascripts/charts/*.js`) recompute some
+of microsim's physics in the browser, so readers can vary parameters freely. That
+code is a copy, and copies drift. This skill is how to check it.
+
+## Layout
+
+- `docs/javascripts/physics/*.js`: the copied physics. Pure functions, no DOM, and
+  no imports from outside `physics/`. Every function that mirrors microsim has a
+  comment naming the Python it mirrors (helpers like `scale`, `total`, and `rng`
+  don't).
+- `docs/javascripts/charts/*.js`: rendering. These should contain **no physics**:
+  they call the physics modules, and only choose parameters, axes, and labels.
+  `lib.js` (plot helpers), `index.js` (mounting), and `photometry-data.js`
+  (decodes `photometry.json`) are infrastructure.
+- `scripts/make_doc_figures.py`: exports the data the charts load, written to
+  `docs/data/generated/` (gitignored): `photometry_data` → `photometry.json`,
+  `detector_data` → `detectors.json`, `noise_data` → `noise.json`,
+  `pinhole_data` → `pinhole.json`. Spectra, presets, default lens values, and
+  images computed by microsim itself.
+- `tests/test_js_physics.py` and `tests/js/*.mjs`: run the physics modules in node
+  on the same inputs as microsim, and compare. Skipped if `node` is not installed.
+
+### The charts
+
+| Chart | Page | Physics used | Data | Starting values |
+| --- | --- | --- | --- | --- |
+| `spectra.js` | `guide/channels.md` | `absorptionRate`, `crossSection`, `filteredEmissionRate`, `fluxDensity`, `irradiance`, `mulArrays` | `photometry.json` | fluorophore and channel menus |
+| `crosstalk.js` | `guide/channels.md` | `photonBudget` | `photometry.json` (incl. `lens`) | EGFP + tdTomato, FITC 500 LP + TRITC; numbers in the caption |
+| `photon-budget.js` | `guide/photometry.md` | `photonBudget` | `photometry.json` (incl. `lens`) | EGFP, FITC, 1 W/cm², 100 ms; "about 6 photons" in the page text |
+| `saturation.js` | `guide/photometry.md` | `absorptionRate`, `totalEmissionRate`, `halfSaturationIrradiance`, `maxEmissionRate` | `photometry.json` | irradiance bands from the table in `guide/channels.md` |
+| `noise.js` | `guide/detectors.md` | `simulateCamera`, `snr`, `emccdExcessNoiseFactor` | `noise.json`, `detectors.json` (`cameras`) | camera presets shared with `detector-snr.js` |
+| `detector-snr.js` | `guide/detectors.md` | `snr`, `emccdExcessNoiseFactor` | `detectors.json` (`cameras`) | ≈71% cap in the caption |
+| `emccd-enf.js` | `guide/detectors.md` | `emccdExcessNoiseFactor` | `detectors.json` (`cameras.EMCCD`) | |
+| `pmt-gain.js` | `guide/detectors.md` | `pmtGain`, `pmtExcessNoiseFactor` | `detectors.json` (`pmts`) | F ≈ 1.3 / 1.5 in the page text |
+| `hyd-dead-time.js` | `guide/detectors.md` | `hydCountingEfficiency`, `hydMaxRate` | `detectors.json` | "about 6% low at 40 Mcps" in the page text |
+| `pinhole.js` | `guide/modalities.md` | none (displays microsim PSFs) | `pinhole.json` | |
+
+Update this table when you add, rename, or remove a chart.
+
+## Procedure
+
+1. **Run the tests.** `uv run pytest tests/test_js_physics.py -v`. They must pass,
+   and must not be skipped (install node if they are). A failure is drift: decide
+   which side is right, and fix the other, or both, together.
+
+2. **Read each mirrored pair.** The tests cover the cases they construct, not every
+   branch. For every function in `docs/javascripts/physics/`, open the Python it
+   names and compare line by line: constants, normalizations (sum vs. peak vs.
+   integral), units (nm vs. m, ns vs. s, W vs. photons), defaults (`|| 1`, `?? null`),
+   how arrays are aligned, and rounding (`np.round` rounds halves to even, unlike
+   `Math.round`; use `roundHalfEven`). Current pairs:
+
+   | JavaScript (`docs/javascripts/physics/`) | Python (`src/microsim/` unless noted) |
+   | --- | --- |
+   | `photometry.js` `decodeSpectrum` | `scripts/make_doc_figures.py` `spectrum_json` (inverse) |
+   | `photometry.js` `mulSpectra` | `schema/spectrum.py` `Spectrum._intensity_op` (outer join, fill 0) |
+   | `photometry.js` `mulArrays` | xarray arithmetic (inner join) |
+   | `photometry.js` `inverted` | `schema/spectrum.py` `Spectrum.inverted` |
+   | `photometry.js` `excitationPath`, `emissionPath` | `schema/optical_config/config.py` `OpticalConfig.excitation`, `.emission`, `._merge` |
+   | `photometry.js` `illumination` | `OpticalConfig.illumination`, `LightSource.scaled_spectrum` |
+   | `photometry.js` `irradiance`, `fluxDensity` | `OpticalConfig.irradiance`, `.illumination_flux_density` |
+   | `photometry.js` `crossSection` | `schema/sample/fluorophore.py` `Fluorophore.absorption_cross_section` |
+   | `photometry.js` `absorptionRate`, `saturationParameter`, `totalEmissionRate`, `filteredEmissionRate` | `OpticalConfig` methods of the same names |
+   | `photometry.js` `halfSaturationIrradiance`, `maxEmissionRate` | derived from `OpticalConfig.saturation_parameter` and `.total_emission_rate` |
+   | `photometry.js` `collectionEfficiency` | `schema/lens.py` `ObjectiveLens.collection_efficiency` |
+   | `photometry.js` `photonBudget` | `Simulation.filtered_emission_rates` (tested end to end) |
+   | `detectors.js` `emccdExcessNoiseFactor` | `schema/detectors/_camera.py` `CameraEMCCD.excess_noise_factor` |
+   | `detectors.js` `pmtGain`, `pmtExcessNoiseFactor` | `schema/detectors/_point.py` `PMT.multiplication_gain`, `.excess_noise_factor` |
+   | `detectors.js` `hydCountingEfficiency`, `hydMaxRate` | `schema/detectors/_point.py` `HyD._counting_efficiency` |
+   | `detectors.js` `roundHalfEven` | `np.round`, as used in `_Detector.simulate` |
+   | `detectors.js` `snr` | derived from `schema/detectors/_detector.py` `_Detector.simulate` and `apply_multiplication_gain` (tested against a Monte Carlo run of `render`) |
+   | `detectors.js` `simulateCamera` | `_Detector.simulate` steps 1-5 for cameras, and `apply_multiplication_gain` (tested statistically against `render`: mean, variance, quantiles) |
+   | `random.js` `poisson`, `normal`, `gamma` | the distributions of `schema/backend.py` `poisson_rvs`, `norm_rvs`, and `scipy.stats.gamma` (same distributions, not the same random streams) |
+   | `random.js` `rng` | no mirror (seeded PRNG for the browser) |
+
+   Charts that only display arrays computed by microsim (`pinhole.js`, the noise-free
+   image in `noise.js`) have nothing to mirror; check instead that the export
+   function in `scripts/make_doc_figures.py` uses microsim, and that captions
+   describe what was computed.
+
+   If the Python has gained a step (a new factor, a new parameter that changes the
+   result), the JavaScript needs it too, or the function's comment (and the chart's
+   caption, if it matters there) must say what it leaves out.
+
+3. **Find physics outside the physics modules.** Search the charts for formulas:
+
+   ```bash
+   grep -nE '1e-?[0-9]|Math\.(exp|sqrt|log|pow)|\*\* ?-?[0-9]|/ ?\(1 \+' docs/javascripts/charts/*.js
+   ```
+
+   Ratios of physics outputs for display (e.g. "fraction passed") are fine.
+   Anything that computes a physical quantity belongs in `physics/`, with a mirror
+   comment and a test.
+
+4. **Check the exported data.** `scripts/make_doc_figures.py` writes presets and
+   spectra from microsim objects. Make sure the chart uses the exported values
+   (e.g. PMT presets, the HyD dead time, the `ObjectiveLens` defaults), not numbers
+   typed into the JavaScript. If two charts share starting values (e.g. the camera
+   presets of `noise.js` and `detector-snr.js`), both must read them from the same
+   export. Display-only settings (slider ranges, `electronsPerAdu` chosen to fit
+   16 bits) can stay in the chart.
+
+5. **Check the captions and page text.** Numbers quoted in the docs (see the
+   "Starting values" column above) should match what the chart shows at its
+   starting values. Regenerate the data first; without it, a chart shows "This
+   interactive chart failed to load":
+
+   ```bash
+   # noise_data needs network access (FPbase, OpenOrganelle) the first time
+   uv run --group docs python scripts/make_doc_figures.py photometry_data detector_data noise_data pinhole_data
+   ```
+
+   Then either compute the numbers in node, importing `docs/javascripts/physics/*.js`
+   with the generated JSON (fastest), or serve the docs and look:
+
+   ```bash
+   uv run --group docs zensical serve
+   ```
+
+## When adding a chart or a physics function
+
+- Put the physics in `docs/javascripts/physics/`, with a comment naming the Python.
+- Add a case to `tests/test_js_physics.py` that compares it with microsim.
+- Add rows to the tables above.

.github/workflows/docs.yml

@@ -10,18 +10,30 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
-  build:
-    if: github.event_name == 'pull_request'
+  docs:
     runs-on: ubuntu-latest
+    permissions:
+      contents: write
     steps:
       - uses: actions/checkout@v7
       - uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7  # v10.2.0
-      - run: uv run --group docs mkdocs build --strict
-
-  deploy:
-    if: github.ref == 'refs/heads/main'
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v7
-      - uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7  # v10.2.0
-      - run: uv run --group docs mkdocs gh-deploy --force
+      # PSFs, ground truth, and OpenOrganelle downloads used by the figures
+      - uses: actions/cache@v4
+        with:
+          path: ~/.cache/microsim
+          key: microsim-docs-${{ hashFiles('scripts/make_doc_figures.py', 'uv.lock') }}
+          restore-keys: microsim-docs-
+      - name: Generate figures and chart data
+        run: uv run --group docs python scripts/make_doc_figures.py
+      # tutorial notebooks, published with the site (Colab opens them from gh-pages)
+      - name: Generate notebooks
+        run: uv run python scripts/make_notebooks.py
+      - name: Build
+        run: uv run --group docs zensical build --strict
+      # zensical has no gh-deploy; push the built site to the gh-pages branch
+      - name: Deploy
+        if: github.ref == 'refs/heads/main'
+        run: |
+          git config user.name "github-actions[bot]"
+          git config user.email "github-actions[bot]@users.noreply.github.com"
+          uvx ghp-import --no-jekyll --push --force site

.gitignore

@@ -108,3 +108,10 @@ microsim/_version.py
 microsim/**/*.c
 jrc_*
 dask-worker-space*
+
+# documentation figures, made by scripts/make_doc_figures.py
+docs/images/generated/
+docs/data/generated/
+
+# tutorial notebooks, made by scripts/make_notebooks.py
+docs/notebooks/

README.md

@@ -20,10 +20,16 @@ The goal of this library is to generate highly realistic simulated data such as
 
 ## Documentation
 
-Start with the [tutorial](https://www.talleylambert.com/microsim/tutorial/), and read about the
-[simulation stages](https://www.talleylambert.com/microsim/stages/), then see the 
-[API Reference](<https://tlambert03.github.io/microsim/api/>) for details
-on the `Simulation` object and options for all of the fields.
+Documentation is at <https://talleylambert.com/microsim/>. Start with
+[First simulation](https://talleylambert.com/microsim/getting-started/first-simulation/),
+read the [User guide](https://talleylambert.com/microsim/guide/) for how each part
+of the simulation works, and see the
+[API reference](https://talleylambert.com/microsim/api/) for every option.
+
+> [!WARNING]
+> microsim is in early development and changes quickly. Releases may break the API
+> or change simulation results for the same parameters. If you depend on it, pin a
+> specific version and record it with your results.
 
 ## Installation
 
@@ -67,7 +73,7 @@ paying attention to your GPU requirements.  Support for torch is planned.
 ## Usage
 
 Construct and run a
-[`microsim.Simulation`](https://www.talleylambert.com/microsim/api/#microsim.schema.simulation.Simulation)
+[`microsim.Simulation`](https://talleylambert.com/microsim/api/microsim/schema/simulation/)
 object.
 
 ```python

docs/about/citing.md

@@ -0,0 +1,23 @@
+---
+icon: lucide/quote
+---
+
+# Citing microsim
+
+If you use microsim in published work, please cite the software through its
+Zenodo record:
+
+[![DOI](https://zenodo.org/badge/483645397.svg)](https://doi.org/10.5281/zenodo.18942661)
+
+The Zenodo page has citation text in several formats, and a DOI for each
+release. Since results can differ between versions, cite the version you used.
+
+The ground truth from OpenOrganelle and the spectra from FPbase come from
+other projects; if you use them, please also cite:
+
+- Heinrich, L. et al. "Whole-cell organelle segmentation in volume electron
+  microscopy." *Nature* 599, 141–146 (2021).
+  [doi:10.1038/s41586-021-03977-3](https://doi.org/10.1038/s41586-021-03977-3)
+- Lambert, T. J. "FPbase: a community-editable fluorescent protein database."
+  *Nature Methods* 16, 277–278 (2019).
+  [doi:10.1038/s41592-019-0352-8](https://doi.org/10.1038/s41592-019-0352-8)