feat: LLM-driven improvements on file writing and testing

.github/workflows/ci.yml

@@ -46,7 +46,7 @@ jobs:
         run: |
           python -m pip install -U pip
           python -m pip install torch==2.9.0 --index-url https://download.pytorch.org/whl/cpu
-          python -m pip install --group dev .
+          python -m pip install --group dev ".[processing]"
 
       - name: 🧪 Run Tests
         env:

.gitignore

@@ -87,6 +87,7 @@ celerybeat-schedule
 
 # virtualenv
 .venv
+.uv-cache/
 venv/
 ENV/
 
@@ -120,7 +121,7 @@ src/cucim
 # Local audit notes, experiment reports, and diagnostic outputs
 /diagnostics/
 /AUDIT_*.md
-/docs/*_audit.md
+/docs/**/*audit*.md
 # Temporary diagnostic runs and generated stitching reports
 /.tmp/
 /stitching_*_diagnostic.json
@@ -130,3 +131,7 @@ src/cucim
 /.agents/
 
 uv.lock
+
+/.test-*
+# Locally supplied reference papers
+/tests/reference/papers/

README.md

@@ -9,11 +9,19 @@ Read the [documentation](https://qi2lab.github.io/opm-processing-v2/), starting
 with [installation](https://qi2lab.github.io/opm-processing-v2/getting-started/installation/)
 and the [first workflow](https://qi2lab.github.io/opm-processing-v2/getting-started/quickstart/).
 
+To select and reconstruct raw regions interactively, follow the
+[ROI export guide](docs/guides/roi.md).
+
 ## Quick start
 
 Requires Python 3.12 and [uv](https://docs.astral.sh/uv/).
 Windows and Linux installations include CUDA dependencies by default; Windows
 also requires the [cuCIM setup](docs/getting-started/installation.md#cucim-on-windows).
+The base distribution provides the yaozarrs/TensorStore IO SDK. Processing,
+visualization and their existing PyTorch dependency are in the `processing`
+extra; the default checkout workflow still selects them. IO consumers install
+the base distribution without extras. In a checkout, `uv sync --no-default-groups`
+selects only IO dependencies.
 
 ```bash
 uv sync
@@ -22,6 +30,10 @@ uv run fuse "/path/to/acquisition"
 uv run export-ome-tiff "/path/to/acquisition"
 ```
 
+Deconvolution reads camera and optical calibration from OME-Zarr. See the
+[required metadata and solver parameters](docs/methods/deconvolution.md).
+Missing read-noise calibration selects the no-read-noise model.
+
 ## Commands
 
 Each command's reference describes its arguments, options, and an example.
@@ -32,18 +44,16 @@ Append `--help` to any command for its installed options and defaults.
 | [inspect-opm](docs/reference/cli/inspect-opm.md) | Inspect acquisition metadata. |
 | [process](docs/reference/cli/process.md) | Calibrate, deconvolve, and deskew. |
 | [fuse](docs/reference/cli/fuse.md) | Register and blend processed tiles. |
-| [display](docs/reference/cli/display.md) | View outputs and select an ROI. |
-| [process-ROI](docs/reference/cli/process-roi.md) | Process and fuse selected raw regions. |
 | [export-ome-tiff](docs/reference/cli/export-ome-tiff.md) | Export fused images as compressed OME-BigTIFF. |
+| [export-rois](docs/reference/cli/export-rois.md) | Select multiple ROIs and reconstruct raw regions into OME-Zarr and optional TIFF. |
 | [export-projections](docs/reference/cli/export-projections.md) | Export annotated projections and movies. |
 | [encode-projections](docs/reference/cli/encode-projections.md) | Encode existing projection TIFFs as movies. |
-| [convert-timelapse](docs/reference/cli/convert-timelapse.md) | Export raw scan-plane timelapses. |
 
 ## Development
 
 See [contributing](docs/development/contributing.md),
 [running tests](docs/development/testing.md), and
-[simulation examples](docs/examples/index.md).
+[deconvolution usage](docs/examples/index.md).
 
 Preview the documentation in a separate environment:
 

docs/development/contributing.md

@@ -16,10 +16,9 @@ units, returned values, and meaningful algorithm conventions. Keep helpers and
 state limited to what the calculation needs. The controlled acquisition format
 is the input contract; support additional formats only when explicitly required.
 
-Core processing belongs in `src/opm_processing`. Simulations, experiments, and
-figure generators belong in `scripts/`, with a clear opening docstring and module
-imports. Generated reports, figures, and temporary acquisitions belong under
-ignored `diagnostics/` or pytest temporary directories.
+Core processing belongs in `src/opm_processing`. Simulation setup, forward models, configurations, and comparison figure generators
+belong under `tests/`. `scripts/` contains only a usage example. Generated outputs
+belong under ignored `diagnostics/` or pytest temporary directories.
 
 ## Validate changes
 
@@ -29,8 +28,9 @@ uv run pytest --strict-markers
 git diff --check
 ```
 
-Tests must be numerical unit tests or simulated disk-to-disk integration tests.
-Use the [testing guide](testing.md) for CUDA selection and physical acceptance.
+Use only unit and integration tests, with shared fixtures and installed storage
+backends in isolated temporary directories.
+Use the [testing guide](testing.md) for CPU and CUDA commands.
 Run performance measurements outside pytest and record the conditions and
 comparison baseline in ignored `diagnostics/`. Deskew, fusion, and deconvolution
 optimizations must retain their stated geometry and numerical behavior.

docs/development/testing.md

@@ -1,78 +1,23 @@
-# Running and adding tests
+# Running tests
 
-Run these commands from the repository root. They use PowerShell syntax.
-
-## Set up and run
-
-Install the development dependencies and run tests that do not require CUDA:
+Run the unit and integration suites from the repository root using an installed development environment:
 
 ```powershell
-uv sync --group dev --no-default-groups
 uv run --no-sync python -m pytest tests/cpu --strict-markers -q
-```
-
-For a CUDA workstation, follow the [GPU installation instructions](../getting-started/installation.md),
-install the GPU dependencies, and require working CUDA during testing:
-
-```powershell
-uv sync --group dev --extra gpu
-$env:OPM_REQUIRE_GPU = "1"
 uv run --no-sync python -m pytest tests/gpu --strict-markers -q
-Remove-Item Env:OPM_REQUIRE_GPU
 ```
 
-Without `OPM_REQUIRE_GPU=1`, unavailable CUDA causes GPU tests to skip. Check
-the summary for skipped tests before treating a run as GPU validation.
+CPU tests run in GitHub Actions. GPU tests require CUDA; inspect the pytest summary for unavailable hardware.
+The existing GPU environment does not need to be synchronized to run tests.
+
+Shared fixtures provide controlled metadata, file structures and pixel arrays.
+Tests use production algorithms and installed packages. Synthetic acquisitions
+share production optics through the forward model in `tests/simulation/`.
+See [test contribution rules](https://github.com/qi2lab/opm-processing-v2/blob/perf/numba-deskew-corrections/tests/CONTRIBUTING.md)
+for adding or changing cases.
 
-## Run a smaller selection
+Run repository checks before submitting:
 
 ```powershell
-uv run --no-sync python -m pytest -q -m unit
-uv run --no-sync python -m pytest -q -m integration
-uv run --no-sync python -m pytest -q tests/cpu/test_depth_intensity_normalization.py
+uv run --no-sync prek run --all-files
 ```
-
-The CPU directory is the GitHub Actions suite and always uses the real CPU
-registration backend. The GPU directory requires CUDA or NVENC. Running
-`python -m pytest` collects both; `-m cpu` and `-m gpu` select the same groups.
-Add `-x` to stop at the first failure, or `-vv` to see each test case as it runs.
-
-## Add a test
-
-Put tests in `test_*.py` files and choose `@pytest.mark.unit` for an isolated
-calculation or state transition, or `@pytest.mark.integration` for a simulated
-object processed from disk input to verified disk output. Use `tmp_path` for
-generated files so tests do not depend on local acquisitions.
-
-Use the shared factories in `tests/fixtures/`; `conftest.py` loads them through
-`pytest_plugins` and enforces test categories. Keep setup with the logic it models:
-
-| Fixture module | Responsibility |
-| --- | --- |
-| `acquisition` | Upstream scan layouts, camera metadata and processing options |
-| `camera`, `illumination` | Detector transfer and known sensitivity fields |
-| `optics`, `tiling` | Physical objects, PSFs, placement errors and acceptance settings |
-| `tensorstore`, `storage`, `export` | Unit dataset mocks, saved NGFF images and OME provenance |
-| `live`, `roi`, `state` | Acquisition lifecycle, physical regions and durable checkpoints |
-| `fusion`, `hardware` | Isolated numerical operators and execution requirements |
-
-Independent optical models and measurements belong in `tests/reference/`.
-Reuse a factory and override the settings that define the case. Keep expected
-physical values and tolerances visible in the test. Integration tests must
-reopen saved pixels; simulated return values and call counts are insufficient.
-Unit tests use the shared TensorStore doubles with functional pixel reads and
-writes. Integration tests retain real TensorStore datasets on disk.
-
-For a concrete integration example, see
-[the depth-normalization test](https://github.com/qi2lab/opm-processing-v2/blob/perf/numba-deskew-corrections/tests/cpu/test_depth_intensity_normalization.py). It:
-
-1. Simulates an object with known geometry and brightness.
-2. Writes its camera pixels and acquisition metadata to disk.
-3. Runs processing and fusion using those files.
-4. Reopens the outputs and compares the saved data with expected values.
-
-Choose tolerances from sampling, camera quantization, or another stated error
-bound. Run the affected tests and `uv run --no-sync prek run --all-files` before
-submitting a change. The review rules below also apply to numerical optimizations.
-
---8<-- "tests/CONTRIBUTING.md"

docs/examples/index.md

@@ -1,21 +1,30 @@
-# Simulation scripts
+# Deconvolution usage
 
-Repository simulations and figure generators live in `scripts/`. Run them as
-modules from the repository root so package imports resolve. Their opening
-docstrings describe inputs, calculations, and outputs. Generated data and figures
-belong in ignored `diagnostics/` directories.
+The example processes an OME-Zarr using its recorded acquisition calibration:
 
 ```bash
-uv run python -m scripts.opm_simulation --output diagnostics/meridian_sphere
-uv run --with matplotlib python -m scripts.plot_opm_simulation diagnostics/meridian_sphere/scan_0.2um
+uv run python scripts/deconvolve.py "/path/to/acquisition.ome.zarr" --output "/path/to/processed"
 ```
 
-| Workflow | Description |
-| --- | --- |
-| [Meridian sphere](../meridian_sphere_simulation.md) | Known fluorescent object, optical forward model, and camera sampling |
-| [Publication figure](../opm_publication_figure.md) | Acquisition and reconstruction panels at multiple scan spacings |
-| [Fixed-plane series](../fixed_plane_sampling_series.md) | Object passage through a stationary oblique plane |
+Its Python call is:
 
-Each workflow page records the physical assumptions and commands used for its
-figures. Experimental outcomes apply to those specified models and sampling;
-they do not establish performance for every specimen or optical mismatch.
+```python
+from pathlib import Path
+from opm_processing.process import process
+
+process(
+    root_path=Path("/path/to/acquisition.ome.zarr"),
+    deconvolve=True,
+    save_float32=True,
+    output=Path("/path/to/processed"),
+)
+```
+
+The pipeline selects each channel, timepoint and position using the stored axes.
+Camera calibration and scan geometry come from the OME-Zarr; missing read-noise
+calibration selects the no-read-noise model. Theoretical PSFs require recorded
+optical calibration and emission wavelengths. Measured PSFs can be supplied
+instead with `decon_psf_paths=[Path(...), ...]`, in channel order.
+
+See [deconvolution parameters and metadata](../methods/deconvolution.md) for the
+array API, units, required optical fields, illumination handling and FFT tiling.

docs/getting-started/installation.md

@@ -2,6 +2,16 @@
 
 Run these commands from a checkout of this repository.
 
+The base distribution is the public yaozarrs/TensorStore IO and metadata SDK.
+It does not require PyTorch, CuPy, napari or deconvolution packages. The
+`processing` extra retains the full processing/visualization dependency stack;
+the `gpu` extra includes processing and CUDA dependencies. The default checkout
+groups select processing and, on Windows/Linux, GPU support as before.
+
+For IO only, use `uv sync --no-default-groups` in this checkout or add the base
+`opm-processing-v2` dependency to a consumer project. Import the public reader
+with `from opm_processing.dataio import read_processed_snapshot`.
+
 Python 3.12 and [`uv`](https://docs.astral.sh/uv/) are required.
 
 ```bash