Repository navigation
Review follow-up: dedupe the stain fit reductions - #21
Closed
selmanozleyen wants to merge 2 commits into
Closed
selmanozleyen wants to merge 2 commits into
selmanozleyen wants to merge 2 commits into
Conversation
selmanozleyen
force-pushed
the
stain/pr4-pooled-fit-review
branch
from
October 9, 2026 20:24
a7a01ee to
7bdae93
Compare
- fit_reinhard pools per-channel (count, sum, sum of squares) instead of materialising the tissue pixels, so the Reinhard fit is lazy again and a pooled cohort never holds a frame resident (103 -> 38 MiB peak on a 2000x2000 pool of two). - accumulate those moments in float64. sigma falls out of E[x^2] - E[x]^2, whose terms are ~5000x the variance they bracket; in float32 that cancellation costs ~1% of sigma and makes the answer depend on chunking (eager 0.10272 vs chunked 0.10410 on the same slide). - drop _masked_channel_stats. apply_reinhard was a second implementation of the same reduction and had already diverged from the fit; both now go through _tissue_lab_moments + _stats_from_moments. - raise StainFittingError (with image_key) on empty tissue, matching the decomposition path, instead of a bare ValueError in its own format. - single _resolve_fit_inputs for the coarse image + tissue mask, shared by estimate_white_point, fit_stain_reference and _normalize_stains. - derive the method vocabulary from StainMethod once in _reference.py; drop the second copy in _normalize.py. - document that the pool is a pool of pixels, weighted by tissue pixel count. - tests: assert the error names the slide (the old pattern also matched the message's own trailing "may be blank"), and add pool-of-duplicates invariance, which pool-of-one cannot catch.
selmanozleyen
force-pushed
the
stain/pr4-pooled-fit-review
branch
from
October 9, 2026 20:30
7bdae93 to
3c52bdf
Compare
luminosity_foreground_mask and absorbance_foreground_mask had no production caller: every fit path composes the conversion and the *_from_lab/_from_sda primitives itself, so the mask reads from the same lazy graph the statistics do, and dask materialises the conversion once. Only tests reached the wrappers. Both conversions already guard the channel dim, so the tests keep their coverage by composing the primitives the way the fits do.
selmanozleyen
force-pushed
the
stain/pr4-pooled-fit-review
branch
from
October 9, 2026 20:32
3c52bdf to
7b12ad0
Compare
Owner
Author
|
Landed directly on scverse#1209 (commits 0b961ed, 7b12ad0). Closing as redundant. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Review of scverse#1209, applied. Base is that PR's head, so this diff is only the review changes.
Correctness / efficiency
The Reinhard fit stopped being lazy.
_tissue_lab_pixelsmaterialised a3xHxWfloat64 frame per image to take a mean and a std. The function it replaced documented the opposite ("never materialising the full image"). It now reduces to per-channel(count, sum, sum of squares)in one bundled pass; pooling is an element-wise sum of moments, so nothing is resident. A/B on a 2000x2000 pool of two, dask-chunked:The reductions have to run in float64.
_working_dtypeis float32, and a float32 sum of squares over a whole slide loses the variance to cancellation (~1% on L) and stops being chunk-stable. The cast is per-chunk, so it stays lazy. Chunked and eager now agree to 1e-10. Thebeforesigma above is itself slightly off for this reason.One error convention.
StainFittingError(reason, image_key=...)exists precisely for attributing a failure to a slide in a cohort fit, and the decomposition path already used it. The Reinhard path raised a bareValueErrorin its own format, so a caller skipping bad slides needed two excepts and two parsers.Deduplication
_masked_channel_statsis gone.apply_reinhardstandardises by the source image's own tissue statistics -- that is a Reinhard fit of the source. So the module held two implementations of the same reduction, and they had already diverged (fit in float64 after the fix above, apply in float32). Both paths now run_tissue_lab_moments->_stats_from_moments. Net -1 function;apply_reinharddrops its own Lab conversion, stays lazy, and inherits float64 for free._resolve_fit_inputs. "Resolve the coarse image, validate its range, resolve its tissue mask" was written out three times (estimate_white_point,fit_stain_reference,_normalize_stains). Thevalidate_rgb_rangestep is exactly the kind of guard that gets added to one site and not the others.The method vocabulary.
_VALID_METHODSand_DECOMPOSITION_METHODSwere each declared twice, in_reference.pyand_normalize.py, alongside theStainMethodliteral that is the real source of truth. A fourth method meant four edits and a silent half-registration. Now derived once viaget_args(StainMethod).Docs / tests
image_keydocstring now states the pool is a pool of pixels: each slide weighted by tissue pixel count at the resolvedscale, not equally. Verified empirically; worth knowing whenscale="auto"resolves per-image.match="blank"could not fail -- it also matched the message's own trailing "may be blank." Nowmatch=r"^\[blank\] ", which only the key produces.pytest.raises((ValueError, RuntimeError))->(ValueError, StainFittingError).test_pool_of_duplicates_is_single: pooling an image with itself must not move the reference. This is what catches a broken concatenate or weighting;pool-of-one == singlecannot, since both sides run the identical branch. It found a real property in passing -- exact for Reinhard, ~2e-5 for Macenko, becausenp.percentileinterpolates between order statistics and doubling N moves the interpolation point._masked_channel_statstests move onto_tissue_lab_moments, so they now exercise the path both callers use.len(set(keys))and names the offending key;_unique_order_preservingimport dropped.148 stain tests pass locally, including the
normalize_stainimage baselines -- the float64 shift on the apply path did not move a pixel.Left alone
das/masks/keysnormalisation is still duplicated across the two fits. It is plumbing with no numerics, so divergence costs nothing, and every home for a shared helper is a module it does not belong in._reinhard_maskand the mask branch in_tissue_odlook alike but resolve in different spaces against different params;_mask.pysays the two variants exist on purpose.(
luminosity_foreground_mask/absorbance_foreground_maskwere in this list as dead-but-out-of-scope; they are now deleted -- see below.)Dead code
luminosity_foreground_maskandabsorbance_foreground_maskare gone. Neither had a production caller: every fit path composes the conversion and the*_from_lab/*_from_sdaprimitive itself, precisely so the mask and the statistics read from the same lazy graph and dask materialises the conversion once. Only tests reached the wrappers. Both conversions already guard the channel dim, so the tests keep their coverage by composing the primitives the way the fits do.