Memoize.function: fix cross-closure cache collision - #3
Merged
Merged
Conversation
loadCache()/saveCache() key on the raw call arguments only, so two different Memoize.function(fun) wrappers sharing the default cache `suffix` can collide whenever they're called with the same argument value -- one wrapper silently gets served *another* wrapper's cached result instead of evaluating its own `fun`. Reproduced: make_fun <- function(offset) function(x) x + offset f1 <- Memoize.function(make_fun(3000)) f2 <- Memoize.function(make_fun(4000)) f1(5) # 3005 (correct) f2(5) # 3005 (WRONG: should be 4005 -- served f1's cached value) This is exactly the shape that arises from wrapping a fresh closure in a loop over several models/datasets (e.g. cristalgo's EGRI/EGRIw/WEGRI robust-inversion algorithms: `mesh_exsets(f = Memoize.function(fi), ...)` inside a per-seed loop, where `fi` closes over that seed's own fitted model). A stale cross-seed value fed into a root-finder that assumes it is bisecting one consistent function silently corrupts its bisection invariants; DiceView:::root() also has no working recursion-depth guard (`max.rec` defaults to NA), so a single corrupted value is enough to make it recurse until R's node stack overflows. Fix: each call to Memoize.function() now gets its own private cache namespace (`.memoize_id`, folded into the cache key), so independent wrappings can never collide regardless of argument overlap, while a single wrapper's own repeated-argument caching (the documented example: `F <- Memoize.function(f); F(5); F(6); F(5)`) is unchanged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A5w4ZLUZpoHhtsbGyWcVtc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
Memoize.function(fun)'s cache (R.cache::loadCache/saveCache) keys purely on the call arguments -- not on whichfunis being wrapped. Two independentMemoize.function(fun)wrappers that happen to share the default cachesuffixand are called with the same argument value will collide: the second wrapper silently gets served the first wrapper's cached result instead of ever evaluating its ownfun.Reproduced:
Where this actually bites
This is exactly the shape produced by wrapping a fresh closure in a loop over several models/datasets -- e.g. cristalgo's EGRI/EGRIw/WEGRI robust-inversion algorithms, which do
mesh_exsets(f = Memoize.function(fi), ...)once per seed inside afor (seed in ...)loop, whereficloses over that seed's own fitted Kriging/WarpKriging model. Sinceroot()'s bisection deterministically probes similar midpoints over the same fixed input domain across seeds, a later seed'sfi(xi)can be silently served an earlier seed's stale cached value at the samexi.root()assumesfis a single, self-consistent function throughout its bisection; a corrupted value breaks that invariant.root()also has no working recursion-depth guard (max.recdefaults toNA, soisTRUE(rec > max.rec)never fires), so a single bad value is enough to make it recurse until R's node/pointer-protection stack overflows -- observed in practice as a Slurm job hanging 80+ minutes with zero progress on one seed (while 9 other seeds using the identical code completed normally in under a minute each). Confirmed by reproducing the exact hung state and showing that (a) the underlying model'spredict()and the wrapped criterion function are both fast and well-behaved in isolation, (b) the same call throughmesh_exsets()reproduces the hang, and (c) removing only theMemoize.function()wrapper (or, as here, fixing it) makes the identical call complete normally.Fix
Each call to
Memoize.function()now gets its own private cache namespace (.memoize_id, generated once per wrapping viatempfile()and folded into the cache key), so independent wrappings can never collide regardless of argument overlap. The documented single-wrapper behavior (F <- Memoize.function(f); F(5); F(6); F(5)-- secondF(5)hits cache) is unchanged; verified withidentical()before/after.🤖 Generated with Claude Code
https://claude.ai/code/session_01A5w4ZLUZpoHhtsbGyWcVtc