Repository navigation
Conversation
… or ghosted vectors per step Main f392a15 aborted the Torino flume run (RANS2P, VOF, NCLS, RDLS, MCorr; 3.86M nodes, 1,920 Cray MPICH ranks) after exactly 12,678 steps, twice (13,213 on 2,304 ranks), with MPICH's `req != NULL` assertion in VecGhostUpdateBegin -> PetscSFBcastWithMemTypeBegin; a hot start ran clean for 10,010 steps. The cause is the per-model scatterDOFArray of proteustoolkit#1321 (284a789, VOF; NCLS and MCorr in the same PR and proteustoolkit#1324): it cached a ParVec_petsc4py per id(array) over the array itself, so every temporary it was handed stayed alive with its PetscSF and that SF's persistent send and receive per neighbour rank. VOF.FCTStep hands it three per step (Rpos, Rneg, limited_solution). 76f3e12 (C0 round 5, in train proteustoolkit#1341) caches one vector per array shape, so current main does not leak; nothing pinned that. step_resource_growth_mpi.py runs test/TwoPhaseFlow/damBreakLevelSet.py (stored mesh, archive every 0.05 s) to t=0.5 and records after every step, on each rank, live MPICH request and communicator handles (read from MPICH's handle pools through ctypes, the reading checked by allocating and freeing 16 handles; skipped on any other MPI) and live ParVec_petsc4py objects (any MPI). The test asserts none grows after a 5-step warm-up; the control (LEAKY=1) puts f392a15's scatterDOFArray back and must grow by at least 3 vectors and 6 requests per step. Measured, macOS arm64, MPICH 5.0.1, 2 ranks, rank 0, step 5 -> step 30 (25 steps): build requests ghosted vectors main f392a15 (real build) 60 -> 210 47 -> 122 (6.0 and 3.0 per step) main c09280c + f392a15 scatter (LEAKY) 62 -> 212 48 -> 123 main c09280c (this branch) 32 -> 32 33 -> 33 With the deck at he=0.05 to t=0.3 (49 steps), steps 10 -> 49: f392a15 84 -> 318, c09280c 32 -> 32, 1.9.x b6c60a5 28 -> 28 (1.9.x has no scatterDOFArray). The leak shows on shared memory and with MPIR_CVAR_NOLOCAL=1 alike, because persistent requests always take a handle. Both tests take 4 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tements to Transport) The floor was recorded on main ce43b73 (33f30b1): 522 of 685 statements, 76.2%. #199 (C0 round 5, 76f3e12) replaced NCLS3P's own scatterDOFArray (15 statements, all covered) by the one on OneLevelTransport, so main c09280c has 507 of 670, 75.67%: the same lines untested, a smaller file. Every pull request on c09280c without #204 (whose staticmethod lines add covered statements to the file) is below the floor: this one's coverage job and #212's both report NCLS3P 75.7%. Line by line, the NCLS3P lines hit in this branch's run and in #204's run (mapped through #204's edits) are the same set. The new floor is the measured value rounded down to a tenth. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 7c2a5d5)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Torino flume run aborted on main f392a15 (RANS2P TwoPhaseFlow, 3.86M nodes, 1,920 Cray MPICH ranks) with MPICH's
req != NULLassertion inVecGhostUpdateBegin -> PetscSFBcastWithMemTypeBegin. It died at exactly 12,678 steps in both attempts, and at 13,213 steps on 2,304 ranks. A hot start ran clean for 10,010 steps.Cause (already fixed on main): the per-model
scatterDOFArrayfrom proteustoolkit#1321 (284a789 VOF, b41f7a2 NCLS; MCorr in proteustoolkit#1324) cached aParVec_petsc4pykeyed byid(array)and built over that array. So every temporary it was handed stayed alive, together with its PetscSF and that SF's persistent send and receive per neighbour rank.VOF.FCTStephands it three temporaries per step (Rpos,Rneg,limited_solution). 76f3e12 (C0 round 5, #199, in train proteustoolkit#1341) caches one vector per array shape instead. That fix is in main c09280c, but no test pinned it.This PR adds only the test. It runs
damBreakLevelSet(the Torino model set) on 2 ranks with the archive on, and records three counts on each rank after every step:ParVec_petsc4pyobjects (any MPI).It asserts that none of them grows after a 5-step warm-up. A control (
LEAKY=1, f392a15'sscatterDOFArrayput back) must grow. Both tests take 4 s.Measured on macOS arm64 with MPICH 5.0.1, 2 ranks, rank 0, step 5 → step 30:
scatterDOFArrayon 1.9.x)Tests run:
test/test_step_resource_growth.py(2 passed) on this Mac; the unmodified helper also fails as expected against a real f392a15 build.Commit 9d52853 (cherry-pick of c0-r6 7c2a5d5, byte-identical
floors.json): NCLS3P.py floor 76.2 -> 75.6. 76f3e12 moved 15 covered statements out of the file, so every branch off c09280c measures 75.7%. The main train proteustoolkit#1343 carries the same hunk via #213.🤖 Generated with Claude Code