Repository navigation
Conversation
…he scatters to zero
test_building_blocks_agree_with_petsc asserted that DistMat's mult and
multTranspose equal PETSc's exactly. They do on every conda build (all CI legs),
but the fleet nightly's pip-built PETSc on aarch64 (spark2, GCC -g -O) differs
by one or two ulps, and the four rank counts failed there every night:
spark2 dev_pip, 1/2/3/5 ranks: mult = multTranspose = 8.9e-16, 8.9e-16, 4.4e-16, 8.9e-16
GCC contracts a*b+c into a fused multiply-add by default in GNU C mode on
aarch64; numpy's sum does not, so the last bit can differ. Neither is wrong.
distmat_mpi.py now prints, for each product, k*eps*max_i sum_j |a_ij||x_j| (k
the most terms in one sum): two evaluations of a k-term sum each lie within
(k/2)*eps*sum|a||x| of the exact one, whatever the order or contraction. The
test holds mult and multTranspose to that bound and keeps the vector scatters
(forward insert, its truth, reverse add) at exactly zero: they only copy and
add exactly representable values.
bound, 1/2/3/5 ranks: mult 5.1e-15, 6.3e-15, 6.3e-15, 8.6e-15;
multTranspose 2.2e-14, 2.3e-14, 3.5e-14, 2.0e-14
spark2 dev_pip (this helper run with its proteus): differences as above, all
under the bound; chewbacca (conda, py3.13): differences 0, 4 passed
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nstead of failing 23 tests The fleet nightly's dev_pip prefix (pip-built PETSc, no MUMPS) failed 23 tests every night that its dev_pip_mumps prefix passes (2026-11-05, main 8f948a1, ceds-model: dev_pip 32 failed, dev_pip_mumps 10, the difference exactly these 23). Each names MUMPS or needs the direct solve it stands for: explicit "linear_solver mumps, but this PETSc was built without MUMPS" or "Could not locate solver type mumps": test_adr_problem (pinned Taylor-Hood, periodic velocity-pressure, the superlu comparison's mumps case, the periodic solver comparison, the 2- and 3-rank periodic and advected-periodic runs, the th_mumps refusal), the 4 parallel DG velocity post-processing cases, the RANS3PF and CLSVOF rank-cut runs; the no-MUMPS fallback (linear_solver petsc, GMRES+ILU(0)): the pressure null space (adr_problem, pro CLI on two ranks: ILU fails on the empty pressure diagonal), pure advection across a rank cut (5 weak-flux classes: L2 inf); test_the_linear_solver_choice_follows_the_spec: an omitted solver on a singular operator picks MUMPS only when PETSc has it. @pytest.mark.mumps(reason) marks them; conftest probes PETSc once (in a subprocess, so petsc4py does not start PETSc in the test process) when a marked test is about to run, and skips it with "this PETSc was built without MUMPS, and <reason>". Under GitHub Actions nothing is skipped: every CI leg's PETSc is conda-forge's, which has MUMPS, and a CI environment that lost it must fail rather than go green with 23 skips. The fallback failures are a known gap, not a test defect: proteus has no scalable no-MUMPS solver for singular and saddle-point systems yet (cekees, 2026-10-29: the nodes get MUMPS for the nightly, and the no-MUMPS option is still wanted). The marker's reason says so in the skip summary. checked here (chewbacca, PETSc with MUMPS): 23 collected by -m mumps; with the probe forced to "no": 23 skipped with their reasons; with GITHUB_ACTIONS=true and the probe forced to "no": they run (choice test: 1 passed); with the real probe: they run (2 passed of the 2 tried) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… cannot nest test_independent_partitions_do_not_have_this_property[node-hex] failed on #198's linux-64/py3.12 leg (run 42024054393, ceds-model-4): missing = 0, i.e. four independently partitioned levels of the 4x4x4 hex mesh nested. The control asked the graph partitioner for each level's cut and asserted it would not nest, so its outcome belonged to the partitioner. Measured, graph-partitioned independent levels, 4 ranks, node partitioning: ceds-model, CI's env (proteus-ci-main, Open MPI 5.0.11), main c09280c, hex 200 runs: missing 1427-6207 (median 3731), offproc 0.35-0.96, never 0 chewbacca (conda MPICH), 30 runs each: hex missing 1585-6269, quad 59-640, tri 96-578, tet 1074-3910; offproc 0.24-0.96 (quad's 0.2368 is the old 0.25-floor failure) PETSC_OPTIONS=-mat_partitioning_type current: hex missing 0 offproc 0.200, quad missing 0 offproc 0.151, tri 26, tet 156 So the CI zero is not the tail of the random draws; the contiguous initial partition reproduces it exactly, but node-quad passed just before node-hex on the same xdist worker, so a leaked PETSC_OPTIONS does not explain it either. Not root-caused; the control no longer depends on it. pinned_partitions.py partitions every level independently by slabs across one coordinate axis that turns with the level (x, y, x / x, y, z), near-equal contiguous runs of nodes (or element barycenters) in that coordinate, ties by index, no empty rank. Consecutive levels are cut across different axes, so a fine node at the far end of its slab has its parents in another rank's slab, beyond one layer of overlap. level_consistent_mpi.py takes it as a fifth argument (`slabs`; `graph` stays the default) and space_transfer_mpi.py as PROTEUS_TEST_INDEPENDENT_PARTITIONS=slabs, so the refusal control (test_operators_without_level_consistency_are_refused_not_wrong, p2 and ncp1) is pinned too. The offproc floor of the control rises from 0.15 to 0.5. pinned, 4 ranks: missing tri 280 / 372 (node / element), tet 2005 / 6057, quad 330 / 168, hex 3262 / 860; node offproc 0.72, 0.70, 0.64, 0.67 refusal: p2 and ncp1 refuse ("not level-consistent") on 3 of 4 ranks test_level_consistent_partitioning.py: 26 passed; the 10 controls 50 times in a loop (-n 4): 50/50 passed; 26 passed with PETSC_OPTIONS=-mat_partitioning_type current in the environment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ule is collected Round 4 found the suite order-dependent: the 39 test files that import reload pass in one process forward and fail reversed. Bisected here by pairs (each earlier file, then the failing file, in one pytest process; 159 pairs on ceds-model, CI's env): six files make the others fail, and all six import a deck at module level that reloads proteus.default_p/default_n and sets attributes on them (matrix_constructor/test_mass_matrix_quad.py and test_mass_matrix_reference_simplex.py; post_processing/test_bdm_projection.py, test_bdm2_mesh8.py, test_bdm_3Dprojection.py, test_elementwise_flux.py: the BDM templates set n.triangleOptions = "Yp", p.nd, p.name, ...). Importing a test module is collecting it, which happens before any test runs and outside _restore_proteus_process_state, whose snapshot is taken after collection. So whatever a module set while it was imported stayed Proteus's defaults, and every deck collected after it star-imported them: the order of the files on the command line decided which. A hookwrapper on pytest_make_collect_report now snapshots the three modules before each pytest.Module is collected and restores them after (dicts and lists copied, so in-place edits are undone too). the 39 files in one process (ceds-model, proteus-ci-main py3.12, Open MPI 5), main c09280c -> this commit: forward 166 passed -> 166 passed reversed 34 failed, 132 passed -> 166 passed random seed 1 32 failed -> 166 passed random seed 2 13 failed -> 166 passed random seed 3 14 failed -> 166 passed random seed 4 15 failed -> 166 passed (the 34: test_gauges' first row 0.5625 for 0.5, 13 poisson_2d cases, vortex2D x2, vortex3D, TADR x6 (IndexError), VOF x6, NCLS x4, CLSVOF+RANS2P) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1383a58 restores default_p/default_n/default_so after each test module is collected. proteus.defaults takes its lists of option names from dir(default_p) and dir(default_n) when it is first imported; when a polluting deck imported it first (in the middle of setting p.rdomain, ...), those names became "options", and after the restore Physics_base() raised KeyError: 'rdomain'. Found here (chewbacca): test_bdm_projection.py then test_poisson2d.py, collection error. On ceds-model's order runs test_gauges.py, collected first in the reversed order, had imported proteus.defaults cleanly, so 1383a58's numbers did not see it. The snapshot now imports proteus.defaults first. That imports petsc4py, which the first proteus test module collected does anyway; Comm.init hands Comm.argv to the options database when PETSc has already started. the 39 reload files in one process (ceds-model, proteus-ci-main): reversed, random seeds 1-8: 166 passed each (base c09280c: 13-34 failed in reversed and seeds 1-4) each of the six polluting files collected first, the rest in list order: 166 passed each (1383a58 alone: collection error in that order) chewbacca: test_bdm_projection.py test_poisson2d.py test_tadr.py: 28 passed (1383a58: 1 error) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tements to Transport) The floor was recorded on main ce43b73 (33f30b1): 522 of 685 statements, 76.2%. #199 (C0 round 5, 76f3e12) replaced NCLS3P's own scatterDOFArray (15 statements, all covered) by the one on OneLevelTransport, so main c09280c has 507 of 670, 75.67%: the same lines untested, a smaller file. Every pull request on c09280c without #204 (whose staticmethod lines add covered statements to the file) is below the floor: this PR's coverage job (run 42113031204, NCLS3P the only file under its floor) and #212's both report NCLS3P 75.7%. The new floor is the measured value rounded down to a tenth. Same hunk as 7c2a5d5 on c0-r6, so the train merges both cleanly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cekees
marked this pull request as ready for review
November 6, 2026 10:11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
C3 round 6 (stream E3): test determinism and the fleet nightly's environment gaps. One commit per cause; numbers in each message.
test_building_blocks_agree_with_petsc: DistMat's mult/multTranspose held to a printed rounding bound (k·eps·max Σ|a||x|), the scatters still exactly 0-g -O): 4.4e-16-8.9e-16 under bounds 5.1e-15-3.5e-14 (was 4 failures every night); conda builds: 0@pytest.mark.mumps(reason): skipped with the reason when PETSc lacks MUMPS, never under GitHub Actionsdev_pipfails anddev_pip_mumpspasses (11-05: 32 vs 10 failed on ceds-model); forced "no MUMPS": 23 skipped with reasons; underGITHUB_ACTIONS=true: they runtest_independent_partitions_do_not_have_this_property, the refusal test) pinned to slabs whose axis turns with the level-mat_partitioning_type current: exactly 0. Pinned: missing 168-6057, offproc 0.64-0.72; 50/50 loops of the 10 controls passdefault_p/default_n/default_soafter each test module is collected (six decks reload and set them at import, before the per-test restore's snapshot)proteus.defaultsfirst (it takes its option names fromdir(default_p)at first import)KeyError: 'rdomain'when a polluting file is collected first; now each of the six first, reversed, seeds 1-8: 166 passedfloors.json: NCLS3P.py 76.2 → 75.6, the same hunk as 7c2a5d5 onc0-r6Whole suite in one process (pytest-ways' serial way), Mac, 33ad04f, file order: 1 failed, 2725 passed; the failure is the env's stale
parun(no--hotStartMode), passes with the tree's. ceds-model, branch conftest, file order and two shuffled file lists: only stale-ymf failures, plus one artifact of pytest 9.1.1 that a shuffled file list triggers (a package revisited after a file in its parent directory loses its conftest's autouse fixtures) and that CI's one-directory invocation cannot. No order dependence left beyond the reload files.Not changed, for review: the coverage job combines only the two linux-64 py3.12 legs (main + FSI suites), not macOS.
needs:cannot name single matrix legs, so starting it early means moving those legs into a job of their own, which rewrites the matrix hunks the dev overlay patches. Written up in the handoff instead.🤖 Generated with Claude Code