Repository navigation
C7.1 Phase D: per-rank subdomain views as the same-rank restart source - #213
Merged
Merged
Conversation
…ain with its own numbering
A per-rank view must give the solver back the subdomain partitionMesh built: the partition's maps
number its element boundaries and edges, so those numbers must come back too. generateFromArrays
numbers boundaries and edges its own way; generateFromNumberedArrays takes each element's boundary
numbers and the edge list and calls what partitioning.cpp calls on a subdomain:
constructElementBoundaryElementsArrayWithGivenElementBoundaryAndEdgeNumbers_{triangle,tetrahedron},
then allocate/computeGeometricInfo, then the element boundary flags. mesh.cpp and partitioning.cpp
are not edited. The arrays are allocated with new[] (the Mesh frees them with delete[]); the node and
element fill is shared with generateFromArrays (_fillNodesAndElements).
Measured (macOS arm64, MPICH, env proteus-bc-261025): every rank's subdomain of square4x4_tagged,
square4x4 and cube_3x3x3, owners scattered and reversed, on 2 and 3 ranks (30 rank subdomains):
- generateFromArrays from the same nodes and elements: 7 arrays differ in every case
(elementBoundariesArray, elementBoundaryNodesArray, elementBoundaryElementsArray,
elementBoundaryLocalElementBoundariesArray, interior/exteriorElementBoundariesArray, edgeNodesArray);
- generateFromNumberedArrays: all 33 arrays equal (nodeDiametersArray aside, which partitionMesh
finishes across the ranks afterwards).
test_numbered_arrays (subdomains from partitionNodesLocal in process): 7 passed, serial and 2 ranks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tart that reads them Decided by cekees 2026-11-05 (decision 3, after C32 round 2): per-rank views written into per-rank archives (archive-modes layouts B and C) as the same-rank restart source. - Write: on a per-rank archive's first mesh write (Mesh.writeMeshXdmf, driver per-rank), each rank writes subdomain_view/<name> into its own <archive>_p<rank>.h5: the step-H ymf grid (its arrays as datasets beside it, the grid in the `ymf` attribute), and what rebuilding the solver's subdomain needs: each element's boundary numbers, the edges, every element boundary's flag (-1 included), the partition's eight maps, the global mesh's counts and scalars, its fingerprint and whether the partition renumbered. - Read: subdomainViewMesh(archive) -- an open archive or its path -- builds on every rank the mesh that stands for the global one (counts, scalars, maps, no arrays; as cartesianSubdomain's) with the subdomain rebuilt by cmeshTools.generateFromNumberedArrays, then _finishPartitioning. Collective: every rank reads its view or every rank raises SubdomainViewUnavailable with the first reason (no per-rank file, no view, another number of ranks, another view format). - generateMesh(..., restartArchive=None): with an archive whose views fit, a one-level hierarchy from them (multilevelMeshFromSubdomainViews), no global mesh and no partitioner; otherwise, or with more than one level, the old path, logged with the reason. NumericalSolution does not pass it yet (C17/C14's file: the hand-off line is agreed separately); the tests' driver passes the run's open archive. - The input maps and fingerprint carry over a views-built mesh (_localInputNumbering, _inputFingerprint), so the restart's archive records them as the global path's does. Measured (macOS arm64, MPICH, env proteus-bc-261025, ymf c2e646f, PETSC_OPTIONS -mat_partitioning_type current): - write then read, square4x4_tagged and cube_3x3x3, 2 and 3 ranks, layouts B and C: on every rank all 33 subdomain arrays, the 8 maps, the 17 global scalars and the -1 marking equal partitionMesh's (cut faces per rank 6/0, 8/0/8, ...). - same-rank restart, vortex2D (ls + rdls + vof + mcorr, 4 models), stopped at step 3, hot started to step 7 from the same archive with the mesh from the views and with the mesh generated and partitioned: on 2 ranks, partitioned (regularGrid global) and per-rank generated meshes, layouts B and C, every dataset of both ranks' files equal: 138/138 (fields, lagged solver state, mesh, maps). On 3 ranks the run is not reproducible against itself on either path: two global restarts differ by up to 1.7e-12 (18 datasets per rank), two view restarts by up to 2.1e-12 once in four, and views vs global by at most 1.0e-12, inside that spread. Not this change's (both paths); for C3/C17. - unpinned (the graph partitioner), the global restart's partition differed from the run it restarted (rank 0: 420 elements against 416), so its per-rank lagged state was another partition's; the views restart keeps the run's partition by construction. - test_subdomain_view_restart 8 passed (35 s); test_archive_*, test_archiver_write_field, test_hotstart_*, test_gauges*, mesh_tests, ci, poisson_2d, test_adr_problem, test_domain_ymf, multilevel, rank-interface, periodic: 1629 passed, 3 xfailed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eed with C17/C14) The hand-off point agreed with the C17/C14 session (2026-11-05): NumericalSolution's mesh setup passes restartArchive=self.ar[0] -- the run's open archive, not its path: a hot start opens each rank's file for append, and HDF5 refuses a second, read-only open in the same process -- unless opts.hotStartMesh is 'global' (an options attribute for now; a parun flag waits for C15). Nothing else in NumericalSolution changes. The vortex2D driver now sets opts.hotStartMesh instead of wrapping generateMesh, and the bit-identity test fails if either hot start logs "no solver state to restore" (a mesh numbered otherwise than the archive's makes a hot start a silent restart; C17/C14's request). Measured: test_subdomain_view_restart + test_hotstart_parallel_state + test_hotstart_state: 16 passed. That log line, counted in the earlier runs: unpinned global restart 4 (one per model: the graph partitioner gave another partition, so every model restarted without its lagged state); views restart 0; pinned global restart 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cekees
added a commit
that referenced
this pull request
Nov 5, 2026
C7.1 Phase D (#213) found that a hot start through the global mesh path, with the graph partitioner unpinned, gets another partition than the run it restarts: every model's solver state is refused (its fingerprint differs) and the run quietly becomes a restart, said only per model at log level 0. The recorded decision (2026-10-29) is that a refused restore continues as a restart with a message, so this keeps the behaviour and makes the message loud: one RuntimeWarning on the first rank (and one WARNING log line) naming the models and the reason, and saying how to continue exactly (the archived partition: same ranks, the per-rank mesh views or a pinned partitioner) or to ask for a restart (--hotStartMode restart). The reason now tells a different partition (same ranks and mesh, other pieces) apart from another rank count or no state at all; the existing texts are unchanged. Whether a refused continue should instead be an error is left to cekees. test_a_continue_on_another_partition_warns_once (harness OTHERPARTITION=1 changes only the partition hashes of the fingerprint): before, no warning; after, exactly one, naming rans2p..mcorr and "different partition" (the run differs from the uninterrupted one by 5.2e-1, as a restart does). test_hotstart_*, test_state_contract, test_step_retry: 48 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cekees
marked this pull request as ready for review
November 5, 2026 22:08
This was referenced Nov 6, 2026
cekees
added a commit
that referenced
this pull request
Nov 6, 2026
… on every rank
archiveSolverState compares the ranks' lists of dataset names before its collective creates and writes
nothing for a model whose lists differ. The lists follow the state() dicts' insertion order, and RANS2P's
numericalFlux.ebqe lists ('u', 0)..('u', 3) in an order set by each process's string hash seed, so on two
ranks the coupled-fields owner's names came out in different orders about one run in two: "the ranks have
different solver state to write ... model rans2p" (level-0 log line only), no solver_state/rans2p in the
archive, and the next same-rank continue fell back to a restart with the RuntimeWarning. The earlier
2-rank "0.0" for damBreakLevelSet was a run that happened to hash alike.
Measured (damBreakLevelSet, 2 ranks, -mat_partitioning_type current, archive every 0.02 to 0.04):
archives missing solver_state/rans2p, random seeds pre-merge c0bed9e 3/6, after the merge of
cekees/main 5/6 (not #213's: same before it)
per-rank seeds equal (1/1, 2/2, 7/7, 5/5) never; different seeds 5/6: always
continue at 0.04 to 0.1, seeds 5 and 6, vs the uninterrupted run (same seeds)
before 3.7e-3 on both ranks, one RuntimeWarning (rans2p)
after 0.0 on both ranks, same accepted times, no warning
The fingerprint stays first and the step stamp last (written last, as the docstring requires).
The order inside an owner's state does not matter on read (by name).
test_a_two_rank_continue_is_exact_whatever_each_rank_hashes: an MPMD launch with PYTHONHASHSEED 5 and 6;
fails without this change, passes with it. With cekees/main afcedf6 merged (#204, #213, #215), -n 4:
test_step_retry, test_hotstart_* (test_hotstart_levelset 13), test_state_contract, test_mpi_launch_retry,
test/TwoPhaseFlow, test/HotStart_3P, test/surface_tension/rising_bubble_rans3p,
test/mesh_tests/test_subdomain_view_restart (skipped: this env's ymf has no ymf.mesh),
test_cython_class_attributes: 114 passed, 1 skipped. The one-warning tests (perturbed partition
fingerprint; 2 ranks archived, 3 continued) pass with #213's per-rank-view restart source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cekees
added a commit
that referenced
this pull request
Nov 6, 2026
…al path's dtype Phase D (#213) tested a solver restart from per-rank views only on vortex2D (triangles, a box whose partition kept the input order, so no CellMapInput was written). This adds the tetrahedra case: restart_views_tet.py, a VOF blob (mprans VOF, backward Euler, fixed step, P1, direct solve) on importer_meshes/gmsh_cube.msh's 1125 tetrahedra through the use_gmsh PLC path, graph-partitioned (pinned with -mat_partitioning_type current), 2 ranks, stopped at step 3 and hot started to step 7 from the views and from the global path, layouts B and C. It found one difference: the restart's archive declared CellMapInput as int64 on the views path (the view stores the maps as int64) and int32 on the global path (the partition's old2new numbering), so the Mesh_Spatial_Domain_4..7 ymf documents differed while every array was equal. The views reader now gives the maps the dtype of the subdomain's own numbering maps, which is the global path's. Measured (osx-arm64, MPICH, 2 ranks, 341 nodes / 1125 tets, rank 0 594 elements, rank 1 1075, both with overlap): before after datasets per rank, views vs global 99, 4 differ 99, 0 differ (Mesh_Spatial_Domain_4..7: CellMapInput precision 8 vs 4) "no solver state to restore" 0 / 0 0 / 0 test_subdomain_view_restart.py 2 failed, 8 passed 10 passed, 13 s Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
C7.1 Phase D (decided by cekees 2026-11-05, after C32 round 2): per-rank subdomain views written into per-rank archives (archive-modes layouts B and C), read back by a hot start with the same number of ranks so that the global mesh and the partitioner are skipped. Branch
c7-phase-doffcekees/main@ c09280c.partitioning.cppandmesh.cppare not edited.cmeshTools.generateFromNumberedArrays: a subdomain rebuilt with its own element-boundary and edge numbering, through the…WithGivenElementBoundaryAndEdgeNumbers_*constructors partitioning.cpp usesgenerateFromArrays): 7 of 33 arrays differ in all 30 rank subdomains tried (2 and 3 ranks, tri/tet)test_numbered_arrays7 passed, serial and 2 rankssubdomainViewMesh;generateMesh(restartArchive=)NumericalSolutionpasses its open archive togenerateMeshon a hot start (opts.hotStartMesh='global'opts out), agreed with C17/C14Found on the way, for C3/C17 (not this change: both paths): vortex2D on 3 ranks is not bitwise reproducible from run to run (≤ 2.1e-12). With the graph partitioner unpinned, a global-path restart gets another partition than the run it restarts and silently drops every model's lagged state; the views path cannot.
🤖 Generated with Claude Code