Repository navigation
Conversation
…mappings.h5 read
partitionNodesFromTetgenFiles wrote each of the element, face and edge
numberings to mappings.h5 collectively (MPI-IO) and then reopened the file on
every rank to read back old->new for its subdomain entities (point
selections), plus four `if (test)` blocks (test=false) that read whole
datasets. The reopen needed the node-leader-first, retried sec2 driver
(openMappingsReadOnly) because Lustre handed concurrent openers a stale size
at >=384 ranks on Carpenter.
Now subdomainNumbersByDirectory gives old->new for every subdomain entity:
the owner pushes its (old, new) pairs to the rank whose default chunk of
[0,nGlobal) holds the old number (exchangeInts, one Alltoallv), then each rank
asks for the ones it does not own (askOwningRanks, two). Memory per rank is
O(subdomain + nGlobal/size); no file is reopened, so openMappingsReadOnly,
setMappingsReadXfer, mappingsNodeComm and PROTEUS_MAPPINGS_READ_DRIVER go
(nothing else used them), with the dead test blocks and an IS that was
created outside a test block and destroyed only inside it (a leak per run).
The datasets are still written (decided in a later commit). The Triangle
path is unchanged here.
Exposed as cpartitioning.subdomainNumbers; meshPartition_exchange_mpi.py
checks it under random owners (scattered owned numbers, uneven and empty
ranges) and random ghosts, at 1-4 ranks (test_files_node_number_exchange).
Measured (macOS arm64, MPICH, -mat_partitioning_type current):
test_meshPartitionCharacterisation + EmptyMesh: 329 passed (base 329),
round 1/2 digests unchanged
A/B vs c09280c with real graph partitioners (c32r3_ab.sh, grouped by the
partition produced, matched on node coordinates): ParMETIS x4 +
PT-Scotch x3 at 2/3/4 ranks, tri + tet: 84 runs, 48 distinct partitions,
11 seen by both builds, 11/11 bit-identical subdomain arrays
partitioning.cpp: -575 / +75 lines
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The element/face/edge numberings of a -F subdomain (old->new, and new->old for elements) were std::maps: read from mappings.h5 until the previous commit, from the directory since. A std::map node costs ~48 bytes an entry; subdomainNumbersByDirectory now returns (old, new) sorted by old, and the partitioner keeps it as a SubdomainNumbering, a sorted vector (8 bytes an entry) with binary-search lookup. A key missing from it throws std::out_of_range naming the entity (an inconsistent set of mesh files) instead of a std::map's silent 0. The face and edge new->old maps, which nothing used, are gone. The subdomain key lists are vectors, not std::sets. Measured, TetGen path, 1.81 M faces / 0.90 M tets / 1.07 M edges cube (tetgen -pqfee), node partition 'current', macOS arm64, peak RSS per rank (/usr/bin/time -l, whole run, base = c09280c): 4 ranks: 2057-2286 MB -> 1795-2011 MB 8 ranks: 1319-1503 MB -> 1169-1354 MB (event times are not reported: the Mac was at load average 54-84 from other sessions; ceds-model timing is requested.) test_meshPartitionCharacterisation + EmptyMesh: 329 passed, digests unchanged A/B vs c09280c (ParMETIS x6 + PT-Scotch x4, 2/3/4 ranks): 120 runs, 54 partitions, 17 seen by both builds (8 tet, 9 tri), 17/17 bit-identical Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o ISAllGather partitionNodesFromTriangleFiles built each of the element, face and edge numberings by ISAllGather-ing the owned new->old lists onto every rank and inverting them into global old->new arrays: two O(global) arrays per entity on every rank (for 1.55 M triangles, 2.33 M edges: 2 x 6.2 M ints, ~50-75 MB a rank whatever the rank count), kept until the subdomain was built. It now uses the TetGen path's numbering: subdomainNumbersByDirectory (push (old, new) to the default-chunk rank, ask for the ghosts), kept as SubdomainNumbering tables of the subdomain's entities only, O(subdomain + nGlobal/size) a rank. The face loop records the subdomain's faces, as the TetGen loop does; the owned new->old valarrays that only fed the gathers go. The two -F paths now number elements, faces and edges the same way, which the next commits use to share steps 5-8. Measured (macOS arm64, MPICH, -mat_partitioning_type current): test_meshPartitionCharacterisation + EmptyMesh: 329 passed, digests unchanged A/B vs c09280c (ParMETIS x6 + PT-Scotch x4, 2/3/4 ranks): 120 runs, 55 partitions, 18 seen by both builds (11 tri, 7 tet), 18/18 bit-identical; an earlier run (x8 + x6) 168 runs, 20/20 (10 tri) 1.55 M-triangle square (triangle -pqea0.000001), peak RSS per rank, whole run: 4 ranks 1126-1311 MB -> 1156-1355 MB, 8 ranks 660-825 MB -> 662-830 MB, i.e. unchanged at these counts (the peak is set elsewhere); the gathers' share grows with the rank count, the directory's shrinks. A first version that kept std::maps cost +100-250 MB a rank at 4 ranks, hence the previous commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e it With the reads gone (b1e5ddd, 27afec1), the TetGen -F path still wrote seven datasets collectively: node new2old (an H5Sselect_elements point selection), and new2old/old2new for elements, faces and edges. Nothing reads them: git grep over proteus, and grep over torino_flume, the air-water-vv inventory, ADH-bayousorrel and the model examples here, find only SW2DCV and GN_SW2DCV, which read the node old2new. So the -F paths now write that one dataset, the same file on both (the Triangle path already wrote only it). SW2DCV/GN_SW2DCV took `list(f.keys())[0]`, the alphabetically first name: edgeNumbering_new2old for a TetGen file. They now read 'nodeNumbering_old2new' by name. Measured (macOS arm64, MPICH): 0.90 M-tet cube at 4 ranks: mappings.h5 31,400,072 -> 613,928 bytes (keys: 8 -> ['nodeNumbering_old2new']; Triangle file unchanged, 1 key) nodeNumbering_old2new is a permutation of the file nodes, tri and tet, 3 ranks, ParMETIS; same bytes as base for tri test_meshPartitionCharacterisation + EmptyMesh: 329 passed, digests unchanged A/B vs c09280c (ParMETIS x6 + PT-Scotch x4, 2/3/4 ranks): 120 runs, 55 partitions, 13 seen by both builds (4 tet, 9 tri), 13/13 bit-identical Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…xDim>) partitionNodesFromTetgenFiles and partitionNodesFromTriangleFiles carried two ~700-line copies of everything after the second element read: the face and edge reads, the element/face/edge numbering, the ghosts of the minimal overlap, the vertex read, the subdomain build and the -1 marking. With the numberings now the same on both (27afec1), they differed only by the dimension, the file kind, logging, and a few lines that released memory early. One template, buildSubdomainFromFiles<simplexDim>, now holds them, as collectSubdomainElements already holds the element read: - nodes per face simplexDim-1, faces per element simplexDim, edges per element simplexDim(simplexDim-1)/2; the face loop is one loop over a face's nodes (a node of no subdomain element keeps a distinct negative number, as the TetGen copy did); - the subdomain geometry builders by `if constexpr`; - both release the element-boundary/edge/element maps as soon as they are used (what the Triangle copy did); both log as the TetGen copy did; - the memory checks run for TetGen (memHardLimit) and stay off for Triangle (it had them commented out; max_rss_gb 0 skips them); - the TetGen path now restores and destroys its node numbering IS after the build (it was never destroyed); the Triangle path's MPI_Barrier after step 6 is gone (no data depends on it). The element maps hold std::vector node lists on both paths (TetGen used valarrays). partitioning.cpp: 5,889 -> 5,240 lines (-1,290 / +641). Measured (macOS arm64, MPICH): test_meshPartitionCharacterisation + EmptyMesh + multilevel test_partition_arrays + test_subdomain_view: 443 passed, 1 skipped (nested-mpiexec skip), digests unchanged; test_meshPartitionFromTetgenFiles under mpiexec -n 2: 2 passed 1.55 M triangles and 0.90 M tets at 4 ranks ('current'): subdomain arrays of all ranks bit-identical to c09280c (sha256 391cd95e45d47457 tri, d0208e5adcfc4e6a tet) A/B vs c09280c (ParMETIS x8 + PT-Scotch x5, 2/3/4 ranks): 156 runs, 58 partitions, 19 seen by both builds (9 tet, 10 tri), 19/19 bit-identical Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Dim>) partitionNodesFromTetgenFiles and partitionNodesFromTriangleFiles were still two ~320-line copies of steps 0-4 (node header read, default node chunks, node stars and the PETSc graph, MatPartitioning, node numbering by owner, the nodeNumbering_old2new write to mappings.h5, the element reread). They are now one template, partitionNodesFromFiles<simplexDim>, which ends by calling buildSubdomainFromFiles<simplexDim> (73ff16d); the public functions are three-line wrappers with unchanged signatures. simplexDim picks the format name, the face file (.face or .edge) and nSpace. partitioning.cpp 5,110 -> 4,696 lines (+100/-514). What the Triangle path loses with its copy (output only, no array changes): the rank-0 stdout line "Max nNodes_subdomain ...", now a logEvent as on TetGen; the attributes warning on every rank, now rank 0 as on TetGen; the single hard-coded 10 GB enforceMemoryLimit after MatCreateMPIAdj (the Triangle entry point has no limit argument and already passed 0.0, "no checks", to buildSubdomainFromFiles). Both paths lose the empty "Recv. ele mask" log event (begin and end with nothing between). The node IS is destroyed right after the element reread instead of at the end; the build does not use it. On a failed read both now return 1 (the TetGen copy returned 0 after a failed subdomain build; the binding ignores the value). Measured (macOS arm64, MPICH, FI_PROVIDER=tcp), 73ff16d against this: - -mat_partitioning_type current, test/ci meshes, 2/3/4 ranks x 3 reps: 36 runs, 6/6 partitions bit-identical (3 tri, 3 tet) - parmetis, same: 36 runs, 7/7 partitions seen by both builds identical (3 tri, 4 tet), no mismatch - 4 ranks, current, whole-run sha256 over every rank's arrays: 1.55 M triangles 961f56c4532e796e both; 0.90 M tets d0208e5adcfc4e6a both (the same tet digest as c09280c in 73ff16d) - test_meshPartitionCharacterisation + test_meshPartitionEmptyMesh + multilevel/test_partition_arrays + mesh_tests/test_subdomain_view: 443 passed, 1 skipped (nested mpiexec); test_meshPartitionFromTetgenFiles under mpiexec -n 2: 2 passed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cekees
marked this pull request as ready for review
November 6, 2026 11:53
Steps 6 (first layer of ghosts), 7 (further layers) and 7b (every node of
every owned or ghost element) were ~210 inline lines of partitionNodesLocal,
the first layer and each further layer repeating the same star walk (node
neighbours, elements, element boundaries, edges, with the same edge lookup
and offproc tests) twice, once into the final sets and once into per-layer
sets. They are now one function, nodeOverlapSets, returning
OverlapSets{nodes, elements, elementBoundaries, edges}: a lambda adds the
stars of one node to a set of OverlapSets; layer 1 applies it to the owned
nodes, each further layer to the nodes the last layer added (the node loop
that was split in two in the march is one loop; the sets are unions, so the
result is the same). partitionNodesLocal calls it and reads the four sets.
It reads the mesh and the numberings it is given and touches no MPI, PETSc
or files, as partitionNodesLocal does not.
partitioning.cpp 4,696 -> 4,646 lines. Behaviour unchanged:
- test_meshPartitionCharacterisation (incl. the 42 digests and
test_partitionNodesLocal_computes_every_rank_in_one_process) +
test_meshPartitionEmptyMesh + multilevel/test_partition_arrays +
mesh_tests/test_subdomain_view: 443 passed, 1 skipped (nested mpiexec);
test_meshPartitionFromTetgenFiles at 2 ranks: 2 passed
- layers beyond the tested 1 and 2: 73ff16d against this, node owners from
-mat_partitioning_type current (deterministic, contiguous chunks, so the
ghosts are partial), tri and tet, 2/3/4 ranks, 1-4 layers: 24/24 whole-run
sha256 identical. On the triangle mesh at 3 ranks rank 1's subdomain grows
86 -> 106 -> 127 -> 145 nodes over the four layers, so every layer branch
ran (the tet test mesh is whole after one layer)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
C32 round 3: the rest of the
parun -Fsec2-vs-mpio site, and one shared subdomain builder for the two -F partitioners. Round 2 was #201 (in c09280c).Every commit is checked three ways:
test_meshPartitionCharacterisation.py+test_meshPartitionEmptyMesh.py, 329 tests);-mat_partitioning_type currentthe new numbering is the file numbering. ParMETIS and PT-Scotch are nondeterministic on this Mac, so runs are grouped by the partition they produced and compared within a group;subdomainNumbersByDirectory: push (old,new) to the default-chunk rank,askOwningRanksfor ghosts). No mappings.h5 reopen, soopenMappingsReadOnlyandPROTEUS_MAPPINGS_READ_DRIVERgo, along with the deadif (test)blocks and a leaked ISstd::maps (~48 B)ISAllGatherof global numberings (O(global) per rank)nodeNumbering_old2newonly; SW2DCV/GN_SW2DCV read it by name (waslist(f.keys())[0])buildSubdomainFromFiles<simplexDim>replaces two ~700-line copiesNCLS3P.py76.2 -> 75.6 (the identical hunk the other round PRs carry, as 7c2a5d5 on c0-r6); this branch does not touch NCLS3PpartitionNodesFromFiles<simplexDim>(node header, default chunks, node graph, MatPartitioning, node numbering, the mappings.h5 write, the element reread), thenbuildSubdomainFromFiles; the two public functions are 3-line wrappers. Triangle loses its rank-0 stdout line, its 10 GB MPIAdj check and its every-rank attribute warning (output only)current36 runs, 6/6 partitions bit-identical; parmetis 36 runs, 7/7 shared identical; 4 ranks whole-run sha256 identical (1.55 M tri 961f56c4532e796e, 0.90 M tet d0208e5adcfc4e6a); 443 passed + TetGen-files test 2/2 at 2 ranks; partitioning.cpp 5,110 -> 4,696 linespartitionNodesLocal(in-memory path): steps 6-7b (ghost nodes/elements/faces/edges, further layers, 7b) arenodeOverlapSets; the star walk that was written twice is one lambdacurrent): 24/24 whole-run sha256 identical (tri rank 1 grows 86->106->127->145 nodes over the layers, so each layer branch ran); partitioning.cpp 4,696 -> 4,646CI on 73ff16d (run 42114715099), read from the logs:
ceds-model-2. ThePermissionError: './proteus_default.log'is not the failure: it is an "Exception ignored in atexit" warning in thecd /tmp; python -c 'import proteus'smoke step, which passed (a/tmp/proteus_default.logowned by another user on that host). The job failed on one test,test_vof_partition_independence.py::test_vof_taylor_galerkin_entropy_viscosity_on_two_ranks_is_the_serial_answer[VOF](2714 passed), a 2-rank launch that hung until the 1800 s timeout after Open MPI reported a shared-memoryopen(2)ENOENT and a TCP peer on the docker bridge 172.17.0.1. That test builds an in-memory structured mesh and never reaches the -F code this PR changes. Runner, not the branch; written up for the coordinator, no runner host touched.proteus/mprans/NCLS3P.pywas below its floor (5d12378).ceds-model timing (2026-11-06 04:23-04:37 CST, 192-core box, runners stopped,
max gh 0throughout; the load average of 100-190 inload.txtis the job's own ranks climbing to the rank count, it starts at 3.8).parun -FTetGen path on a 2.53 M-node / 15.63 M-tet unit cube (tetgen -pqfee), ParMETIS,-bind-to core,-log_viewstage "Mesh Partition", one run per cell. Buildold= 78e66bd (node numbers read back from mappings.h5),base= c09280c,r3= 73ff16d (the head of this PR when it ran; 4ce83ab's arrays are bit-identical, see below). Event time = max over ranks, seconds.Read: against c09280c the stage is 119.2 -> 96.3 s (-19%) at 192 ranks and 88.2 -> 83.5 s (-5%) at 96; the gains are in Global ele nmbr (3.5 -> 0.45 s), Subd faces, Subd edges and Reread eles at 192 ranks (not isolated per commit: the events do not separate the exchange, the sorted tables and the shared builder), and the peak RSS per rank roughly halves (1375 -> 700 MB at 192, 1495 -> 837 MB at 96; which allocation set the old maximum was not measured). Against 78e66bd the stage is 182.5 -> 96.3 s and 126.8 -> 83.5 s. Caveats: one run per cell, no repeats, so differences under ~10% (all of the 96-rank base-vs-r3 columns except Global ele nmbr/Subd edges) are not distinguished from noise; the RSS lines of 183-189 of 192 ranks parsed (interleaved stderr), min/mean are over those. mappings.h5 shrinks from 519 MB (
old/baseat 192 ranks) to 9.7 MB. Raw logs and the parser:claude_proteus_build/c32_r3_timing/(parse.py).🤖 Generated with Claude Code