Repository navigation
Reduce memory footprints for 1D compression, improve domain decomposition, and add data compatibility mode - #262
Merged
Conversation
MGARD_ENABLE_MDR used find_package(MPI REQUIRED), so a machine without MPI could not build MDR-X at all, even though only the parallel pmdr-x-qoi and pmdr-x-qoi-io executables use MPI. Look MPI up without REQUIRED and build those two executables only when it is found; the MDR-X library, mdr-x and mdr-x-qoi build either way. The configure summary reports the result as "pmdr-x (MPI): ON/OFF" under MDR. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
_level_volumes and _level_volumes_reciprocal were dense (l_target + 1) x D x max_width arrays, but level l only fills level_shape[l][d] entries per dimension. In 1D max_width is N and l_target is ~log2(N), so the two arrays alone took ~50x the input size (1600 MB each for a 2^24-element float array) and dominated the 1D compression footprint (peak 120x the input). Pack each (level, dim) segment back to back and add a (l_target + 1) x D _level_volume_offsets table (Hierarchy::level_volume_offsets()); the total is sum_l sum_d level_shape[l][d], ~2N in 1D. The levelwise quantizer kernel reads level_volumes[offset(level, d) + idx[d]], which is the same element as before: idx[d] < level_shape[level][d] for a node of that level. EstimateMemoryFootprint uses the packed size. 1D float, SERIAL, peak device memory: 2^24 elements 7680 -> 1792 MB (multi-dim) and 7680 -> 2121 MB (hybrid). Compressed output is byte-identical to before across 1D-4D, float/double, s = inf/0/1, rel/abs, multi-dim and hybrid. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CompressorCache and RefactorCache inserted with (*hierarchy_cache)[key] = hierarchy, a copy assignment that deep-copies every device array (Hierarchy declared a copy constructor, so it had no implicit move). For a moment two full hierarchies were live: 1792 MB instead of 896 MB for a 2^24-element 1D float input. Give Hierarchy defaulted move constructor/assignment (Array's move hands over the buffers and leaves the source empty; moving into a populated Array frees it first) and an explicitly defaulted copy assignment, which declaring the moves would otherwise delete. Both caches now insert with std::move(hierarchy); shape is read before the move. mgard-x output is byte-identical across 1D-4D, float/double, s = inf/0/1, rel/abs, multi-dim and hybrid; mdr-x refactored components and reconstruction errors are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ition The hybrid compressor used a full Hierarchy of the input grid, of which the block-local stage reads only the shape, plus a second Hierarchy for the global stage that HybridHierarchyDataRefactor::Adapt rebuilt on every call (once per subdomain). In 1D the input-sized hierarchy alone was ~14x the input. HybridHierarchy derives from Hierarchy. Its base part is shape-only (new protected Hierarchy::init_shape_only: one level equal to the input shape, no device arrays), so l_target()/level_shape()/total_num_elems() keep working for the pipelines and the cache. It stores the input shape, the local/global level counts, the padded fine/coarse block grid of each local level (computed once by compute_local_shapes, replacing four copies of the 8 -> 5 rule), and owns the global Hierarchy, built once, only when M > 0, on the coarsest local grid (the input grid when L == 0). - HybridHierarchyCompressor::HierarchyType is HybridHierarchy; the compressor, hybrid/block-local refactors and both quantizers take a HybridHierarchy& and read grids and the global hierarchy from it. check_config() rejects a config whose L/M differ from the hierarchy's. - CompressorCache::InHierarchyCache takes the Config and drops entries for which matches_config() fails (always true for a plain Hierarchy). - DomainDecomposer builds OperatorType::HierarchyType and sizes it with the new static HierarchyType::EstimateMemoryFootprint(shape, config). - Hybrid tests construct a HybridHierarchy. Hybrid peak device memory, 1D float 2^24 elements (SERIAL): L=1 2121 -> 1225 MB, L=1/M=1 2913 -> 1705 MB, L=2/M=2 2643 -> 1563 MB, L=0/M=2 3465 -> 1897 MB; with L=1 the hierarchy allocates no device memory. SERIAL output is byte-identical across 1D-4D, float/double, s = inf/0, multi-dim and hybrid L1/L2/L1G1/L2G2/L0G2. SERIAL and CUDA (RTX PRO 6000 Blackwell) unit tests pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
max_larget_level capped the levels of the multi-dimensional Hierarchy and num_global_refactoring_level set the levels of the hybrid global stage, which is itself a multi-dimensional Hierarchy capped the same way. Merge them into num_global_refactoring_level: it now caps every multilevel hierarchy (multi-dim, MDR-X, hybrid global stage). Both level counts are now SIZE; defaults are num_local_refactoring_level = 1 and num_global_refactoring_level = 3 (previously: all levels for multi-dim, 0 global levels for hybrid). Non-hybrid headers did not record the level count -- the reader rebuilt the hierarchy from the shape -- so a non-default cap could not be decompressed. Record it for every compressed and MDR-X stream in a new optional FunctionDecomposition.num_global_refactoring_levels. On read it sets the Config; streams without it (everything written before) were decomposed with all levels and are read that way. Hybrid streams still take L/M from their HybridHierarchy message. The mgard-x CLI -ll/-gl now default to the Config defaults instead of hard-coded 1/0, and -gl also applies to multi-dim compression. Behavior change: default compression now uses at most 3 global levels. Under rel 1e-3, multi-dim L-inf CR rises (1D 2.89 -> 3.44, 3D 2.57 -> 2.85); multi-dim s = 0 in 1D drops (4.08 -> 3.65); all bounds hold. Hybrid with an s-norm bound now warns that it falls back from 3 global levels to 0 by default. Files from the previous build (multi-dim and hybrid, 1D-4D, float/double, SERIAL and CUDA) decompress byte-identically, and new compression with matching levels reproduces the old output; old MDR-X streams reconstruct identically. SERIAL and CUDA unit tests pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Serializer: Serialize aligns each value to sizeof(T) but Deserialize did not skip the padding. A subdomain stored uncompressed whose byte count is not a multiple of 8 (e.g. a 9x3x5x25 float block) shifted every later subdomain size field, and decompression failed or read garbage. Deserialize now aligns the same way. - MDR reconstruction sized level_signs to the raw per-level element count, while the bitplane decoder writes a sign for every slot of its last 32-value batch. The refactoring Resize already rounded up; the reconstruction Resize now does too (it takes the reconstructor for its BATCH_SIZE). The overrun corrupted the heap on SERIAL (free(): invalid pointer) and wrote out of bounds on CUDA whenever a later subdomain grew a level's buffer. - MDR reconstruction copied the last subdomain from abs((current_buffer - 1) % 3), which is 1 instead of 2 when current_buffer is 0: with a multiple of 3 subdomains the last one was written back from the wrong buffer and the error bound failed. - MDR refactor/reconstruct sized their reused buffers from subdomain 0; size them from the largest subdomain, as the compression pipelines do. - Zstd: read the stored size as size_t (it was truncated to 32 bits) and check ZSTD_decompress, so a corrupt stream fails instead of handing garbage to the Huffman decoder. - Compression output buffer: reserve room for non-uniform coordinates, which the header stores as doubles, and fail with OutputTooLargeFailure instead of overflowing when the header does not fit. - Metadata: delete[] for the coordinate copies allocated with new[]. - mgard-x CLI: check the status of decompress() in both paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… pipeline Data written by MGARD 1.6.x differs from the current format in many places (hierarchy geometry and level volumes, Huffman layout, basis, defaults such as an unlimited level cap). Rather than branch on the version throughout the pipeline, add a complete copy of the 1.6.0 decompression path in its own tree and namespace (include/mgard-x-160, src/mgard-x-160, namespace mgard_x_160), kept faithful to 1.6.0. - Compression always writes the current format; only decompression is routed. DynamicAPI's decompress overloads and release_cache first read just the header versions (new MetadataBase::DeserializeVersion, which does not validate the rest of the header) and send streams written by 1.6.x or earlier (WrittenByV160OrEarlier) to mgard_x_160 through Bridge.cpp.in, which converts the Config and results between the two namespaces. The compatibility decoder starts from the 1.6.0 defaults and takes from the caller only the settings 1.6.0 took. - CMake option MGARD_ENABLE_DATA_COMPATIBILITY (default OFF; it roughly doubles the MGARD-X build). Without it, 1.6.0 data is reported with the new status DataCompatibilityNotEnabledFailure. - The bug fixes made to the current pipeline also apply to the compatibility one: MaxDim slab offsets along a dimension other than the slowest, zstd size truncation and unchecked decompression, and the Serialize/Deserialize alignment mismatch. Files that 1.6.0 itself could not decode because of these now decode. Verified against a build of MGARD 1.6.0: 61 CLI-written files (1D-5D, float/double, huffman/huffman-zstd, s = inf/0/1, rel/abs) decompress byte-identically on SERIAL and within 1.5e-6 of range on CUDA; 68 API-only files (reorder, non-uniform, SingleDim, MaxDim and Block decomposition, CPU lossless, double) and 60 zstd files likewise. Current data is unaffected by the routing, and an OFF build reports 1.6.0 data with the new status. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…composition The three domain decomposition types become one: the domain is cut into a grid of blocks of a common, possibly non-cubic shape, so a 400x300x200 array with 100x300x200 blocks is the old MaxDim split and cubic blocks the old Block split. Along each dimension block i starts at i * block_shape[d]; a remainder narrower than 3 nodes is merged into the block before it, so every block is a valid hierarchy. Config::domain_decomposition (and block_size, domain_decomposition_dim, domain_decomposition_sizes) is replaced by: - domain_decomposition_mode = Auto (default): one block when the data fits the memory limit; otherwise the largest block extent (the slowest dimension on ties) is halved until the largest block, merged remainder included, fits. It fails if even 3-node blocks do not fit. - domain_decomposition_mode = Manual with domain_decomposition_block_size: extents larger than the data are clamped; compression fails with an error if the size count does not match the dimensions, any extent is below 3, or the largest block does not fit the memory limit. Shape adjustment is rejected in Manual mode. DomainDecomposer is rewritten around the block grid (subdomain_start / subdomain_shape / block_shape replace the per-type branches and dim_subdomain_offset), and its per-dimension copy loops are replaced by one generic N-D box copy: dimensions contiguous on both sides are merged, then the box is a single Copy1D/Copy2D/Copy3D, looping only over dimensions beyond the third. This also fixes the old MaxDim offsets for a dimension other than the slowest, which copied past the end of the array. RuntimeX: MemoryManager::CopyND is renamed Copy2D, and Copy3D is added (cudaMemcpy3DAsync / hipMemcpy3DAsync; a loop of Copy2D on the other backends). SYCL and Kokkos Copy2D now honour the leading dimensions. Format: DomainDecomposition gains repeated block_shape and is written as BLOCK (NOOP when not decomposed). Older MAX_DIMENSION and BLOCK headers are read as the equivalent block shape; VARIABLE is no longer supported. Metadata stores domain_decomposition_block_shape instead of ddtype/dim/size. CLIs: mgard-x and mdr-x take -dd auto|manual and -dd-size <D> <b_1> ... <b_D>. The MDR QoI tools always used one subdomain per variable (now Manual blocks of shape[0] / 3) and drop -dd. Tested on SERIAL and CUDA: compression of 1D-4D data with Auto (fitting and memory-limited), MaxDim-like, cubic, merged-remainder, last-dimension, clamped, 2D and 4D blocks (incl. uncompressed blocks) stays within the error bound, also when decompressed standalone; MDR refactor/reconstruct of the same layouts stays within 1e-2/1e-4/1e-6; files with the previous MaxDim and Block headers decode within the bound; Manual-mode errors are reported; unit tests pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Release notes: add MGARD 1.6.x data compatibility, the generic block domain decomposition (Auto/Manual), Copy3D, MDR-X ZSTD bitplane compression, the breaking Config changes (domain decomposition settings, num_global_refactoring_level), the memory reductions, optional MPI for MDR, and the recent fixes. MGARD-X guide: document MGARD_ENABLE_DATA_COMPATIBILITY and decompressing data from earlier versions, the domain decomposition mode and block size, and the DataCompatibilityNotEnabledFailure status. Bring the CLI section in line with the current mgard-x options, and fix stale Config entries (projection_mode, huff_block_size default, nonexistent num_dev). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
get_device_type mapped "sycl" to device_type::HIP (copied from the hip case), so -d sycl failed with a missing-backend error in SYCL-only builds and ran on the AMD GPU in builds with both HIP and SYCL. Affects mgard-x, mdr-x, mdr-x-qoi, pmdr-x-qoi and pmdr-x-qoi-io. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Formatting only: line wrapping, comment reflow, and one long string literal split into adjacent literals. Covers the mgard-x-160 compatibility tree, the domain decomposer, Metadata, the device adapters, the executables, and Hierarchy/HybridHierarchy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.