Fix 14 pre-existing defects found while working on fz

[ayzk]

Every defect below was found while working on the fz branch and then verified to reproduce on master. None of the new fz modules are included — this is only the fixes to code master already has.

Memory safety

Module	Defect
Lossless_bypass::compress	memcpy'd srcLen bytes into dst regardless of dstCap, so any payload larger than the caller's buffer was an unconditional heap overflow. Lossless_zstd already checks and throws; bypass now does the same. Caught by AddressSanitizer on Linux: ALGO_BIOMDXTC is the one algorithm that pairs its codec with Lossless_bypass, so nothing shrinks the payload, and glibc reported it as munmap_chunk(): invalid pointer. macOS's allocator does not abort on it.
MemoryUtil::read	remaining_length was guarded by assert, which Release compiles out, so a truncated or corrupt stream read past the end of the buffer. Now throws.
RegressionPredictor::load	debited remaining_length by coeff_size * sizeof(int) (the decoded bin count) while decode() advances the cursor by the much smaller encoded byte count, so the budget ran out early. This is what the MemoryUtil guard was hiding: on the HDF5 filter path with ALGO_LORENZO_REG at eb = 1e-4 it read past the buffer on every miranda velocity field.
LinearQuantizer::recover_unpred	no bounds check on the unpredictable list, so a stream with more zero bins than stored values read past the end.
RunlengthEncoder	no size_est(), so SZGenericCompressor sized its buffer without accounting for the encoder and encode() overran it.
KmeansUtil	reserve() followed by operator[] writes (UB), and uniform_int_distribution(0, num) indexing data[num]. Live in mdz when dims[1] > 5000.
H5Z-SZ3	sized the compressed buffer as sizeof(T) * num * 2, which is below SZ_compress's own minimum for a small chunk. An explicit chunk of a few hundred elements aborted with "The buffer for compressed data is not large enough".
NoPredictionDecomposition, InterpolationDecomposition	no size_est() override, so the compressor sized its buffer from 0 while the quantizer's unpredictable list could be arbitrarily large.

Correctness

Module	Defect
TimeSeriesDecomposition	with data_ts0 == nullptr the timestep loop predicted from stale values: block_data's compress-side path has no write-back, so timestep 0 in data was never updated with its reconstruction while decompress() predicted from its own. Measured 1.94x the error bound; 0.997x after the fix.
ArithmeticEncoder	bytesToInt64_bigEndian(bytes) >> 20 sign-extends when the top byte's MSB is set, pushing the value outside the 44-bit MAX_CODE window. A 60-stream sweep gave 30 correct, 23 wrong, 7 SIGSEGV.
HuffmanEncoder	stateNum = max - offset + 2 narrows to int, so a wide bin range goes negative and the state-table malloc is UB. Now throws and points at HuffmanEncoderV2, which switches to a sparse map.

Portability

• 6 headers used std::make_shared without <memory>. libc++ provides it transitively and libstdc++ does not, so this is latent until a translation unit changes its include order — it broke the Linux build on fz once a new test file changed the include order.
• 24 headers were not self-contained under one or both standard libraries (missing <limits>, <cmath>, <cstddef>, <array>, SZ3/def.hpp, or a project header). All 57 now compile standalone under both clang/libc++ and GCC 16/libstdc++. The script that checks this lives on the fz branch (tools/test/check_headers.py) and is not part of this PR.
• .gitignore's bare test pattern matched tools/test/, hiding every test file from git. Now /test.

Verification

• clang and GCC 16, Release + BUILD_TESTING=ON + BUILD_H5Z_FILTER=ON: 0 errors, 0 warnings, all tests pass
• Compressed output for the default algorithm is unchanged
• Each fix was A/B'd against its unfixed state; the TimeSeriesDecomposition and H5Z-SZ3 numbers above are from those runs

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR backports a set of verified defect fixes from the fz branch onto master, focusing on memory-safety hardening, correctness fixes in encoding/prediction paths, and portability improvements (self-contained headers + gitignore/test tooling).

Changes:

• Add runtime bounds checks and safer buffer sizing to prevent out-of-bounds reads/writes and too-small compressed buffers.
• Fix correctness issues in regression coefficient loading, arithmetic decoding, time-series reconstruction, and k-means sampling.
• Improve portability by making headers self-contained and adding a header-compilation checker; fix .gitignore to avoid hiding tools/test/.

Reviewed changes

Copilot reviewed 32 out of 33 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tools/test/check_headers.py	New script to compile each header standalone to catch transitive-include breakages.
tools/H5Z-SZ3/src/H5Z_SZ3.cpp	Uses SZ3 size bound to ensure minimum compressed buffer capacity for small chunks.
include/SZ3/utils/Statistic.hpp	Adds missing standard headers for self-contained compilation.
include/SZ3/utils/Sample.hpp	Adds missing <cassert> include ordering for self-contained compilation.
include/SZ3/utils/QuantOptimization.hpp	Adds missing <cmath> / SZ3/def.hpp for self-contained compilation.
include/SZ3/utils/MemoryUtil.hpp	Replaces release-stripped assert with exceptions for truncated streams.
include/SZ3/utils/KmeansUtil.hpp	Fixes UB in sampling vector sizing and adjusts random index distribution bounds.
include/SZ3/utils/Iterator.hpp	Adds missing SZ3/def.hpp include for self-contained compilation.
include/SZ3/utils/Extraction.hpp	Adds missing STL + project includes for self-contained compilation.
include/SZ3/utils/BlockwiseIterator.hpp	Adds missing includes and introduces block_data::values() to access unpadded layout.
include/SZ3/quantizer/Quantizer.hpp	Adds helper to query optional quantizer size_est() at compile time.
include/SZ3/quantizer/LinearQuantizer.hpp	Adds bounds check on unpredictable list during recovery.
include/SZ3/preprocessor/Transpose.hpp	Adds missing standard headers for self-contained compilation.
include/SZ3/preprocessor/PreFilter.hpp	Adds missing standard headers for self-contained compilation.
include/SZ3/predictor/RegressionPredictor.hpp	Fixes remaining-length accounting for Huffman-decoded coefficients.
include/SZ3/predictor/LorenzoPredictor.hpp	Adds missing standard headers for self-contained compilation.
include/SZ3/lossless/Lossless.hpp	Adds missing <cstddef> and project include for self-contained compilation.
include/SZ3/lossless/Lossless_bypass.hpp	Adds missing <cstdlib> include for self-contained compilation.
include/SZ3/encoder/XtcBasedEncoder.hpp	Adds missing includes for self-contained compilation.
include/SZ3/encoder/RunlengthEncoder.hpp	Implements size_est() to avoid compressor buffer under-sizing.
include/SZ3/encoder/HuffmanEncoder.hpp	Adds guard for excessively wide bin ranges to prevent invalid state-table sizing.
include/SZ3/encoder/ArithmeticEncoder.hpp	Fixes sign-extension issue by shifting an unsigned 64-bit value.
include/SZ3/decomposition/TimeSeriesDecomposition.hpp	Fixes timestep prediction to use reconstructed timestep-0 values consistently.
include/SZ3/decomposition/SZBioMDXtcDecomposition.hpp	Adds missing <limits> include for self-contained compilation.
include/SZ3/decomposition/NoPredictionDecomposition.hpp	Adds size_est() using quantizer estimate to prevent under-sized buffers.
include/SZ3/decomposition/InterpolationDecomposition.hpp	Adds size_est() and missing includes to avoid under-sized buffers and portability issues.
include/SZ3/decomposition/Decomposition.hpp	Adds missing includes for self-contained compilation.
include/SZ3/decomposition/BlockwiseDecomposition.hpp	Fixes include set/order for self-contained compilation.
include/SZ3/compressor/SZGenericCompressor.hpp	Adds missing <memory> include for self-contained compilation.
include/SZ3/compressor/specialized/SZExaaltCompressor.hpp	Adds missing includes for self-contained compilation.
include/SZ3/api/impl/SZAlgoInterp.hpp	Adds missing <memory> include for self-contained compilation.
include/SZ3/api/impl/SZAlgoBioMD.hpp	Fixes include ordering/requirements for self-contained compilation.
.gitignore	Changes test pattern to /test to avoid hiding tools/test/.

Suppressed comments (1)

include/SZ3/utils/KmeansUtil.hpp:310

• The random sampling loop intends to avoid duplicate indices, but sampledkeys is never updated, so duplicates are still possible. Also uniform_int_distribution<> uses int bounds, which can truncate/overflow when num exceeds INT_MAX. Use a size_t distribution and insert into the set as you sample.

        std::uniform_int_distribution<> dis2(0, static_cast<int>(num) - 1);
        std::unordered_set<size_t> sampledkeys;
        //            printf("total_num=%lu, sample_num=%lu\n", num, sample_num);
        for (size_t i = 0; i < sample_num; i++) {
            do {

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[ayzk]

Superseded by #144, which consolidates these fixes together with the eight open bug-fix pull requests (#131–#139) reviewed line by line.