feat(writing): keep file structure together, front-loaded by default

[Blackclaws]

Structure is currently allocated in encode order, so it ends up spread evenly through the file. A reader that fetches ranges rather than seeking freely therefore pays for the whole file to read the structure: on a 620 MB reporting file, walking the 8.2% of bytes that are structure touched 591 of 591 one-megabyte ranges.

H5WriteOptions.MetadataPlacement adds two alternatives to the layout the writer produces today, which stays the default:

Placement	Allocation	Effect
Interleaved	in encode order	the default, and the layout the writer produces today, byte for byte
Aggregated	from blocks that double in size, forming a few clusters	needs no estimate of the total
FrontLoaded	from one region reserved at the front	a reader fetches all structure in one range

FrontLoaded sizes its reservation by measuring, so callers do not have to — and because the measurement is exact, it costs no file size at all.

Measured in RangeReadCostTests, over a block-fetching stream that counts what an HTTP range client would have transferred (600 groups of deflated measurement series, 256 kB blocks):

interleaved    fetched 4,050,543 B of 4,050,543 B   100.0% of file   16 requests
aggregated     fetched   786,432 B of 4,052,705 B    19.4% of file    3 requests
front-loaded   fetched   262,144 B of 4,050,543 B     6.5% of file    1 request

And in MetadataLayoutTests, on 2,000 groups each with a contiguous dataset and a string attribute, at 1 MiB ranges:

interleaved       16,878,158 bytes   structure in 17/17 ranges
front-loaded      16,878,158 bytes   structure in  1/17 ranges

Same byte count, one range instead of seventeen.

How it works

The writer allocates every address before writing and then seeks to it, so segregating regions needed no change to any encoding logic — only the allocator, plus classifying the allocation sites as structure or payload. Three of those are not what they look like:

• An implicit chunk index is not an index. The chunks sit contiguously from the address in the layout message and a chunk's address is arithmetic on it, so what is allocated there is the whole payload of an unfiltered chunked dataset.
• The global heap serves both kinds. An attribute's variable-length value is structure — it is what a viewer reads while browsing — while a dataset's own variable-length elements are payload. One classification for the whole heap has to be wrong for one of them, so the manager keeps a collection open per kind.
• A compact dataset's payload lives inside its object header and makes no allocation of its own, so for datasets under 64 kB payload is structure and there is nothing to separate. Files that lean on PreferCompactDatasetLayout see little from this.

The reservation is sized by a pass that encodes the file against a stream discarding everything and reads the metadata total off the allocator. It is exact rather than estimated because it is the same encoder and the same allocator, which is what lets it account for what an estimate cannot: a chunk index's size scales with chunk count, and global heap collections are allocated at a 4 kB minimum each and left partly empty. An earlier estimator missed both and over-reserved threefold, so it was removed rather than kept as a fallback.

The pass does not compress, and for fixed-size element types it does not touch the data at all — no selection walk, no chunk buffers, no byte copying. Best of seven runs on a 64 MB file, against a full write of the same file:

unfiltered    34.9 ms -> 1.3 ms    43% of a write -> 2%
filtered      29.4 ms -> 0.9 ms   2.6% of a write -> 0.1%

MetadataLayoutTests.TheSizingPassMeasuresExactlyWhatAWriteAllocates asserts per shape — contiguous, compact, chunked filtered and unfiltered, single-chunk, variable-length strings, Nullable<T>, attributes, object references, many groups — that the pass measures exactly what a write allocates. That equality is what both shortcuts rest on, and an inequality either way is a defect: over-counting wastes the reservation's tail, under-counting spills.

Set MetadataReservation to skip the pass. Nothing else requires it — including a deferred write through BeginWrite, since nothing a caller writes later changes how much structure the file has.

Worth flagging

• There is no free list, so a region's unused remainder is abandoned rather than reclaimed. A measured reservation abandons nothing, so this is only a cost when you set MetadataReservation yourself, or when a reservation comes up short and spills into blocks — which loses locality for the remainder rather than failing.
• Metadata blocks grow rather than being fixed. A block is claimed in full, so a fixed size is a floor on file size: at the 8 MB default, 33 kB of metadata produced an 8.4 MB file. Blocks now start at 64 kB and double up to MetadataBlockSize, which bounds what is claimed at roughly twice what is used. MetadataBlockSize is therefore a ceiling, and must be positive for any placement that uses blocks.
• The end-of-file address covers allocated rather than written space, because an address can be referenced without ever being written to: a dataset declared through BeginWrite and never written has its payload allocated and named by its layout message. Under a clustered placement nothing extends the stream over it, and h5dump then reports "invalid dataset size, likely file corruption". The stream is extended to the allocator's mark instead — translated by the base address, since the allocator counts from the superblock and the stream from the start of the file.
• Overlaps feat(writing): size a string attribute to its own value #179 only in H5WriteOptions, where both add a property to the record body; whichever lands second takes a one-line conflict.
• No CHANGELOG.md entry here, on the assumption you would rather write that yourself — happy to add one.

Changes since the first review

Everything below came out of your two comments and of reviewing the result again afterwards.

• Interleaved is the default, as requested. The feature is now purely additive: files written with default options are byte-for-byte what they were.
• The sizing pass no longer walks fixed-size data — your suggestion, implemented; see the reply on that thread for the two corrections it needed.
• Corrected the BeginWrite claim, twice. It was never chunk indexes, and it is no longer variable-length data either. NothingDeferredOutrunsTheSizingPass asserts the shortfall is zero for all three.
• Front loading now costs nothing. The reservation added 12 kB of slack, justified on a reading of the superblock accounting that was backwards — it left the region long, not short. Subtracting instead makes it exact.
• A dataset's variable-length values are no longer counted as structure. They shared the global heap with attribute values, so a variable-length string dataset with no attributes measured 97.3% structure and a front-loaded region swallowed its payload. Now 0.0%.
• Fixed a corruption bug of my own: the extension above compared an allocator address against a stream length without translating for UserBlockSize, so a user block plus a never-written deferred dataset produced a file h5dump rejects.
• Fixed a misclassification of my own: the implicit chunk index, per the bullet above. Worth knowing that the measurements in the first version of this description therefore did not cover unfiltered chunked data.
• Made three assertions able to fail. TestUtils.DumpH5File strips h5dump's error stack so the fixture comparisons do not depend on the installed version — which also strips every trace of failure, so DoesNotContain("error") always passed. On lzf.h5, which h5dump rejects with exit code 1 and 2,910 bytes of error stack, the stripped output contains "error" zero times. There is now a RunH5Dump that reports the exit code, and it also returns null when h5dump is absent so that Skip.If(dump is null) skips instead of failing — it was dead code, and those tests failed on a machine without h5dump.
• Reworked the RangeReadCostTests payload from a strided integer ramp to a measurement series. The ramp is pathological for a fast match finder: .NET 9 replaced the bundled zlib with zlib-ng, and that shape compressed 73% worse there, moving every figure the test reports. The figures above now land within 0.2% on both runtimes with identical request counts — and the request count is now asserted rather than merely printed.
• MetadataBlockSize <= 0 and a negative MetadataReservation now throw instead of silently turning Aggregated into Interleaved.
• Added a Metadata placement section to doc/writing/index.md, since the feature is opt-in and would otherwise only be discoverable from XML docs.

Two pre-existing issues turned up along the way, neither related to placement and neither touched here: #184 (deflate at compression-level: 1 expands data on .NET 9+) and #183 (any UserBlockSize above 512 produces a file PureHDF cannot reopen). #183 is why the new UserBlockSize coverage uses 512 and no larger value.

[Apollo3zehn]

Thanks for this PR as well! This is a nice feature for high latency environments :-)

I started the review and put some time into figuring out why it is necessary to still write the data into the Stream that measures the metadata size because preparing the data for write does not come for free and it would be good to skip unnecessary heavy work. The answer is two fold: There are cases where we can skip that and some where it is not possible, i.e. when writing the data itself causes metadata modifications which is the case for variable length data. So I asked GPT for a plan. Please tell me if you agree to its content or if you have any doubts. If the plan is fine I will add a commit with the proposed changes. The plan:

Suggested improvement: skip raw data writes during metadata sizing for fixed-size datasets

The metadata sizing pass currently avoids compression for filtered chunks, but it still walks dataset data, allocates chunk buffers, copies bytes, and registers raw chunk data. For fixed-size unmanaged datasets, that work appears unnecessary because it does not affect metadata size.

Why this should be safe

MeasureMetadataSize already performs the metadata-relevant work:

• It encodes the file structure with MetadataPlacement = Interleaved and SizeOnly = true.
• It creates each dataset layout and H5D_Base.
• It stores each dataset in Context.DatasetToInfoMap.
• It disposes the writer, which disposes each H5D and finalizes chunk index metadata.
• It encodes global heap collections before writing the final superblock.

For fixed-size unmanaged dataset payloads, WriteData only copies raw data into the target layout. That raw data copy does not change the amount of metadata that must be reserved.

Chunked layout details

For chunked datasets, skipping raw data writes during the sizing pass should still preserve metadata sizing:

• Filtered multi-chunk datasets use fixed-array indexing.
• The fixed-array metadata size depends on WriteChunkInfos.Length, ChunkSizeLength, and page information derived from the entry count.
• It does not depend on the actual chunk address, compressed chunk size, or filter mask values.
• If data writing is skipped, chunk entries may remain at default values, but their encoded width and count remain unchanged.
• Unfiltered multi-chunk datasets use implicit indexing; their data region is allocated during layout creation, so WriteData does not add metadata.
• Single-chunk datasets store the relevant indexing information in the layout message.
• Contiguous datasets allocate payload as AllocationKind.RawData, not metadata.

This matches the existing observation that compressed output size is not metadata-size-relevant: fixed-array entries are fixed-width, and ChunkSizeLength is derived from the uncompressed chunk size.

Keep data encoding where it has metadata side effects

The sizing pass must still run WriteData for datasets whose element encoding can allocate or alter metadata:

• Compact datasets, because the encoded data is stored in the object header.
• Variable-length strings and variable-length sequences, because encoding allocates global heap metadata.
• Object references, because encoding may recursively encode referenced groups or datasets.
• Nullable/reference-containing element types, because they are routed through the per-element encoding path.

Proposed code shape

Add a small guard in the write path, after dataset encoding/layout creation but before WriteData:

if (context.SizeOnly && !RequiresSizeOnlyDataWrite<T>(dataset))
    return;

A possible predicate:

private static bool RequiresSizeOnlyDataWrite<T>(H5Dataset dataset)
{
    return
        dataset.Layout is DataLayoutMessage4
        {
            Properties: CompactStoragePropertyDescription
        }
        || DataUtils.IsReferenceOrContainsReferences(typeof(T))
        || Nullable.GetUnderlyingType(typeof(T)) is not null;
}

The exact location would likely be InternalWriteDataset<T, TElement>, since InternalEncodeDataset should still run in the sizing pass to create layouts, datatype messages, dataspace messages, H5D instances, and deferred metadata structures.

Expected benefit

This should avoid unnecessary work during metadata measurement for fixed-size datasets:

• no source-data traversal
• no chunk-buffer allocation
• no raw byte copying
• no raw chunk registration/allocation for filtered chunks
• no selection walking beyond what is needed for metadata

Compression is already skipped, so this targets the remaining fixed-size raw-data overhead in the measurement pass.

[Apollo3zehn]

Apart from that, due to the complexity involved and the limitations being imposed ("BeginWrite needs an explicit MetadataReservation") I would like to keep Interleaved the default.

BeginWrite needs an explicit MetadataReservation. The measuring pass cannot know what a caller will write later, so it under-counts those chunk indexes and the shortfall is placed as Aggregated would place it.

I am not sure I fully understand this statement. Why is it under-counting chunk indexes? The number of chunks are always known beforehand and will not change during BeginWrite.

[Blackclaws]

I'm on it. I've taken your proposal and Claude implemented it after fixing a couple of things it found in there that didn't work. Also found a number of other edge cases, an update to the PR is coming either today or tomorrow.

[Blackclaws]

Thanks — and yes, I agree with the premise, and with where the plan draws the line. I have implemented it (perf(writing): skip fixed-size data in the metadata sizing pass) rather than leaving you to write the commit, since checking it turned up two things worth flagging. Drop that commit and take your own if you would rather.

Where the plan is right: nothing about a fixed-size dataset's metadata footprint depends on its data. I checked every encode size involved — SingleChunkIndexingInformation.GetEncodeSize and ImplicitIndexingInformation.GetEncodeSize ignore their values entirely, and the chunk-index case is the one you raise on the other thread. The list of things that must still be encoded is right too: compact, variable-length, object references, Nullable<T>.

Two corrections it needed:

The guard has to sit at the WriteData call inside InternalEncodeDataset, not in InternalWriteDataset. InternalWriteDataset is only the deferred path — what writer.Write(dataset, data) reaches. The sizing pass never calls it: it calls writer.Write() and then Dispose, so all of its data encoding goes through InternalEncodeDataset's own WriteData call. Guarding InternalWriteDataset would have skipped nothing at all, and nothing would have failed to say so.

The predicate has to test TElement, not T. T is the dataset type, and an array is itself a reference, so IsReferenceOrContainsReferences(typeof(int[])) is true — asked of T the predicate answers "needs data" for every array-backed dataset, which is very nearly all of them. Same silent no-op. On TElement it answers as intended, and it also covers the object-reference case for free, since H5ObjectReference is a class.

One addition: Nullable<T> is worth keeping in the predicate for a reason the plan does not give. It is not that nullables go through the per-element path — it is that GetTypeInfoForNullableValueType encodes them as a variable-length sequence of one and calls GlobalHeapManager.AddObject, so they take global heap space. Nullable<int> contains no reference, so the reference test does not see it. MetadataLayoutTests has a nullable ints shape that pins this: drop the clause and it measures 200 bytes against the 12,488 a write allocates.

What I did not do is make the compact case conditional. It is safe to skip too — CompactStoragePropertyDescription.Data is allocated by size in DataLayoutMessage4.Create, so the object header's size is already settled — but a compact dataset is capped at 64 kB, so there is nothing to win, and keeping it documents that payload and structure are the same bytes there.

Measured on a 64 MB file, sizing pass against a full write of the same file, best of seven runs each — the fastest run being the one least perturbed by whatever else the machine is doing:

unfiltered    34.9 ms -> 1.3 ms    43% of a write -> 2%
filtered      29.4 ms -> 0.9 ms   2.6% of a write -> 0.1%

The safety net is TheSizingPassMeasuresExactlyWhatAWriteAllocates, which asserts across ten shapes that the pass reports exactly what a write allocates. Equality, not a bound: over-counting wastes the reservation's tail and under-counting spills, so the number has to be the number. That is what makes both shortcuts — no compression, and now no fixed-size data — checkable rather than argued, and it went on to catch two allocation bugs of mine that had nothing to do with this commit.

[Blackclaws]

Interleaved is the default now. Agreed on the reasoning, and it makes the change purely additive — default-option files are byte-for-byte what they were.

On the under-counting: you are right, and my statement was wrong. I checked it rather than argued it, and the shortfall for a deferred chunk index is exactly zero:

filtered chunked int (fixed array)        measured 495   allocated 495   shortfall 0
unfiltered chunked int (implicit index)   measured 192   allocated 192   shortfall 0

Exactly as you say: the chunk count is known before anything is written. WriteChunkInfos is sized to TotalChunkCount in H5D_Chunk4.Initialize, the fixed-array data block is sized from WriteChunkInfos.Length, and Initialize runs during the structure write, which the sizing pass performs in full. Writing the data later fills entries in an array whose length was already fixed.

I then replaced that wrong claim with a second wrong claim, so it is worth being explicit about where it ended up. I said the real shortfall was variable-length data, which was true at the time and is no longer: variable-length values were being allocated on the global heap as structure, and reviewing this again showed that is wrong for a dataset's own elements. An attribute's value is structure — it is what a viewer reads while browsing a file — but a dataset's variable-length elements are its payload, and both went through the same manager. A dataset of variable-length strings with no attributes at all measured 97.3% structure, which meant a front-loaded region swallowed the payload and there was nothing left to separate. Silently, too: the file stays valid and the same size, and only the locality quietly fails to materialise.

With that fixed, a variable-length value's heap space is payload, so writing more of it later does not move the metadata total either:

contiguous strings (global heap)          measured 200   allocated 200   shortfall 0

So the honest answer to your question is better than either of my attempts at it: nothing a caller writes after BeginWrite changes how much structure the file has, and MetadataReservation is not needed for a deferred write at all. A deferred variable-length write under FrontLoaded now opens one region, abandons nothing, and produces a file the same size as the interleaved one. NothingDeferredOutrunsTheSizingPass asserts all three shortfalls are zero — this guidance has been wrong twice, so it seemed worth pinning rather than restating.

The same review turned up a second misclassification of the same kind — an implicit chunk index is not an index, so what is allocated there is an unfiltered chunked dataset's whole payload — plus a corruption bug and a few other things. Those are in the rewritten description rather than here. The pattern they share is worth one sentence though: all of them were invisible from outside, because a misclassified allocation still produces a valid file of the same size and only the locality quietly fails to appear. It follows that the measurements in the first version of this description did not cover unfiltered chunked data or variable-length datasets, and the ones now in it do.

[Blackclaws]

Force-pushed, and the description is rewritten rather than amended — worth a fresh read, since editing it does not notify.

The two replies above answer your comments. Everything else came out of reviewing the result again afterwards, and is listed under Changes since the first review rather than repeated here. Four of those are defects in code you had already read, so they are the ones I would not want buried:

• The extension that covers allocated-but-unwritten space compared an allocator address against a stream length without translating for UserBlockSize, so a user block plus a never-written deferred dataset produced a file h5dump rejects.
• The reservation added 12 kB of slack that was never needed. Front loading now costs zero bytes rather than nearly zero.
• A metadata block was claimed in full, so the 8 MB default was a floor on file size: 33 kB of metadata produced an 8.4 MB file. Blocks now grow.
• Three h5dump assertions could not fail, because TestUtils.DumpH5File strips the error stack the assertions were looking for.

Two unrelated pre-existing bugs turned up on the way and are filed separately: #183 and #184. Neither is touched by this PR.

Two things still open for you rather than for me: whether you want a CHANGELOG.md entry from me or would rather write it, and whether Aggregated is worth keeping at all — front loading now dominates it on every axis I can measure except needing no sizing pass, so if you would rather carry one option instead of two, dropping it is a smaller surface to maintain.