A seal costs six fsyncs and rewrites the whole manifest; imports pay the same per segment

[PaulSnow]

Found while fixing #27/#29 against 7766d52, and measured on a Samsung 990 PRO / ext4.

A seal costs six fsyncs, and the manifest it commits is rewritten in full every time.

The six, all on the one SegmentStore.seal path:

#	site	what
1	promoteLiveFile, segstore.go:1156	fsync the sealed data file
2	promoteLiveFile, segstore.go:1167	fsync the directory after the rename
3	writeIndexFile, segstore.go:1751	fsync the index file
4	writeIndexFile, segstore.go:1762	fsync the directory again
5	writeManifest, segstore.go:732	fsync the manifest
6	writeManifest, segstore.go:742	fsync the directory a third time

Measured

Sealing a one-record tail repeatedly, reporting cost per seal and manifest size as the segment count rises:

  segments        µs/seal manifest bytes
       250          34792          46974
       500          34123          93974
       750          35122         140974
      1000          36343         187976
      1250          36368         235226
      1500          36155         282476
      1750          36172         329726
      2000          37124         376976

strace -c over 200 of those seals: 1207 fsyncs (six per seal) and 603 renames (three per seal). At the ~5.5 ms per fsync this disk gives (see #32), six fsyncs is ~33 ms — which is the ~36 ms measured. The fsync count is the whole cost of a seal.

Two things follow, and it is worth keeping them apart:

• The fsync count dominates today. ~36 ms per seal, flat in segment count.
• The full-manifest rewrite is real but secondary — for now. The manifest grows ~188 bytes per segment (47 KB at 250 segments, 377 KB at 2000), and rewriting it costs only ~2 ms more across that whole range. It is O(N) per commit, so it is O(N²) over a run that seals N times, and it overtakes the fsyncs eventually — but it is not what is expensive at these sizes. ImportSegmentFile commits a manifest per adopted segment (segstore.go:1561), so a sync of N segments pays the same O(N²).

Asks, in the order they pay

1. Fewer barriers per seal. The three directory fsyncs are for renames into the same directory; one at the end orders them all. The data and index files can be fsynced before a single directory sync rather than each taking its own pair. Six could plausibly be two without weakening the crash contract — the invariant that matters is that a file is durable before the manifest names it, not that each gets its own barrier.
2. Do not rewrite the whole manifest per commit. An append-only record of segment additions and removals, checkpointed occasionally, makes a commit O(1) in the segment count and takes the O(N²) out of both a long run of seals and a sync.

Both are independent of #30's tiered merge, which reduces how many segments exist; this is about what each seal costs regardless. #32 is the special case where the seal has nothing to seal at all.

[PaulSnow]

Two of the six fsyncs are spent on data that does not need to be durable at all.

The index is derived, and the store already knows it

Every key is written twice: once in the .dat record header (key(32) valueLen(8) value) and once in the .idx entry (key(32) offset(8) length(8)). The .dat copy is authoritative — it is fsynced before the rename in promoteLiveFile, it is the only thing ExportBlock transports, and buildIndexFor reconstructs the entire index from it by scanning. openSegment already calls that when the .idx is absent, and ImportSegmentFile calls it for every segment a peer sends.

So the index is a cache. writeIndexFile nonetheless ends with:

if err = out.Sync(); err != nil {   // fsync the index
	return err
}
...
return syncDir(filepath.Dir(indexPath))   // and its directory

Losing an index costs a rebuild. Fsyncing it costs ~11 ms of device wait on every seal, forever, to avoid that rebuild.

What it is worth

At the ~5.5 ms per fsync this disk gives, dropping both takes a seal from six barriers to four — the measured ~36 ms to roughly ~24 ms, a third off, with no durability lost because the .dat still carries every key.

What it needs to be safe

Not simply deleting the two calls. Today a missing index is handled (rebuild) but a present but corrupt one is not: openSegment checks the magic, trusts the count, and reads the bloom bitmap at a computed offset. An index left half-written by a crash could pass the magic check and yield wrong lookups — silently wrong answers, which is worse than the rebuild being avoided.

So the index has to become self-checking before its fsyncs can go:

1. Put a checksum of the index body in the manifest, which is fsynced and is already the commit point. Verify on open; rebuild on mismatch. The manifest is trusted, so the check does not depend on the artifact being checked.
2. Or put the checksum in the index header and verify it on open — simpler, but it is the corrupt file vouching for itself, so it only catches truncation and bit-rot rather than a stale-but-plausible file.

(1) costs a read of each index at open. Seals happen far more often than opens, so trading a per-seal barrier for a per-open read is the right direction — and recoverOrphans already rebuilds indexes on the recovery path, so the machinery exists.

Filed as part of this issue rather than separately because it is the same six fsyncs; it just turns out two of them are avoidable for a different reason than the other four.