Small reads stop reading whole files - #192
Merged
Merged
Conversation
Five places read a whole file to use a few bytes of it: - migrate_rings read the entire .rings on every writer open to compare an 8-byte magic. It now reads the magic, and the file only when it is v1. - grain_header read the entire .grain on every ship turn to keep its first 16 bytes. - tape_end and the rotate summary parsed the entire .rings for the last record. read_index_last reads the header and that one record. - Session::coverage walked every chunk of the destination on every ack, which is every received chunk. The session now keeps the runs as chunks arrive; nothing in a receive session removes chunks, so the list stays exact. The records sink re-parsed .bark about four times a second to look for a retention or wal change. It now uses the stat-gated LivePolicy that append already uses, so a steady state is one stat per tick. A manifest that stops parsing keeps the last good policy with one warning, as it does for append, where the sink used to skip enforcement silently. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First of the changes from the full-read sweep: the sites where a whole file is read to use a few bytes of it. Each is small and independent.
migrate_rings(every writer open).ringsto compare an 8-byte magicserve::grain_header(every ship turn).grainto keep 16 bytesship::tape_end, rotate summary.ringsfor the last recordformat::read_index_last: header plus one recordSession::coverage(every received chunk).bark~4×/s to look for a retention or wal changeLivePolicy, asappendalready isSession::coverageis exact because nothing in a receive session removes chunks (no retention, rotation or reset there), and the list is seeded from the existing chunks when the session opens.Behaviour change
The sink used to skip retention silently when the manifest stopped parsing. It now keeps the last good policy and warns once, as
appenddoes: never silently unbounded.Measured
Records sink with retention declared, steady state (strace, ~6 s after startup):
.barkopens.barkstatsI did not trace the residual opens. A live
timberfs set wal=trueon a running sink is still applied within a few seconds.Tests
read_index_lastequals the end of a full parse (v2, a later longer header, empty, a partial trailing record, v1 numbering);push_seqis the same fold asruns_of;grain_headeris the first 16 bytes andNonefor a missing or short grain;migrate_ringsleaves a missing, empty, short or non-v1 index alone. Mutating the session to stop updating its runs fails 7 tests.Not here
The intake's tick has the same per-second
.barkparse. Its wal sync iterates every store under one lock, so gating it needs per-store state; that is a separate change. The structural items (resident index per open store,RingsViewfor readers, head-drop rewrites) are not in this PR.🤖 Generated with Claude Code