Skip to content

Small reads stop reading whole files - #192

Merged
torstei merged 1 commit into
mainfrom
bugfix/bounded-small-reads
Oct 2, 2026
Merged

torstei merged 1 commit into
mainfrom
bugfix/bounded-small-reads

Conversation

@torstei

@torstei torstei commented Oct 2, 2026

Copy link
Copy Markdown
Owner

First of the changes from the full-read sweep: the sites where a whole file is read to use a few bytes of it. Each is small and independent.

Site Was Now
migrate_rings (every writer open) read the whole .rings to compare an 8-byte magic reads the magic; the file only when it is v1
serve::grain_header (every ship turn) read the whole .grain to keep 16 bytes reads 16 bytes
ship::tape_end, rotate summary parsed the whole .rings for the last record format::read_index_last: header plus one record
Session::coverage (every received chunk) walked every chunk of the destination: quadratic over a replication the session keeps its runs as chunks arrive
records sink tick re-parsed .bark ~4×/s to look for a retention or wal change stat-gated LivePolicy, as append already is

Session::coverage is exact because nothing in a receive session removes chunks (no retention, rotation or reset there), and the list is seeded from the existing chunks when the session opens.

Behaviour change

The sink used to skip retention silently when the manifest stopped parsing. It now keeps the last good policy and warns once, as append does: never silently unbounded.

Measured

Records sink with retention declared, steady state (strace, ~6 s after startup):

.bark opens .bark stats
before 4.3 /s 0
after 0.3 /s 1 /s

I did not trace the residual opens. A live timberfs set wal=true on a running sink is still applied within a few seconds.

Tests

read_index_last equals the end of a full parse (v2, a later longer header, empty, a partial trailing record, v1 numbering); push_seq is the same fold as runs_of; grain_header is the first 16 bytes and None for a missing or short grain; migrate_rings leaves a missing, empty, short or non-v1 index alone. Mutating the session to stop updating its runs fails 7 tests.

Not here

The intake's tick has the same per-second .bark parse. Its wal sync iterates every store under one lock, so gating it needs per-store state; that is a separate change. The structural items (resident index per open store, RingsView for readers, head-drop rewrites) are not in this PR.

🤖 Generated with Claude Code

Five places read a whole file to use a few bytes of it:

- migrate_rings read the entire .rings on every writer open to compare an
  8-byte magic. It now reads the magic, and the file only when it is v1.
- grain_header read the entire .grain on every ship turn to keep its first
  16 bytes.
- tape_end and the rotate summary parsed the entire .rings for the last
  record. read_index_last reads the header and that one record.
- Session::coverage walked every chunk of the destination on every ack,
  which is every received chunk. The session now keeps the runs as chunks
  arrive; nothing in a receive session removes chunks, so the list stays
  exact.

The records sink re-parsed .bark about four times a second to look for a
retention or wal change. It now uses the stat-gated LivePolicy that append
already uses, so a steady state is one stat per tick. A manifest that stops
parsing keeps the last good policy with one warning, as it does for append,
where the sink used to skip enforcement silently.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@torstei
torstei merged commit 930b83b into main Oct 2, 2026
7 checks passed
@torstei
torstei deleted the bugfix/bounded-small-reads branch October 2, 2026 22:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant