tally --blocks writes the grid instead of the lines

[torstei]

Base is #184. Six commits, +644/−21 over 6 files. Two pieces: the block writer, and the manifest identity it records into.

What this provides, and what it does not

timberfs tally --blocks DIR writes the numbers as columnar blocks instead of tally lines on stdout — and a block store now carries its own id and the id of the tape it was derived from.

⚠ The flag is a harness, not a fleet feature. Nothing in a provisioned deployment invokes it: production is tally --provision plus the follower running tally --run, and this touches neither. What it buys is that the fold has a block destination, and that destination provably agrees with the shipped one.

⚠ And it names a DIRECTORY, which every other timberfs argument deliberately does not — "a forest is the ONE thing a timberfs command names by path; every other argument names a store, and a store is found by what it declares." That is the absence of an interface rather than a choice: nothing could address a block store until this PR gave it an id, and there is still no forest to find one in. The flag's help and the man page both say so, so the scaffold cannot quietly become the interface.

The writer, verified against the shipped path

300,000 real log lines from a production performance log, through that site's own extractor, into a store and out through feed's records stream:

--block-flush	block writes	vs the line path
1000	43	✅ 0 differences
10000	5	✅ 0 differences
50000	1	✅ 0 differences

47,523 samples either way, and --unpack renders back exactly what tally wrote as lines. So it is the same tally stored differently, and the merge path is right across 43 commits as well as one.

It attaches at the one seam the fold already had: every batch — sealed, provisional and final — goes through emit(), so routing it to a block store is a single branch.

⚠ The buffer is the write-amplification control, not a latency choice. A block is a day, so each commit rewrites up to a megabyte, and one real day is ~268,000 samples — hence a default of 50,000. Buffering costs visibility (a sample is not in a block until flushed); losing it to a crash costs a re-read, the position not having moved. A WAL for samples is the durable form and is not this.

The manifest identity

Its own id — a UUID with a created stamp, minted once and never touched, following bark's rule for bark's reason: a path is an address, the id is what the store IS.

⚠ Option rather than required, because a manifest written before identity existed has none — and minting one on read would hand an old store a new identity in silence. Missing is a defect to repair deliberately, which is how timberfs identity treats the same absence.

The source store's id, taken from the records stream's own source record, which carries one. Load-bearing rather than provenance: a citation is an offset into that store's tape.

⚠ A second source is REFUSED, and this is the part worth having:

Error: these blocks were derived from store 8a169289-… and this run reads
6c77265f-… — a citation is an offset into ONE tape

Verified end to end. Without it, one store's records folded into another's blocks would leave every citation pointing into the wrong tape — silently, the offsets being plausible either way. The happy path checks out too: a real run wrote its own id plus a source matching the source store's bark id exactly.

Left short of done, deliberately: labels is a field nothing populates, because the source record carries an id but no labels — they arrive from whatever resolves the source store, which is the provisioning. And retain_ms/retain_bytes are declared, not enforced: retain_ms validates as a whole number of blocks, but nothing sets them, so no manifest claims a policy that is not honoured.

Two boundaries drawn on purpose

⚠ One store in, one directory out — why this is on the stdin path and not --run. A provisioned run serves a selection, with a Sink per source store, and where each one's blocks go is a provisioning question.

⚠ The unit announcement is suppressed, not redirected. A block store carries its own definitions and a unit is a property of one, so emitting !meta would be the same fact in two places. Nothing reaches stdout under --blocks.

⚠ And routing the provisional drain into blocks is correct only because Block::merge replaces a cell. A provisional bucket states its total so far; the complete one later states the whole total again. When merge becomes additive for partials this double-counts — the comment at idle() says so.

Notes added while building

Three things the implementation forced into the design note: a record stream is forward-only, which is fine for a crash (safe_offset keeps the position behind every open bucket) and not for a repair; a repair is a bounded read, not a rewind — query --records --from/--to | tally already exists, so recomputing a window never stops the live path; and the fold is fed sequentially by a follower, with the measurements for why parallel workers are not the first thing to reach for.

Seven new tests (493 lib tests pass, fmt and clippy --all-targets -- -D warnings clean).

🤖 Generated with Claude Code