Conversation
torstei
force-pushed
the
feature/tally-no-seal
branch
2 times, most recently
from
September 8, 2026 19:42
6e79bcd to
1bd97a1
Compare
Sealing is the tape's answer to getting one write per bucket: a line, appended, never revisited. A block is a file replaced by temp-and-rename and addressed so a range can be re-stated, so the premise does not hold here — and it was the seal that manufactured the open region, a block having been assumed immutable and the filling edge therefore needing somewhere else to live. Deleted: OPEN, checkpoint, read_open, Roller::open_buckets and five tests. A bucket still filling is in a block like any other, so `query` and `--unpack` have one source instead of two. ⚠ What bounds the rewriting is Manifest::floor, not grace and not the source's retention — an earlier draft said the latter and had it backwards: retention drops the OLDEST source bytes while a late entry arrives from the newest end, carrying an old stamp. So source retention bounds re-derivation and bounds lateness not at all. ⚠ And a sample below the floor is REPORTED, not stored. A marker is keyed by bucket, so recording one there would store a fact about a bucket the store has forgotten, in a store organised entirely by range. So no !late: one whose block exists needs no annotation, the number being simply right, and one below the floor has nothing to annotate. That leaves !drop as the only marker with a claim on the grid — a claimed line that could not be read makes a stored bucket short — with !meta subsumed by the definitions and !cap gone with the cap. ⚠ No lateness limit should be imposed to make part of the store stable: that trades a number that improves for one that is knowingly incomplete, and correcting a number in place is the whole reason a mutable cell was the point. What survives of grace is write batching, whose cost is real — one late sample rewrites a whole day block, ~917 KB measured — and whose answer is a WAL for samples in the .sap shape, not a bound. Also recorded: commit renames the block into place before saving the manifest, so a rewrite at the same (t0, generation) leaves a window where read_block fails its own "a sealed block may not do this" check. Exceptional before, the common case now, and fixed by the third identity component the partials note already asks for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
!drop counts lines a metric CLAIMED and then could not read — not lines that failed to match, which are skipped and counted nowhere. So it exists only where a definition claims more than it can measure: a loose capture, an optional group that did not participate, a decode source whose claimed line lacks the measured key. A number that appears only when the definition is wrong does not belong in the storage format. The "not mine" versus "mine but unusable" distinction is a diagnostic and already has a home in --try's per-metric report, against a sample, where a loose capture is found and fixed. ⚠ And the general alarm beats the specific one: a producer whose format changes makes the metric flatline either way, and if the claim stops matching there are no drops at all. Outcome::Dropped and the Seen counters stay as accounting; note_drop writing a sample into the tally is what goes. With !meta subsumed by the definitions, !cap gone with the cap and !late with displacement, nothing is left to place — so Block::pack's refusal is right by design rather than a placeholder, and tally-as-a-tally's speculation about columns beside the presence bitmap is struck. ⚠ A decision about the BLOCK design only. The tape writes !drop today and that shipped in 0.33.0; while the tape is being superseded, leave it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DECLARE is passed straight to bark::cmd_create, so it IS bark's vocabulary — and a block store has no bark. Which splits it three ways: index=true, timestamp_regex and the rest are meaningless; retain and retain_size have an analogue; provenance and labels are essential and had nowhere to go. The manifest is already the commit point and already read on every query, so it takes all of it. Its own id, a block store being a store in its own right. ⚠ The SOURCE store's id, and load-bearing rather than decorative: a citation is an offset into the source's tape and is uninterpretable without knowing which store it indexes. And the source's labels COPIED at creation, because the source may be deleted long before a two-year tally — a record of what it said then, not a live view, the rule the definitions already follow. Retention drops whole blocks, so its granularity IS the block range, settled at a day: retain=730d is honest and anything finer is a rounding dressed as a setting. retain_size should exist too, easier here than on a tape since Entry.bytes is already in the manifest. ⚠ And it is not a nicety: the size bound is the backstop the cardinality cap was standing in for. Removing max_series leaves the question of what stops an unbounded label filling the disk, and the answer is a bound on the STORE dropped from the oldest end, not a guess in a document nobody revisits. Two ceilings, and only the memory one is still open. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four notes and 2,834 lines of argument, every one of them touching a mechanism that was removed — so the design had to be reconstructed from the threads that reached it, and the tape-shaped answers are the ones that read as obvious. This states it once. What a tally is: a grid in day-sized columnar blocks, its manifest the commit point holding identity, the source store's id, the source's labels copied at creation, retention in whole blocks and the floor. Definitions copied in, one immutable file per applied set named by its offset, applied in one act. Recovery is re-derivation; copying is a file sync or a bundle; there is no replication protocol. ⚠ And a "what a tally is NOT" section, which is the point of the file: no seal, no open region, no immutability, no displacement, no revisions, no cardinality cap, no stored markers, no .bark declaration, no replication protocol, no grain index. Every one was in the design and was removed, and most are what an append-only tape needs rather than what a tally needs. The three design notes now say they are arguments and point at the statement; tally.md says it describes the TAPE, which ships, and must not be read as the direction. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two sketches in this note would have been taken as the design: a directory with a .bark, a b/ subdirectory and an open file, and a block whose series row carried a metric name, a unit and a definition id. Both are now what the prototype writes — manifest.json and blocks beside it, definitions/ for the documents; a metric table carrying the definition id, series indexing into it, the unit in the definitions. The subdirectory existed so 730 files would not sit beside a .bark that a forest scan looks for, and there is no .bark. Segments survive as a batching candidate, with their trigger renamed from a seal to a flush, which is what it would actually be. Wording matters here more than usual: "the buckets that sealed since the last append" is how the concept gets back in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The manifest section listed identity, source, labels and retention beside the five fields that exist, which reads as a description of the struct. Marked. And the marker question is answered rather than "easier" — there are none. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
torstei
force-pushed
the
feature/tally-no-seal
branch
from
September 8, 2026 22:16
1bd97a1 to
43f7769
Compare
Ten bullets saying what a tally is NOT, and every one of them was a property of a timberfs store — so the section was ten restatements of one fact, each phrased as a difference from a thing the reader was never told a tally was. And the reason it felt necessary is this author's history with the design, not anything a reader brings to it. It is now one statement and a test. The statement: a tally store is a directory of files with a manifest, not a .bark/.trunk pair. The test, which is the property everything else follows from: a TAPE gets one write per bucket, so sealing, grace, displacement and revisions all serve a premise a block — replaced whole by temp-and-rename — never has. When a mechanism suggests itself, ask what it assumes. The list is redundant with the rest of the document, which states the properties positively already; the mechanisms that were tried and removed are argued out one at a time in tally-partials.md, where the history belongs. A reader learning what a tally is should not have to learn what it once was. Two gaps the rewrite exposed: the note referenced "the grid holds numbers" as established and never stated it, so "What a tally is" now does, along with the mutable cell; and a heading read "why there is no replication protocol" where "copying is a file sync or a bundle" says the same thing forwards. The same standard applied to the man page paragraph this PR added — it explained that there is no sealed-versus-open distinction and no separate file, which only parses for a reader who knew the previous design — and to Series's doc comment, which spent four lines on where the definition is NOT. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The note narrated itself: a title saying "as agreed", a head calling itself "the STATEMENT" against "the arguments that reached it", a warning opening "that is the whole warning, and it is enough" — a sentence that only parses for someone who knows a list used to sit there — and a closing paragraph explaining why the file does not list what it once did. None of that is about a tally. What a reader needs from those places is two facts and they are now stated plainly: this file is authoritative where the notes beside it differ, and whether a mechanism belongs is decided by "It is not a timberfs store". Also gone: "do not read this section as a description of the struct" (an instruction about the document, where the fact — which fields exist and which are designed — was the useful half), "purity is a property this design GAINS" (the property is that a sample's bucket comes from its own stamp; nothing consults a watermark), and a copying paragraph that narrated the decision before reaching it. The pointers in ROADMAP, the plans index and the three notes' banners carried the same framing and say the precedence rule instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Base is #183. Six commits, +528/−397 over 10 files. Two thirds of it is a design settled and written down; one third is code deleted because of it.
The headline:
docs/plans/tally-design.mdThis is the file to read — 212 lines stating the tally design rather than arguing it. The three notes beside it (
tally-as-a-tally.md,tally-series-identity.md,tally-partials.md) are 2,834 lines of argument, every one of them touching a mechanism that has since been removed, so the design had to be reconstructed from the threads that reached it — and the tape-shaped answers are the ones that read as obvious.⚠ Its centre is "What a tally is NOT", listing every tape mechanism that was in the design and was removed, each with its reason:
The rest states it once: what a tally is, the on-disk shape, the write order, the three bounds, the two prunings, definitions, recovery, copying, what is measured, what is open. And it says that where an argument note disagrees, the statement wins.
The code: no seal, so no open region
Sealing is the tape's answer to getting one write per bucket. A block is a file replaced by temp-and-rename, addressed so a range can be re-stated — so the premise does not exist. And the seal is what manufactured the open region: a block assumed immutable needed the filling edge to live elsewhere, in a separate file with its own read path and the only bytes in the store that nothing vouched for.
Deleted:
OPEN,checkpoint,read_open,Roller::open_buckets, five tests.queryand--unpacknow have one source instead of two. Verified after the deletion on a real day: 268,136 lines round-trip with zero value differences, and the query still prunes.What the design settled, commit by commit
The bound on rewriting is
Manifest::floor— notgrace, and not the source's retention. The partials note said the latter and had it backwards: retention drops the oldest source bytes, while a late entry arrives from the newest end carrying an old stamp. So source retention bounds re-derivation and bounds lateness not at all.No markers in the grid, which answers where they live by deletion.
!metais subsumed by the definitions,!capgoes with the cap,!latewith displacement — and!dropcounts lines a definition claimed and could not measure, so it exists only where a definition claims more than it can measure. A number that appears only when the definition is wrong does not belong in the storage format; the distinction lives in--try's report, where a loose capture is found and fixed. SoBlock::packrefusing markers is right by design rather than a placeholder.Identity and retention live in the manifest.
DECLAREis passed straight tobark::cmd_create, so it is bark's vocabulary — and a block store has no bark. That splits it three ways:index=trueandtimestamp_regexare meaningless,retain/retain_sizehave an analogue, and provenance and labels are essential and had nowhere to go. ⚠ Including the source store's id, which is load-bearing rather than decorative: a citation is an offset into that tape.Retention drops whole blocks, so its granularity is the block range — a day.
retain=730dis honest and anything finer is a rounding dressed as a setting. ⚠ Andretain_sizeis the backstop the cardinality cap was standing in for.And a consistency pass, because banners were not enough
Two sketches in
tally-as-a-tally.mdwould have been copied as the design: a directory with a.bark, ab/subdirectory and anopenfile, and a block whose series row carried a metric name, a unit and a definition id. Both now match what the prototype writes. The subdirectory existed only so 730 files would not sit beside a.bark— which no longer exists.tally.mdnow leads with "this describes the TAPE, which ships and works… do not read this file as the direction", since it is the biggest note and every tape mechanism in it is true of what runs today.Then a mechanical check: every rejected concept, in every argument note, paragraph by paragraph, asking whether each mention is marked as amended or scoped to the tape. Four flags remained, all legitimate — two measurement descriptions and two rejection paragraphs.
Corrections to my own text, kept visible
!latewas briefly "sharpened" rather than removed. It is removed: a marker is keyed by bucket, so recording one below the floor stores a fact about a bucket the store has forgotten.486 lib tests,
fmtandclippy -D warningsclean, man page corrected.🤖 Generated with Claude Code