Skip to content

There is no seal, so there is no open region - #184

Open
torstei wants to merge 8 commits into
tallyfrom
feature/tally-no-seal
Open

torstei wants to merge 8 commits into
tallyfrom
feature/tally-no-seal

Conversation

@torstei

@torstei torstei commented Sep 8, 2026 •

Copy link
Copy Markdown
Owner

Base is #183. Six commits, +528/−397 over 10 files. Two thirds of it is a design settled and written down; one third is code deleted because of it.

The headline: docs/plans/tally-design.md

This is the file to read — 212 lines stating the tally design rather than arguing it. The three notes beside it (tally-as-a-tally.md, tally-series-identity.md, tally-partials.md) are 2,834 lines of argument, every one of them touching a mechanism that has since been removed, so the design had to be reconstructed from the threads that reached it — and the tape-shaped answers are the ones that read as obvious.

⚠ Its centre is "What a tally is NOT", listing every tape mechanism that was in the design and was removed, each with its reason:

no seal · no open region · no immutability · no displacement · no revisions · no cardinality cap · no stored markers · no .bark declaration · no replication protocol · no grain index

The rest states it once: what a tally is, the on-disk shape, the write order, the three bounds, the two prunings, definitions, recovery, copying, what is measured, what is open. And it says that where an argument note disagrees, the statement wins.

The code: no seal, so no open region

Sealing is the tape's answer to getting one write per bucket. A block is a file replaced by temp-and-rename, addressed so a range can be re-stated — so the premise does not exist. And the seal is what manufactured the open region: a block assumed immutable needed the filling edge to live elsewhere, in a separate file with its own read path and the only bytes in the store that nothing vouched for.

Deleted: OPEN, checkpoint, read_open, Roller::open_buckets, five tests. query and --unpack now have one source instead of two. Verified after the deletion on a real day: 268,136 lines round-trip with zero value differences, and the query still prunes.

What the design settled, commit by commit

The bound on rewriting is Manifest::floor — not grace, and not the source's retention. The partials note said the latter and had it backwards: retention drops the oldest source bytes, while a late entry arrives from the newest end carrying an old stamp. So source retention bounds re-derivation and bounds lateness not at all.

No markers in the grid, which answers where they live by deletion. !meta is subsumed by the definitions, !cap goes with the cap, !late with displacement — and !drop counts lines a definition claimed and could not measure, so it exists only where a definition claims more than it can measure. A number that appears only when the definition is wrong does not belong in the storage format; the distinction lives in --try's report, where a loose capture is found and fixed. So Block::pack refusing markers is right by design rather than a placeholder.

Identity and retention live in the manifest. DECLARE is passed straight to bark::cmd_create, so it is bark's vocabulary — and a block store has no bark. That splits it three ways: index=true and timestamp_regex are meaningless, retain/retain_size have an analogue, and provenance and labels are essential and had nowhere to go. ⚠ Including the source store's id, which is load-bearing rather than decorative: a citation is an offset into that tape.

Retention drops whole blocks, so its granularity is the block range — a day. retain=730d is honest and anything finer is a rounding dressed as a setting. ⚠ And retain_size is the backstop the cardinality cap was standing in for.

And a consistency pass, because banners were not enough

Two sketches in tally-as-a-tally.md would have been copied as the design: a directory with a .bark, a b/ subdirectory and an open file, and a block whose series row carried a metric name, a unit and a definition id. Both now match what the prototype writes. The subdirectory existed only so 730 files would not sit beside a .bark — which no longer exists.

tally.md now leads with "this describes the TAPE, which ships and works… do not read this file as the direction", since it is the biggest note and every tape mechanism in it is true of what runs today.

Then a mechanical check: every rejected concept, in every argument note, paragraph by paragraph, asking whether each mention is marked as amended or scoped to the tape. Four flags remained, all legitimate — two measurement descriptions and two rejection paragraphs.

Corrections to my own text, kept visible

  • !late was briefly "sharpened" rather than removed. It is removed: a marker is keyed by bucket, so recording one below the floor stores a fact about a bucket the store has forgotten.
  • The manifest section listed identity beside the five fields that exist, which reads as a description of the struct. Now marked built-versus-designed.

486 lib tests, fmt and clippy -D warnings clean, man page corrected.

🤖 Generated with Claude Code

@torstei
torstei force-pushed the feature/tally-no-seal branch 2 times, most recently from 6e79bcd to 1bd97a1 Compare September 8, 2026 19:42
@torstei
torstei changed the base branch from tally to feature/tally-metric-table September 8, 2026 21:52
@torstei
torstei changed the base branch from feature/tally-metric-table to tally September 8, 2026 22:13
torstei and others added 6 commits September 9, 2026 00:14
Sealing is the tape's answer to getting one write per bucket: a line, appended,
never revisited. A block is a file replaced by temp-and-rename and addressed so
a range can be re-stated, so the premise does not hold here — and it was the
seal that manufactured the open region, a block having been assumed immutable
and the filling edge therefore needing somewhere else to live.

Deleted: OPEN, checkpoint, read_open, Roller::open_buckets and five tests.
A bucket still filling is in a block like any other, so `query` and `--unpack`
have one source instead of two.

⚠ What bounds the rewriting is Manifest::floor, not grace and not the source's
retention — an earlier draft said the latter and had it backwards: retention
drops the OLDEST source bytes while a late entry arrives from the newest end,
carrying an old stamp. So source retention bounds re-derivation and bounds
lateness not at all.

⚠ And a sample below the floor is REPORTED, not stored. A marker is keyed by
bucket, so recording one there would store a fact about a bucket the store has
forgotten, in a store organised entirely by range. So no !late: one whose
block exists needs no annotation, the number being simply right, and one below
the floor has nothing to annotate. That leaves !drop as the only marker with a
claim on the grid — a claimed line that could not be read makes a stored
bucket short — with !meta subsumed by the definitions and !cap gone with the
cap.

⚠ No lateness limit should be imposed to make part of the store stable: that
trades a number that improves for one that is knowingly incomplete, and
correcting a number in place is the whole reason a mutable cell was the point.
What survives of grace is write batching, whose cost is real — one late sample
rewrites a whole day block, ~917 KB measured — and whose answer is a WAL for
samples in the .sap shape, not a bound.

Also recorded: commit renames the block into place before saving the manifest,
so a rewrite at the same (t0, generation) leaves a window where read_block
fails its own "a sealed block may not do this" check. Exceptional before, the
common case now, and fixed by the third identity component the partials note
already asks for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
!drop counts lines a metric CLAIMED and then could not read — not lines that
failed to match, which are skipped and counted nowhere. So it exists only where
a definition claims more than it can measure: a loose capture, an optional
group that did not participate, a decode source whose claimed line lacks the
measured key. A number that appears only when the definition is wrong does not
belong in the storage format.

The "not mine" versus "mine but unusable" distinction is a diagnostic and
already has a home in --try's per-metric report, against a sample, where a
loose capture is found and fixed. ⚠ And the general alarm beats the specific
one: a producer whose format changes makes the metric flatline either way, and
if the claim stops matching there are no drops at all. Outcome::Dropped and
the Seen counters stay as accounting; note_drop writing a sample into the
tally is what goes.

With !meta subsumed by the definitions, !cap gone with the cap and !late with
displacement, nothing is left to place — so Block::pack's refusal is right by
design rather than a placeholder, and tally-as-a-tally's speculation about
columns beside the presence bitmap is struck.

⚠ A decision about the BLOCK design only. The tape writes !drop today and that
shipped in 0.33.0; while the tape is being superseded, leave it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DECLARE is passed straight to bark::cmd_create, so it IS bark's vocabulary —
and a block store has no bark. Which splits it three ways: index=true,
timestamp_regex and the rest are meaningless; retain and retain_size have an
analogue; provenance and labels are essential and had nowhere to go. The
manifest is already the commit point and already read on every query, so it
takes all of it.

Its own id, a block store being a store in its own right. ⚠ The SOURCE
store's id, and load-bearing rather than decorative: a citation is an offset
into the source's tape and is uninterpretable without knowing which store it
indexes. And the source's labels COPIED at creation, because the source may be
deleted long before a two-year tally — a record of what it said then, not a
live view, the rule the definitions already follow.

Retention drops whole blocks, so its granularity IS the block range, settled
at a day: retain=730d is honest and anything finer is a rounding dressed as a
setting. retain_size should exist too, easier here than on a tape since
Entry.bytes is already in the manifest.

⚠ And it is not a nicety: the size bound is the backstop the cardinality cap
was standing in for. Removing max_series leaves the question of what stops an
unbounded label filling the disk, and the answer is a bound on the STORE
dropped from the oldest end, not a guess in a document nobody revisits. Two
ceilings, and only the memory one is still open.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four notes and 2,834 lines of argument, every one of them touching a mechanism
that was removed — so the design had to be reconstructed from the threads that
reached it, and the tape-shaped answers are the ones that read as obvious.
This states it once.

What a tally is: a grid in day-sized columnar blocks, its manifest the commit
point holding identity, the source store's id, the source's labels copied at
creation, retention in whole blocks and the floor. Definitions copied in, one
immutable file per applied set named by its offset, applied in one act.
Recovery is re-derivation; copying is a file sync or a bundle; there is no
replication protocol.

⚠ And a "what a tally is NOT" section, which is the point of the file: no
seal, no open region, no immutability, no displacement, no revisions, no
cardinality cap, no stored markers, no .bark declaration, no replication
protocol, no grain index. Every one was in the design and was removed, and
most are what an append-only tape needs rather than what a tally needs.

The three design notes now say they are arguments and point at the statement;
tally.md says it describes the TAPE, which ships, and must not be read as the
direction.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two sketches in this note would have been taken as the design: a directory
with a .bark, a b/ subdirectory and an open file, and a block whose series row
carried a metric name, a unit and a definition id. Both are now what the
prototype writes — manifest.json and blocks beside it, definitions/ for the
documents; a metric table carrying the definition id, series indexing into it,
the unit in the definitions.

The subdirectory existed so 730 files would not sit beside a .bark that a
forest scan looks for, and there is no .bark. Segments survive as a batching
candidate, with their trigger renamed from a seal to a flush, which is what it
would actually be.

Wording matters here more than usual: "the buckets that sealed since the last
append" is how the concept gets back in.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The manifest section listed identity, source, labels and retention beside the
five fields that exist, which reads as a description of the struct. Marked.

And the marker question is answered rather than "easier" — there are none.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@torstei
torstei force-pushed the feature/tally-no-seal branch from 1bd97a1 to 43f7769 Compare September 8, 2026 22:16
torstei and others added 2 commits September 9, 2026 00:32
Ten bullets saying what a tally is NOT, and every one of them was a property
of a timberfs store — so the section was ten restatements of one fact, each
phrased as a difference from a thing the reader was never told a tally was.
And the reason it felt necessary is this author's history with the design, not
anything a reader brings to it.

It is now one statement and a test. The statement: a tally store is a
directory of files with a manifest, not a .bark/.trunk pair. The test, which
is the property everything else follows from: a TAPE gets one write per
bucket, so sealing, grace, displacement and revisions all serve a premise a
block — replaced whole by temp-and-rename — never has. When a mechanism
suggests itself, ask what it assumes.

The list is redundant with the rest of the document, which states the
properties positively already; the mechanisms that were tried and removed are
argued out one at a time in tally-partials.md, where the history belongs. A
reader learning what a tally is should not have to learn what it once was.

Two gaps the rewrite exposed: the note referenced "the grid holds numbers" as
established and never stated it, so "What a tally is" now does, along with the
mutable cell; and a heading read "why there is no replication protocol" where
"copying is a file sync or a bundle" says the same thing forwards.

The same standard applied to the man page paragraph this PR added — it
explained that there is no sealed-versus-open distinction and no separate
file, which only parses for a reader who knew the previous design — and to
Series's doc comment, which spent four lines on where the definition is NOT.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The note narrated itself: a title saying "as agreed", a head calling itself
"the STATEMENT" against "the arguments that reached it", a warning opening
"that is the whole warning, and it is enough" — a sentence that only parses
for someone who knows a list used to sit there — and a closing paragraph
explaining why the file does not list what it once did.

None of that is about a tally. What a reader needs from those places is two
facts and they are now stated plainly: this file is authoritative where the
notes beside it differ, and whether a mechanism belongs is decided by "It is
not a timberfs store".

Also gone: "do not read this section as a description of the struct" (an
instruction about the document, where the fact — which fields exist and which
are designed — was the useful half), "purity is a property this design GAINS"
(the property is that a sample's bucket comes from its own stamp; nothing
consults a watermark), and a copying paragraph that narrated the decision
before reaching it.

The pointers in ROADMAP, the plans index and the three notes' banners carried
the same framing and say the precedence rule instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant