Skip to content

Commit 36e5d17

Browse files
committed
D-SPG-6 re-scoped: the sealed batch is SoA-owned; a consumer that opens a Dataset has already lost
Operator, 2026-09-07: "that's not a convenience you're violating lance-graph 879 batchwriter SOA owned" — then "879 909..912 1049 1198". Read against the merged code, and the finding is worse than the framing I gave it. LanceCycleWriter (graph::cycle_sink, #911 -> #912 Phase A) is the SOLE application writer and the topology is enforced by the type: non-Clone, commit_cycle(&mut self), one long-lived owned Dataset handle. SoA owners are fire-and-forget producers that cast; they are never Lance writers. The consumer-side seal_alpha_cycle I had written violated four properties at once: a second unowned writer reopening per call; a consumer as Lance writer; an unconditional write (#911's empty-cycle versioning was REMOVED, I reintroduced it); and a read-version-then-append TOCTOU whose "refuse, not renumber" guard cannot work, because Append rebases even single-attempt and the compensating delete #911 added was itself removed by #912. sealed_version = base_version + 1 is a VERIFIED IDENTITY inside that sink, not caller arithmetic. Re-deriving it from outside the component that proves it is the transferable tell: the question is not whether the arithmetic is right but who is entitled to say it. - STATUS_BOARD + spec F6/D-SPG-6: re-scoped. The alpha cycle CASTS a descriptor (mailbox, dirty row-range, cycle), never owned rows; read-back is temporal::QueryReference::at + deinterlace. Blocked upstream, not queued in MedCare: LanceShardSink does not exist (#879 honesty ledger) and the read side is unwired (#1198). Gates re-registered as NoChange / Reconciled / HashConflict. - SpogTenants::merged_rows doc: the sentence calling it "the row form the sealed batch per cycle is built from" pointed a future session at exactly this violation. Corrected in place, with the zero-copy sink ruling named (the cast payload is a descriptor, deltas stay in the SoA store and the sink reads them via NodeRowPacket::as_le_bytes at flush). Method unchanged; it is a reading, not a write payload. - EPIPHANIES E-A-CONSUMER-THAT-OPENS-A-DATASET-HAS-ALREADY-LOST-1, including why the DataFusion weight was the symptom: bypassing the owned writer is what reaches for the umbrella crate. "datafusion does joins, we do masking Ops, no joins ever" (operator) — a join materialises the rejected world; the crosswalk only looks like one and is a chain of masked equality sweeps. Gates: fmt 0, clippy --no-deps -D warnings 0, contract spog_tenants 8/8, append-only OK, no new citation decay, supersession index regenerated last. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KCGhDYoQBXs3poaR7sFuqp
1 parent 48e29a1 commit 36e5d17

4 files changed

Lines changed: 102 additions & 6 deletions

File tree

‎.claude/board/EPIPHANIES.md‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,76 @@
1+
## 2026-09-07 — E-A-CONSUMER-THAT-OPENS-A-DATASET-HAS-ALREADY-LOST-1 — re-deriving a proven identity from outside the thing that proves it is the tell
2+
3+
**Status:** FINDING, operator-caught (2026-09-07: *"that's not a convenience you're
4+
violating lance-graph 879 batchwriter SOA owned"*, then *"879 909..912 1049 1198"*).
5+
Verified against the merged code and the #879/#911/#912 arc entries.
6+
**Confidence:** High — every property below is quoted from the shipped module docs.
7+
8+
**What I did.** Writing D-SPG-6 ("sealed batch per cycle"), I wrote a free
9+
`async fn seal_alpha_cycle` in a CONSUMER that did `Dataset::open` to read the
10+
version, computed `cycle = version + 1`, and `Dataset::write`-appended. I
11+
justified it as convenience — reaching for the `lance` umbrella crate because a
12+
neighbouring function already did. That framing was wrong twice over: it was not
13+
convenience, it was an ownership violation; and the DataFusion weight I was
14+
worrying about was a symptom, not the disease.
15+
16+
**What already existed.** `LanceCycleWriter` (`lance-graph::graph::cycle_sink`,
17+
#911 → #912 Phase A) is the **SOLE application writer**, and the topology is
18+
enforced by the TYPE: non-`Clone` (a second handle cannot be minted),
19+
`commit_cycle(&mut self, …)` (two commits cannot interleave), one long-lived
20+
owned `Dataset` handle (no per-operation reopen). The 64k SoA owners are
21+
*"parallel PRODUCERS (fire-and-forget: they cast on behalf of their mailbox and
22+
receive no acknowledgement), never Lance writers"*.
23+
24+
**The four properties my version lacked**, each hard-won in a review round:
25+
26+
| shipped | mine |
27+
|---|---|
28+
| sole writer, non-`Clone`, owned handle | a second unowned writer, reopening per call |
29+
| producers cast, never write | consumer written as a Lance writer |
30+
| **no semantic change → no write → no version** (#911's empty-cycle versioning REMOVED) | wrote unconditionally — an empty saccade mints a version |
31+
| no rollback; durable `(cycle, batch_hash)`, reconcile FIRST, `HashConflict` fails closed | read-version-then-append: a TOCTOU |
32+
33+
**The sharpest of them.** `Append` in Lance **rebases even on a single attempt**
34+
(strict no-rebase exists only for `Overwrite` — measured in
35+
`lance-9.0.0/src/io/commit.rs`). So my "refuse, not renumber" guard — read the
36+
version, compare, then append — *cannot do what its own error message claims*.
37+
#911 first fixed this with a compensating `Dataset::delete`; #912 then REMOVED
38+
that too, because a published manifest is HISTORY and a delete is another
39+
version, not a rollback. I had reinvented a mechanism that was already tried and
40+
already superseded.
41+
42+
**The transferable tell, and it is cheap to check.**
43+
`sealed_version = base_version + 1` is recorded on the #911 entry as *"a verified
44+
identity, not an assumption"* — verified INSIDE the sink, which is what lets
45+
readers derive the cycle↔version mapping from the co-committed frame row with
46+
zero sidecar state. **I re-derived that identity from outside the component that
47+
proves it.** Whenever code computes `next = current + 1` for state another
48+
component owns, the question is not "is the arithmetic right" (it was) but "who
49+
is entitled to say this" — and if the answer is a type you did not call, the
50+
write is already an orphan. A consumer that reaches for `Dataset::open` has
51+
answered that question wrongly before writing a line.
52+
53+
**Why the DataFusion weight was the symptom.** Bypassing the owned writer meant
54+
reaching for the `lance` umbrella crate, which drags the query engine in
55+
transitively. Operator, same session: *"datafusion does joins, we do masking Ops,
56+
no joins ever"* — a join materialises the rejected world (two relations in, a
57+
third out), which is the exact complement of `resident ⊗ A ⊗ B ⊗ C`. The
58+
crosswalk is the thing that LOOKS like a join (MONDO ↔ CUI ↔ LOINC ↔ SNOMED) and
59+
is a chain of masked equality sweeps (§3.2), so there is no join formulation to
60+
carry. Read against
61+
`E-TOPOLOGY-MASKS-MAGNITUDE-COMPOSE-NEVER-COLLAPSE-1` (same day): a query engine
62+
on this path is a fourth thing that collapses topology, masks and magnitude back
63+
into relational algebra.
64+
65+
**Falsifier.** Any consumer-side `Dataset::open` / `Dataset::write` in the
66+
alpha/mask chain; any caller computing a version successor for state it does not
67+
own; any cast payload carrying owned rows rather than a `(mailbox, row-range,
68+
cycle)` descriptor. The withdrawn implementation is banked in the session
69+
scratchpad as the worked example rather than deleted, because the four-row table
70+
above is only legible next to the code that got each row wrong.
71+
72+
---
73+
174
## 2026-09-07 — E-TOPOLOGY-MASKS-MAGNITUDE-COMPOSE-NEVER-COLLAPSE-1 — Mississippi Queen, TERNLOG chaining and BLASGraph pay for ONE operation, from three sides
275

376
**Status:** OPERATOR RULING (2026-09-07, verbatim in substance), recorded on

‎.claude/board/STATUS_BOARD.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
| D-SPG-3 | Rung × tenant cross via `mask_ternlog` on `words()`; `unlooked[D]` read (temporal 09 Stage 2) | ~~Queued~~ **Shipped 2026-09-07** (MedCare-rs `6bf7764`, `spog_masks::{rung_tenant_cell, unlooked}`; Sonnet worker from spec, orchestrator-gated: fmt/clippy clean, 11/11 incl. the Codex #1221 sibling-classid falsifier; mutation-fired: AND2→AND_ANDNOT2 in the cell fails the count test). Measured on the real image: 500 MONDO + 300 CUI claims at rung 3 → cell counts equal the materialized scanpath filter in all 16 tenants; `unlooked(disease) = disease − 500`, disjoint from `any_rung`, tiles the domain with the cell **⊘ 2026-09-07 (rebase onto `main` after #1220):** the cited MedCare-rs sha is on NO pushed ref. Checked against every `origin/*` ref of AdaWorldAPI/MedCare-rs at main `9f9b7be` (i.e. after #620 AND #621 merged): `git cat-file -t` reports MISSING for `6bf7764`, and `spog_masks` appears in no ref. The measurements below stand as reported by the session that ran them; the STATUS is regraded **Shipped-unpushed** — it is not verifiable from any public history until that branch is pushed. **⊘ 2026-09-07, later the same day: PUSHED** as MedCare-rs `e5febf9` on `origin/claude/medcare-rs-continue-ufsazd` (rebased onto `9f9b7be`; 11/11 after the rebase). Status: **Shipped** (pushed, PR open, merge pending #1221). | cell count == materialized scanpath filter; `AND_ANDNOT2 ≠ AND3` wherever `any_rung` non-empty |
1111
| D-SPG-4 | PROBE-CROSSWALK-MASK-1, gates (a)–(h), real image; W0 pins FK columns before timing | Queued (MedCare-rs probe; record here) | plan §6 |
1212
| D-SPG-5 | **Migrate the hand-rolled MedCare alpha onto the #1198 contract alpha** (operator 2026-09-07, spec F9): `attention::WatchedRows`' `RefCell<AlphaOverlay>` + `domain_rung` writer, `backreference::combined_base`, and the bare-overlay `nodesoa::alpha` writer all become consumers of `AlphaTunnel` lanes + `SpogTenants` G routing + `merge()`; frontier dispatch routes through tenants over `all-lanes.soa`; `reflection` = tenant `attended_mask` OR; `domain_rung` retired as rung writer | ~~Queued~~ **Partial 2026-09-07** — MedCare-rs #621 merged (`9f9b7be`). **Leg (i) SHIPPED:** `attention::WatchedRows` holds a `SpogTenants` built by `over_census`, `land()` claims at the CALLER's processing rung, `reflection(tenants, domain)` concatenates the shadows whose block resolves to that domain through the one `Domain::of_classid` mapping, and `domain_rung`/`rung_for` are gone — `origin/main` carries those names only inside a historical doc comment. The equivalence gate was met as stated: 48/48, with BOTH reflection tests and every order-sensitive `obo::` test unchanged. **Legs (ii)–(iv) NOT done**, measured on `origin/main` rather than assumed: `backreference::combined_base` still builds its own base (16 bare `AlphaOverlay` references); `medcare-nodesoa::alpha::{overlay_to_batch, write_alpha_overlay}` still take a bare overlay (15 references — and still ZERO callers outside their own module, so that leg migrates a writer nothing calls yet); `frontier_dispatch` still calls `dispatch_thought(base, …)` with no tenants. Two bare-overlay sites this row never named also remain: `medcare-cohorts::graph_feed` and `medcare-soa::patient` (1 reference each). **⊘ later the same day (this session): legs (ii)–(iv) SHIPPED** — MedCare-rs `e722dd1` (PR #622, draft, blocked on this PR): (ii) `claim_domain_and_patient` borrows the caller's `AlphaAllocation` and returns `SpogTenants::over_census` over the combined base, the patient = tenant `graph_of(patient_address)` (falsifier: `NoTenant` when its G is undeclared; `PatientSession` unchanged — it only hands out the allocation); (iii) `nodesoa::alpha::{tenants_to_batch, write_alpha_tenants}` over the new contract `SpogTenants::merged_rows` (this PR) — the bare-overlay pair stays as the single-overlay form; (iv) `FrontierDispatch::tenants()`, the per-G reading of the agnostic dispatcher's scanpath (the dispatcher itself stays agnostic — `dispatch_thought` is upstream). Gates: first-thought 49/49, nodesoa 15+8, clippy 0. **Still open from the count above:** the two sites `graph_feed` / `medcare-soa::patient` (1 reference each) — **read, same hour: both are DOC COMMENTS, not code** (`graph_feed.rs:4313` describes what the debugger reads; `medcare-soa/src/patient.rs:179` is a doc link, and a stale one — it names `medcare_nodesoa::alpha::AlphaOverlay`, a path that moved upstream in #1112). Zero code references ⇒ the F9 residue is **0**; D-SPG-5 reads **Shipped**. What stays open is F5's scalar-vs-SIMD cell question (operator) and the stale doc link (trivial, MedCare-side). | tenant reflection == `domain_rung` reflection as sets ×8 domains; stamps carry ladder rungs 1..=9 |
13-
| D-SPG-6 | Sealed batch per cycle: `merge()` → `node_rows_to_batch` → one append; `cycle = version + 1` | Queued (MedCare-rs) | two cycles = two versions; a read at v never sees v+1; `enable_stable_row_ids` grep-fenced |
13+
| D-SPG-6 | ~~Sealed batch per cycle: `merge()` → `node_rows_to_batch` → one append; `cycle = version + 1`~~ **⊘ RE-SCOPED 2026-09-07 (operator: "879 909..912 1049 1198" — the batch writer is SoA-owned).** The write was never MedCare's. `LanceCycleWriter` (`graph::cycle_sink`, #911 → #912 Phase A) is the **SOLE application writer** (non-`Clone`, `commit_cycle(&mut self, …)`, one owned long-lived `Dataset` handle); SoA owners are fire-and-forget PRODUCERS that `cast` and are never Lance writers; `sealed_version = base_version + 1` is a **verified identity inside that sink**, not caller arithmetic; and **no semantic change → no write → no version** (#911's empty-cycle versioning was REMOVED). Correct scope: MedCare's alpha cycle CASTS a descriptor `(mailbox, dirty row-range, cycle)` — never owned rows — and reads back via `temporal::QueryReference::at` + deinterlace. | **Blocked (lance-graph), not queued (MedCare-rs)** — the gap is upstream and already on the ledger: `LanceShardSink` does not exist (#879 honesty ledger: "durability FAKE"), and the read side is unwired (#1198: `deinterlace` has no production caller, `cast()` has zero production call sites). A withdrawn consumer-side implementation is banked in the session scratchpad as the worked example of the violation. | ~~two cycles = two versions; a read at v never sees v+1; `enable_stable_row_ids` grep-fenced~~ — re-registered when the cast path is wired: an empty cycle performs ZERO Lance operations (`CommitOutcome::NoChange`), a re-submitted identical batch reconciles (`Reconciled`), and a same-cycle different-hash batch fails closed (`HashConflict`) |
1414
| D-SPG-7 | ogar-r2il consumer (`RANK` + `TERNLOG 0x86`) via `lance-graph-ogar` | Queued — gates on D-SPG-4 | lifted program survivor mask == hand chain bit-for-bit |
1515
| D-SPG-8 | DataFusion containment (`with_row_id`/`with_row_addr` OFF + red-if-flipped test) | ~~Queued~~ **Shipped 2026-09-07** — MedCare-rs #621 merged (`9f9b7be`). Production registers `LanceTableProvider::new(ds, /* with_row_id */ false, /* with_row_addr */ false)` (`medcare-server/src/state.rs:945`), and the red-if-flipped test OBSERVES that call site instead of building its own provider: `row_identity_columns_are_absent_from_the_registered_provider` (`:1264`) reads the schema of the table `AppState::build_session_context` actually registered, and asserts a hand-built identity-on provider over the SAME dataset carries exactly two more columns — two-sided, so the silent half is a measurement and not a statement about an empty schema. Its own doc records that an earlier draft hardcoded the flags in the test and stayed GREEN when production was flipped; the disable run is what caught it. The pre-existing grep fence `row_identity_containment::no_lance_row_identity_consumer_exists` (`:1609`) stays sharp because the test deliberately never spells either column name in code. **Gate met:** the scan schema carries neither identity column, and a flip is detectable. | scan schema carries neither `_rowid` nor `_rowaddr` |
1616

0 commit comments

Comments
 (0)