Skip to content

Commit ae17bb7

Browse files
committed
Commit the chunker positive-control diagnostic (CodeRabbit review, PR #1144)
CodeRabbit correctly flagged: the mechanism-null conclusion for semantic_chunker relied on a diagnostic that was run once as a throwaway script and deleted, never entering git — a later reader could reproduce the KILL but not the evidence for calling it a mechanism null rather than a harness bug. Fixed by adding the control as a 4th, committed arm in chunker_falsifier.rs itself: the module's own adversarial synthetic- corners shape (centroids 0-4 vs 250-254) against the same real table, reported at every threshold. Re-ran after landing — reproduces the same zero. Updated the plan (§4c), EPIPHANIES, and STATUS_BOARD to point at the committed control instead of the deleted one. The ratio-ambiguity and public-signature findings from the same review were already addressed on the current head (the earlier ratio-fix commit already states both directions; from_engine_state has zero in-tree callers, verified by repo-wide grep, and the crate is workspace-excluded so no external consumer exists to break) — replied on those threads rather than re-editing settled text. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PFnYKqw6d7TTiB9cT8eFdK
1 parent 4e19d1d commit ae17bb7

4 files changed

Lines changed: 55 additions & 9 deletions

File tree

‎.claude/board/EPIPHANIES.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
**What the closure plan asked.** The two remaining §1c rows each had a promotion path gated on a falsifier: `semantic_chunker` → deepnsm-v2's text side, `spiral_segment` → the codec certification battery. Both probes ran on real data (the tier-1..4 calibration corpus + the Jina v5 tokenizer + `jina-v5-codebook` for the chunker; five real baked 256x256 tables — jina-v3, bge-m3, reranker, jina-v5 u8 and i8 — for the spiral codec). Both came back KILL, and the two KILLs are structurally different failures worth telling apart.
66

7-
**`semantic_chunker` fails at the MECHANISM — recall = 0.000 at every threshold swept (0.30 / 0.45 / 0.60), across 168 cross-topic passages.** An all-zero result is exactly the shape a broken harness produces, so before trusting it a non-committed diagnostic (built, run, deleted — never entered git) re-ran the module's OWN pre-existing adversarial positive control — the synthetic maximally-separated centroid corners from its `detects_boundary_between_topics` test — against the SAME `jina-v5-codebook` table. **It also produced zero boundaries.** So the null is not a probe-design artifact: the perturb→think→top-k-Jaccard convergence pattern this module is built on simply does not diverge on an HDR-encoded table, even on inputs engineered to be as separable as the 256-centroid space allows. The module's own existing test already carried a hedge to this effect ("On uniform HDR tables the convergence patterns may not diverge strongly") — this is that hedge turned into a measured, falsifiable KILL.
7+
**`semantic_chunker` fails at the MECHANISM — recall = 0.000 at every threshold swept (0.30 / 0.45 / 0.60), across 168 cross-topic passages.** An all-zero result is exactly the shape a broken harness produces, so before trusting it a diagnostic re-ran the module's OWN pre-existing adversarial positive control — the synthetic maximally-separated centroid corners from its `detects_boundary_between_topics` test — against the SAME `jina-v5-codebook` table. **It also produced zero boundaries.** So the null is not a probe-design artifact: the perturb→think→top-k-Jaccard convergence pattern this module is built on simply does not diverge on an HDR-encoded table, even on inputs engineered to be as separable as the 256-centroid space allows. The module's own existing test already carried a hedge to this effect ("On uniform HDR tables the convergence patterns may not diverge strongly") — this is that hedge turned into a measured, falsifiable KILL. (The control was first run as a throwaway script and only its result recorded here; CodeRabbit's review on PR #1144 correctly flagged that a deleted diagnostic leaves the mechanism-null conclusion unreproducible, so it is now a 4th committed arm in `chunker_falsifier.rs` — same corners, same zero.)
88

99
**`spiral_segment` fails at the RATIO on u8 tables (fidelity is fine there); on i8 it fails BOTH gates — corrected post-merge by a Codex review comment on this PR (`chatgpt-codex-connector[bot]`, P2), see the addendum below.** Pearson r and Spearman rho clear the 0.9980 gate on every u8 table at every max_error tried — the codec reconstructs distance rows accurately there. The i8 table does NOT clear it at any max_error tested: at its best configuration (0.005) r = 0.9993 passes but rho = 0.9975 falls short of the 0.9980 floor, and rho only gets worse as max_error loosens (0.9906 / 0.9590 / 0.7137). What kills every table, u8 or i8, is `bytes(spiral) <= bytes(u8)/2`: the best compression measured is ~0.28x on u8 (i.e. the spiral encoding is **~3.6x LARGER** than the u8 table it would replace) and ~0.80x on i8 at its loosest setting, nowhere near the module's own "51x compression" doc-comment claim or even the ecosystem's 2x floor. Root cause is visible in the segment count: a real per-row CDF from a trained codebook needs ~114-143 spiral segments (8 bytes each) to hit even a loose 0.05 max_error, because the codec's fitting premise — few segments suffice per row — only holds on smooth synthetic curves, and a trained model's distance rows are not smooth. The 51x claim was never false against the data it was tested on; it was false against the distributional shape of data it was never tested on.
1010

‎.claude/board/STATUS_BOARD.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@
2020
| D-TEH-0 | census: live footprint measured (one required consumer via `bridge_gate`; one optional via `with-engine`), 51-file fate table, open-row reconciliation, four-wave closure, idea harvest, four rulings asked | **Shipped (plan-only, this PR)** |
2121
| D-TEH-1 | W1: `bridge_gate` (seven items) → `lance_graph_contract::bridge_gate`; callcenter re-imports and drops the path dep; thinking-engine keeps a re-export shim | **Shipped 2026-09-02** (before: required dep, 6 crossing sites, dep-drop alone fails 6 × E0433; after: zero thinking-engine deps in callcenter metadata, contract 1303/1303, callcenter 156/156, driver default + `with-engine` green, ALU artery files byte-identical). `with-engine` re-point deferred by stop condition: D-TTV-1 not landed |
2222
| D-TEH-2 | W2: ghost prior → planner `nars/ghost_prior.rs` over `WisdomMarker`, per-thought, two-sided falsifiers; crate `ghosts.rs` deleted | **Shipped 2026-09-02** — `GhostPrior`/`PriorFloor` in planner `nars/ghost_prior.rs`, 14 tests incl. the calibration gate (default floor = `Marker`, the gate reversed the first declaration); `ghosts.rs` + `think.rs` deleted; TD-GHOST-ECHO-DUP-1 resolved; consumer D-HOUSE-4 unblocked |
23-
| D-TEH-3 | W2: calibration MATH → `jc` (ruling 4, 2026-09-02: lift if correct, perfect in jc if not; crate copies die); glue stays in the lab crate; `semantic_chunker` / `spiral_segment` decided by their falsifiers | **Shipped 2026-09-02/03, all three halves closed** — `jc::drift` (re-encode drift + delta summary) and `jc::quorum` (pairwise agreement, bands, Cronbach report) added; cronbach LIFTED (same formula; `f32` copy loses the `1e7`-shifted fixture), spearman PERFECTED-IN-JC (retired copy was tie-blind: 1.000 vs 0.9487 on one tie); lab `cronbach.rs` deleted, `reencode_safety` / `silu_correction` / `ground_truth` are glue over jc. `semantic_chunker`: **KILL** — recall 0.000 at every pre-registered threshold, confirmed a real mechanism null via a non-committed positive-control probe; stays LAB. `spiral_segment`: **KILL** — fidelity passes on every u8 table (r/rho >= 0.998) but NOT on i8 (rho tops out at 0.9975, below the 0.998 floor, corrected 2026-09-03 per Codex review on #1144); compression fails on every table regardless (~0.28x a u8 table, not the claimed 51x); stays LAB, no certification battery scheduled. Plan `thinking-engine-harvest-closure-v1.md` §4c |
23+
| D-TEH-3 | W2: calibration MATH → `jc` (ruling 4, 2026-09-02: lift if correct, perfect in jc if not; crate copies die); glue stays in the lab crate; `semantic_chunker` / `spiral_segment` decided by their falsifiers | **Shipped 2026-09-02/03, all three halves closed** — `jc::drift` (re-encode drift + delta summary) and `jc::quorum` (pairwise agreement, bands, Cronbach report) added; cronbach LIFTED (same formula; `f32` copy loses the `1e7`-shifted fixture), spearman PERFECTED-IN-JC (retired copy was tie-blind: 1.000 vs 0.9487 on one tie); lab `cronbach.rs` deleted, `reencode_safety` / `silu_correction` / `ground_truth` are glue over jc. `semantic_chunker`: **KILL** — recall 0.000 at every pre-registered threshold, confirmed a real mechanism null via a committed positive-control arm in `chunker_falsifier.rs` (the module's own adversarial synthetic-corners shape, added 2026-09-03 per CodeRabbit review on #1144 so the evidence is reproducible, not just asserted); stays LAB. `spiral_segment`: **KILL** — fidelity passes on every u8 table (r/rho >= 0.998) but NOT on i8 (rho tops out at 0.9975, below the 0.998 floor, corrected 2026-09-03 per Codex review on #1144); compression fails on every table regardless (~0.28x a u8 table, not the claimed 51x); stays LAB, no certification battery scheduled. Plan `thinking-engine-harvest-closure-v1.md` §4c |
2424
| D-TEH-4 | W3: ENTROPY M8 engine collapse with dtype parity suite; 5 cascade shapes + 3 lens modules collapse | Queued |
2525
| D-TEH-5 | W4: residue deleted, crate renamed `thinking-lab` with a `--manifest-path` CI line; §2 rows closed; `TD-THINKING-ENGINE-EXCLUDED-DEBT-1` paid | Queued — closes the chapter |
2626

‎.claude/plans/thinking-engine-harvest-closure-v1.md‎

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -287,13 +287,17 @@ threshold | recall | null p95 | false splits | bnd/pass | verdict
287287

288288
recall = 0.000 at every threshold means `find_boundaries` never fired ONCE
289289
across all 168 cross-topic passages, at any of the three thresholds swept.
290-
Before trusting an all-zero result, a non-committed diagnostic (built,
291-
run, then deleted — never entered git) ran the module's OWN adversarial
292-
positive-control shape from its `detects_boundary_between_topics` test
293-
(synthetic centroid corners 0-4 vs 250-254, maximally distant in the
294-
256-centroid table) plus a real mixed-topic sentence, both against the same
295-
`jina-v5-codebook` table the falsifier used. **The positive control ALSO
296-
produced zero boundaries.** So this is a genuine mechanism/table-level null:
290+
Before trusting an all-zero result, a diagnostic ran the module's OWN
291+
adversarial positive-control shape from its `detects_boundary_between_topics`
292+
test (synthetic centroid corners 0-4 vs 250-254, maximally distant in the
293+
256-centroid table) against the same `jina-v5-codebook` table the falsifier
294+
used. **The positive control ALSO produced zero boundaries.** This was first
295+
run as a throwaway, non-committed script — a real gap Codex review caught on
296+
this PR (#1144): a deleted diagnostic means a later reader can reproduce the
297+
KILL, but not the reasoning for calling it a mechanism null rather than a
298+
harness bug. Fixed by committing the control as a 4th arm inside
299+
`chunker_falsifier.rs` itself (reproduces the same zero, confirmed by
300+
re-running it after landing). So this is a genuine mechanism/table-level null:
297301
the perturb-think-top-k-Jaccard convergence pattern at `max_cycles: 10` does
298302
not discriminate on this HDR-encoded table at all, even on inputs designed to
299303
be maximally separable. The module's own pre-existing test

‎crates/thinking-engine/examples/chunker_falsifier.rs‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,20 @@
2828
//! AND false splits ≤ 2 of the 8 coherent passages.
2929
//! KILL otherwise → the chunker stays LAB (not ported).
3030
//!
31+
//! A 4th, committed arm — POSITIVE CONTROL: an all-zero recall on the three
32+
//! arms above is exactly the shape a broken harness (wrong table, wrong
33+
//! tokenizer, a wiring bug) would also produce, so a null result is not
34+
//! trustworthy on its own (CLAUDE.md's falsifiability rule: "a null result is
35+
//! a claim about the measurement apparatus until proven otherwise"). This arm
36+
//! reproduces the module's OWN adversarial positive-control shape from
37+
//! `semantic_chunker::tests::detects_boundary_between_topics` — two
38+
//! maximally-separated synthetic centroid clusters (corners 0-4 vs 250-254 of
39+
//! the 256-centroid space) — against the SAME real table the falsifier uses,
40+
//! and reports its boundary count at every threshold. If this control ALSO
41+
//! returns zero, the all-zero falsifier result is a genuine mechanism null,
42+
//! not a harness artifact; if it returns boundaries where the falsifier did
43+
//! not, the harness (not the mechanism) is the thing to investigate.
44+
//!
3145
//! Usage:
3246
//! JINA_V5_TOKENIZER=/path/to/tokenizer.json \
3347
//! cargo run --release --manifest-path crates/thinking-engine/Cargo.toml \
@@ -244,4 +258,32 @@ fn main() {
244258
"\nVERDICT: {} — pre-registered: recall ≥ 0.75 AND recall ≥ null p95 + 0.15 AND false splits ≤ 2, at one threshold",
245259
if any_pass { "PASS (port-eligible)" } else { "KILL (stays LAB)" }
246260
);
261+
262+
// POSITIVE CONTROL — the module's own adversarial shape (two maximally
263+
// separated synthetic centroid clusters), against the same real table.
264+
// See the module doc comment above for why this arm exists.
265+
let mut corners: Vec<u16> = Vec::with_capacity(48);
266+
for i in 0..24u16 {
267+
corners.push(i % 5);
268+
}
269+
for i in 0..24u16 {
270+
corners.push(250 + i % 5);
271+
}
272+
println!("\n{:>9} | {:>18} | note", "threshold", "control boundaries");
273+
let mut control_ever_fires = false;
274+
for &t in THRESHOLDS {
275+
let cfg = config(t);
276+
let n = find_boundaries(&mut engine, &corners, &cfg).len();
277+
control_ever_fires |= n > 0;
278+
println!("{:>9.2} | {:>18} |", t, n);
279+
}
280+
println!(
281+
"\nPOSITIVE CONTROL: {} — synthetic corners (centroids 0-4 vs 250-254, the module's own `detects_boundary_between_topics` shape) {} boundaries on this table",
282+
if control_ever_fires {
283+
"FIRES (the falsifier's all-zero result is NOT a mechanism null — investigate the harness)"
284+
} else {
285+
"STAYS AT ZERO TOO — the falsifier's all-zero result is a genuine mechanism null, not a harness artifact"
286+
},
287+
if control_ever_fires { "produces" } else { "produces zero" }
288+
);
247289
}

0 commit comments

Comments
 (0)