Skip to content

test(ingest): peer falsification of #1639 author-model contract on direct + arbitrated writers (Refs #655) - #1652

Draft
EtanHey wants to merge 2 commits into
claude/orc-final-credit-batch-z2uwd0from
claude/draft-pr-review-proof-kra7vl
Draft

EtanHey wants to merge 2 commits into
claude/orc-final-credit-batch-z2uwd0from
claude/draft-pr-review-proof-kra7vl

Conversation

@EtanHey

@EtanHey EtanHey commented Nov 5, 2026 •

Copy link
Copy Markdown
Owner

TASK MARKER: cloud-credit-20261105-brainlayer-hammer-peer1639

Draft, stacked on #1639. This is a peer proof/report only: one new test file, no production source change. It is not final cross-vendor acceptance. It does not alter #1639's branch. Refs #655 (not Fixes).

Change and evidence

Base and head

What the file does

Every contract test feeds the same synthetic transcript to both watcher writers and compares the results:

  • the direct writer: create_flush_callback(arbitrated=False);
  • the arbitrated writer: create_flush_callback(arbitrated=True), then enqueue_watcher_chunk, the queue JSONL, and drain_once into a separate scratch DB.

The tests also assert on the queued event's own metadata.model, so a drain-side default cannot make them pass.

Test (passes at head) What it pins
test_mixed_claude_codex_batch_… One batch holds: a user turn claiming a model, assistant model A, assistant model B (padded with whitespace and a tab), <synthetic>, an assistant with no model, a returned tool_result claiming a model, and a realtime-normalized Codex reply whose rollout named a model. The direct and drained rows must match each other and the expected {content: (sender, model)} map.
test_busy_direct_write_spills_… A forced apsw.BusyError with a 0 s deadline rolls the direct write back and spills to the queue. The event carries the author model; the drain persists it; the user row stays unknown.
test_replayed_flush_and_replayed_queue_file_… A flush is replayed (offsets lost), and the same queue file is drained twice (crash before unlink). Each path ends with one row, the author unchanged, and seen_count == 2.
test_author_model_never_bleeds_… Three interleaved sessions across two batches, on both paths. No model carries into a session or batch that did not record one.
test_adversarial_model_names_… (14 cases) Exactly 128 characters is kept and 129 is rejected. Also rejected: 1 MiB, RLO U+202E, an isolate pair, ZWJ, BOM, an inner NBSP, a lone surrogate, NUL, TAB, True, and a dict. A homoglyph plus combining mark is stored verbatim. The results are identical on both paths.
test_secret_shaped_model_string_… A key-shaped model string, assembled at runtime and fake, never appears raw in either DB.
test_copied_content_keeps_each_turns_own_author A verbatim copy inside one session folds into the first writer's row (same chunk_id), identically on both paths. An assistant quoting the user never makes the user's text model-authored, and a user pasting the answer back never strips its author.

Old-behaviour control and mutants (each applied to the head, then restored; git status src clean afterwards)

Command: python -m pytest <file> -q -p no:cacheprovider, run on the original #1639 file and on this file.

Mutant #1639's own file (16 tests) this file
M0: pre-#1639 classify.py + codex.py from d6b82f0 6 failed 15 failed, and 8 strict xfails XPASS (see the F1/F3 note)
M1: enqueue_watcher_chunk drops model from the event 16 passed (survives) 9 failed
M2: drain _apply_watcher drops model 16 passed (survives) 9 failed
M3: only the busy-spill enqueue_chunk drops model 16 passed (survives) 1 failed
M4: control filter becomes "\n" in model instead of isprintable() 16 passed (survives) 8 failed
M5: length check > becomes >= 16 passed (survives) 1 failed
M6: the model bleeds into the next entry that has none 9 failed 14 failed
M7: scrub_chunk_values skips metadata 16 passed (survives) 1 failed
M8: drain stamps model A on every row 16 passed (survives) 16 failed

This confirms #1639's own note: its tests do not cover the arbitrated path, the spill path, bidi/boundary validation or metadata scrubbing. This file kills all 8 mutants.

Other commands (Linux cloud container, Python 3.13.18, editable .venv)

  • pytest tests/test_author_model_writer_paths.py: 20 passed, 9 xfailed, stable over 3 reruns (about 5.9 s).
  • --runxfail on the 9 findings: all 9 fail on the intended invariant assertion. None fails on a setup error.
  • Neighbours: the 342-test set from fix(ingest): persist per-turn authoring model for Claude Code and Codex chunks (Refs #655) #1639, plus test_arbitration.py, test_drain.py, test_dedupe.py and this file: 502 passed, 9 xfailed.
  • BRAINLAYER_PREPUSH_SCOPE=changed-only BRAINLAYER_CHANGED_FILES_RANGE=a03bca7..HEAD scripts/run_tests.sh: "BrainLayer test gate passed." (rc 0).
  • ruff check and ruff format --check pass on the new file.
  • core.hooksPath is unset in this cloud clone, so .githooks/pre-push did not run on push. Nothing was bypassed on purpose; the scoped gate above was run by hand.

Findings (severity, location, reproduction)

Each finding is pinned as an xfail(strict=True) test. A fix makes it XPASS, which fails the suite until the marker is removed. No production fix is in this PR; the owner decides whether each needs a fresh issue or PR.

F1 — MEDIUM: a near-duplicate merge attributes another author's text to the first author's model. Direct and arbitrated paths alike, crossing sessions and vendors.

  • Where:
    • src/brainlayer/dedupe.py:647 merge_duplicate_chunk: the updates dict at :693 never reads or reconciles metadata, while _merged_content appends the incoming text as record 2.
    • Reached from the direct writer at src/brainlayer/watcher_bridge.py:615 and from the arbitrated drain at src/brainlayer/drain.py:573 (_insert_or_merge_chunk).
  • Repro (test_near_duplicate_merge_…, 4 parametrizations):
    1. A Claude session (project proj) writes an 80-word assistant turn with model=claude-fixture-model-a.
    2. A different session writes the same text with one word changed (SimHash distance 2, which is within SIMHASH_THRESHOLD = 3; asserted as a precondition). That second session is either (a) a Codex rollout under the same project, which the realtime path attributes to no model, or (b) Claude claude-fixture-model-b.
    3. Result on both writers: one row, seen_count 2, content that contains the second author's text, and metadata.model == "claude-fixture-model-a".
  • Scope: dedupe merging existed before fix(ingest): persist per-turn authoring model for Claude Code and Codex chunks (Refs #655) #1639. The false claim is new. Under M0 (pre-fix(ingest): persist per-turn authoring model for Claude Code and Codex chunks (Refs #655) #1639) these 4 tests XPASS, because no row claimed a model. Same-session exact copies (see the copied-content test) fold first-writer-wins too, but there the text is identical, so I did not count that as misattribution.

F2 — LOW (conditional): the Codex batch parser carries a model across a session boundary.

  • Where: src/brainlayer/ingest/codex.py:153–156. A later session_meta resets session_id but not model, which was set at :164 by the earlier turn_context.
  • Repro (test_codex_session_boundary_…): session_meta S1, then turn_context model=X, then a reply, then session_meta S2, then a reply. The S2 reply is stored with session_id S2, model X.
  • Claim limit: I have not verified that real Codex rollouts (fork or resume) contain a second session_meta; no real JSONL was read. The parser itself treats one as a session boundary for session_id.

F3 — LOW: Codex model_provider is stored unvalidated on every chunk, and an absent one is inferred.

Not a defect (checked):

Precise claim limits / UNRUN

Risks

  • Strict xfails turn a future fix into a red test until the marker is removed. That is deliberate, so a fix cannot land silently.
  • The test file depends on today's dedupe constants (SIMHASH_THRESHOLD, the stopwords). The F1 precondition asserts the SimHash distance, so a constants change fails loudly instead of silently passing.

Crash sweep / feature batch

Not applicable: test-only, with no sweep, launchd, CI or pyproject touch.

Owner handoff → @orc (BrainLayer lead snoozed)

Implementer/harness: Claude Code cloud worker (Anthropic Claude, effort high, fast mode off). No subagents, no extra sessions, no paid API calls. A cost meter was not available in-session. Work ran from about 04:53Z to 05:15Z, inside the 07:00Z bound.

Loop-record: implementer=anthropic/claude-code-cloud/high; reviewer=none; rounds=0; defects=3 (F1 medium, F2 low-conditional, F3 low)


Generated by Claude Code

Note

Add tests for author-model contract on direct and arbitrated writers

  • Adds test_author_model_writer_paths.py covering both the direct and arbitrated writer paths, plus the queue-drain fallback, using shared fixtures for Claude and Codex entries
  • Tests that sender and model metadata persist consistently across writer paths, that replayed flushes and queue files increment the seen count without changing stored authorship, and that model metadata does not leak between interleaved sessions or batches
  • Includes adversarial cases: model names at the length boundary, control characters, invalid types, homoglyphs, secret-shaped strings, and malformed Codex model-provider values
  • Adds expected-failure regressions for near-duplicate merges attributing another author's text to the first model, and for Codex sessions inheriting a prior session's model

Macroscope summarized 8305398.

…ex chunks (#655)

Claude Code transcripts record message.model on every assistant turn, but
_extract_entry_metadata dropped it, so realtime-watcher and batch-index rows
could not answer 'which model wrote this?'. Read it into chunk metadata
['model'] for assistant entries only, verbatim, and leave it unknown when it
is absent, non-string, malformed, or Claude Code's '<synthetic>' placeholder.

The Codex parser, cited as the working path, actually stored model_provider
('openai') under 'model' and kept the first value for the whole session.
It now takes each turn_context model as the author of the following turn,
records model_provider separately, and stops attributing user and tool
chunks to a model.

Authoring provenance stays distinct from enrichment_model/enrichment_backend.

Refs #655

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3NoZDshC2wAT1TQqUTtFa
…rated writers (Refs #655)

Peer proof for PR #1639. Runs the same synthetic mixed Claude/Codex
transcripts through the direct watcher writer and the arbitrated
queue -> drain_once writer and compares what each persisted, including
busy-spill, replayed flushes and replayed queue files, interleaved
sessions, adversarial model names and secret-shaped model strings.

Three findings are pinned as strict xfails (no production change here):
F1 near-duplicate merges fold another author's text into a row that
keeps the first author's metadata.model, on both writers; F2 the Codex
batch parser carries a model across a later session_meta; F3 Codex
model_provider is stored unvalidated and inferred as 'openai' when absent.

Refs #655

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3NoZHJ6i9UuC5RngPppFG
@coderabbitai

coderabbitai Bot commented Nov 5, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@deepsource-io

deepsource-io Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

DeepSource Code Review

We reviewed changes in a03bca7...8305398 on this pull request. Below is the summary for the review, and you can see the individual issues we found as inline review comments.

See full review on DeepSource ↗

PR Report Card

Overall Grade   Security  

Reliability  

Complexity  

Hygiene  

Code Review Summary

Analyzer Status Updated (UTC) Details
Python Nov 5, 2026 5:11a.m. Review ↗
Swift Nov 5, 2026 5:11a.m. Review ↗
JavaScript Nov 5, 2026 5:11a.m. Review ↗
Shell Nov 5, 2026 5:11a.m. Review ↗
Secrets Nov 5, 2026 5:11a.m. Review ↗

Important

AI Review is run only on demand for your team. We're only showing results of static analysis review right now. To trigger AI Review, comment @deepsourcebot review on this thread.

def _codex_path(tmp_path: Path, model_line: dict | None = None) -> Path:
path = tmp_path / "home" / ".codex" / "sessions" / "2026" / "11" / "05" / "rollout-fixture.jsonl"
path.parent.mkdir(parents=True, exist_ok=True)
lines = [{"type": "session_meta", "payload": {"id": "cx", "cwd": "/Users/fixture/Gits/proj"}}]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Appending to list immediately following its definition


Instead of appending items to a list just after creation, it should be refactored to add those items into the list definition itself.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown

BrainLayer ratchet

Every Value below was measured by this run. A row this machine cannot measure says n/a — <reason> instead of a number; baselines in Notes name their own machine, method and date and were not measured here.

Hosted success gives no verdict on mapped bytes, search p50/p95, idle CPU, hotlane backlog CPU, or fallback replay debt.

Row Status Value (measured by this run) Method Notes
commit provenance 🟢 GREEN measured 8305398bab7e == PR head · checkout 8305398bab7e commit graph + live PR head · in-process · runner Which commit this whole table is about. On a pull_request event the checkout is GitHub's synthetic merge ref, whose sha is not on the PR — #759's table printed 13fa724278bf while that PR's head was 4632f979 — so this row names the PR-head parent instead, the sha a reviewer can actually see. The comparison sha is read live from repos/{owner}/{repo}/pulls/{n} when the table is collected, not taken from the event payload, because the payload cannot know the run has been overtaken. Residual window, stated rather than papered over: a push landing between that read and the comment being posted is not caught here — the run for that push refreshes the table.
baseline attestation 🟢 GREEN baseline f421d1a7c5e6 matches the main attestation (run 41876229562 · main d6b82f04f6ba · 2026-11-04T17:18:27Z) main attestation artifact via Actions API · in-process · runner What every comparison is measured AGAINST, and who says so. The baseline fields of tests/fixtures/sprint_gate/corpus.json (queries, latency_baseline_ms, thresholds) are compared to the ratchet-attestation artifact of the latest successful push or (no-input) workflow_dispatch run of ratchet-attest.yml on main, fetched through the Actions API — a PR run cannot write to another run's artifacts. A field that differs is RED unless that main run measured the new value. The calibrated socket collector can license p50/p95; every absent measured path stays locked, so missing collection never passes as permission for a hand edit. Boundary: the comparator is this PR's checkout of ci_ratchet_table.py, diff-reviewable, not tamper-proof.
provenance 🟢 GREEN stamped 8305398bab7e == HEAD, tree clean wheel stamp · in-process · runner Sha half of #749 keg-mode provenance: a keg built from this wheel can answer __build_sha__. The helper-age and served-process predicates need a running BrainBar and are measured only by scripts/sprint_gate.py on an installed Mac. The sha here is the checkout's — the explicit PR head in ratchet.yml — because that is what publish.yml stamps at release time; the PR-head sha this table describes is the one in commit provenance above.
fallback replay debt ⚪ n/a n/a — no fallback queue on this machine: the pending memories live in ~/Gits/*/docs.local/decisions, and docs.local/ is gitignored, so a runner checkout has no copy of them to count docs.local walk · machine with the fallback queue local-only; not enforced by hosted CI. intended_brain_store: true with no chunk_id means a memory reached disk and never reached the DB, so it answers no brain_search. Budget: 0. Any pending or unparseable file is a finding, never a band -- 122 of these sat from 2026-06-28 to 2026-09-05 because nothing counted them where a reader would look. Measured by walking the tree, so it is only ever measured on a machine that HAS the tree.
pre-push: delete-only push runs no suite 🟢 GREEN 0 suite starts · exit 0 real checkout hook · delete-only stdin · sentinel suite runner Bug SHA: 44ecbf5ba6f1ada3b965797559eee90c94b45595 (RED replay, 2026-10-06). Fix: #1084; measured at 8305398bab7e5e2d57d23d8c29057058c4d3b1dc. Ceiling: 0 suite starts; hook must exit 0 and announce the delete-only skip.
mapped bytes ⚪ n/a n/a — no BrainBar daemon at /tmp/brainbar.sock: this row needs the daemon, its hybrid helper and the indexed corpus running together, and no GitHub-hosted runner has them (macOS included) — only a self-hosted Darwin/arm64 runner on an installed Mac would socket · installed Mac local-only; not enforced by hosted CI. Baseline 26.2 GB — installed Mac, socket, 2026-09-03, after R2 drained 15,070 → 0. Up from 16.8 GB because the drain left more vectors mapped under the same cap: the change is the drain, not a leak. Not measured by this run.
search p50/p95 ⚪ n/a n/a — no BrainBar daemon at /tmp/brainbar.sock: this row needs the daemon, its hybrid helper and the indexed corpus running together, and no GitHub-hosted runner has them (macOS included) — only a self-hosted Darwin/arm64 runner on an installed Mac would socket · installed Mac local-only; not enforced by hosted CI. Margin p50: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Margin p95: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Calibrated on MacBook-Pro.local at 2026-09-01T08:42:22Z under active_sprint_load (tests/fixtures/sprint_gate/corpus.json). Not measured by this run.
idle CPU ⚪ n/a n/a — no BrainBar daemon at /tmp/brainbar.sock: this row needs the daemon, its hybrid helper and the indexed corpus running together, and no GitHub-hosted runner has them (macOS included) — only a self-hosted Darwin/arm64 runner on an installed Mac would ps sampling · installed Mac local-only; not enforced by hosted CI. Ceiling: average CPU < 30% over a 60 s window (resource_budget in scripts/sprint_gate.py), ratified and kept as a hard budget. Margin daemon: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Margin helper: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Margin watcher: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Needs the BrainBar daemon, helper and watcher actually running. Not measured by this run.
hotlane backlog CPU ⚪ n/a n/a — live hotlane CPU evidence requires an installed Mac CPU/wall for backlog-encoding cycles · rolling 600 s · installed Mac local-only; not enforced by hosted CI. Default budget 60% of one core; BRAINLAYER_HOTLANE_CPU_BUDGET_PERCENT overrides. At least 30s of active backlog cycles required; yielded time excluded. Measures installed code.
signature_valid ⚪ n/a n/a — the macOS signature-parity job is trigger-gated and did not run on this PR: it touches no release or signing path (pyproject.toml, scripts/release-*, scripts/brainlayer-version-check.sh, publish.yml, ratchet.yml) and carries no ratchet:signatures label — a GitHub macOS runner bills at ~10× Linux minutes and rebuilds the keg venv from source codesign · installed keg scripts/release-verify-signatures.sh <keg> codesign-verifies every *.so/*.dylib under libexec/venv. The macOS parity job installs the published tap formula (etanhey/layers/brainlayer), so this row measures the release path — formula, published sdist and Homebrew's relocation — and not this PR's tree. Release-time baseline for the same keg on a different machine: 442 valid / 0 invalid — installed Mac (M4 Max), brew --prefix brainlayer 1.5.11, 2026-09-03.
BrainBar renders no enrichment status 🟢 GREEN 19 actual-view captures; active pending-store/replay retained macOS DEBUG binary · AppKit actual views · Vision OCR + icon RGBA comparison · private synthetic state Missing render capability/evidence is RED. Historical data is retained; this is source-build proof, not installed-app proof.
Watcher heartbeat freshness and recovery ⚪ n/a n/a — macOS native probe trigger-gated: no affected path or ratchet:heartbeat label private Python producer -> compiled Swift consumer Preserves genuine silence; independent agent stores cannot revive the watcher. Not a load or installed-UI measurement.
maintenance quiesce: services stay down under a reviving watchdog ⚪ n/a n/a — macOS replay trigger-gated: no affected path or ratchet:quiesce label real launchd · GitHub macOS runner · real maintenance + fleet-watchdog Fix: #1076 (bc6d690f4331387485678d0247e19e14e6ef8d97). Ceiling: 0 revivals; supervisor held, both resumed, no hold marker.
no cloud model call reachable anywhere 🟢 GREEN 520 source files; 460 installed imports; 3 native tests; wheel e5a6679def93; 0 candidate attempts immutable installed wheel (default + dev) + AST source + actual native BrainBar Synthetic private fixtures; SDK absence gated on merged R10b. Exact-head candidate proof, not deployment.

🟢 GREEN measured, within budget · 🔴 RED measured, out of budget — a finding to clear before merge · ⚪ n/a not measurable on this machine, never guessed.

No RED rows.

Measured on Linux/x86_64 · measured 8305398bab7e · PR head 8305398bab7e · checkout 8305398bab7e · run · updated 2026-11-05 06:26:15 UTC

@EtanHey
EtanHey force-pushed the claude/orc-final-credit-batch-z2uwd0 branch from a03bca7 to 1dd44ef Compare November 5, 2026 07:28

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants