You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Distiller elision markers are accepted as verbatim evidence quotes #229
Version: 0.1.39 (main at 031523d); first seen on 0.1.32.
What happens.src/distill.js inserts its own elision markers into the distilled trace (line 33: [... N chars elided ...], line 179: [... middle of session elided: ~N tokens. Open the raw transcript below if a claim needs it ...]). sanitizeEvidence (src/analyze.js) only requires that a quote appear in the distilled trace it cites, so a marker passes as a verbatim quote.
Observed impact. In a user-scope run (976 sessions) the analysis model cited these markers as negative evidence for "context grew too large / output truncated", and the synthesis step turned it into a proposed add edit ("checkpoint when outputs are repeatedly truncated or session history is large/elided"). All three quotes behind that edit were distiller markers, so the edit's evidence floor (3 sessions) was met by backpass's own trimming rather than by anything the agent did. The edit looked well-evidenced in the review surface.
Why it matters. It violates the spirit of "quotes must be found in the trace they cite": the text is in the distilled trace but not in the session. It also feeds back into proposals about context handling, which is exactly the area where distillation changes what the model sees.
Suggested fix (not a text-shape classifier in buildProposal). Reject, in sanitizeEvidence, any quote whose matched span overlaps a region the distiller synthesized. The simplest structural way is for distill.js to record the byte ranges of the markers it inserts, or to emit them in a form that cannot match a quote check, so the check is "found in session text" rather than "found in distilled text".
Speaking as Kun's firstmate: thanks @loxodont, this is a clear and well-evidenced report.
I confirmed it on latest main (031523d). src/distill.js still inserts [... N chars elided ...] and [... middle of session elided ...] markers, and sanitizeEvidence in src/analyze.js only checks that a quote appears in the distilled trace, so a quote made entirely of a distiller marker passes. That lets backpass's own trimming stand in for session evidence, which cuts against VISION's "Evidence is the only currency" rule and its "harder to fabricate" alignment test. Your structural direction (reject quotes overlapping distiller-synthesized spans, rather than a text-shape classifier) fits "enforced in code" and "fix at the root".
Labelling this ready-for-pr. A PR should include a test that fails when the overlap check is removed.
Speaking as Kun's firstmate: re-checked against latest main (c8d60c2). This is still real there, and the open PR #237 by @WTF-Am-ID is the fix in progress. That PR is CLEAN with green CI and live evidence, and is waiting on Kun's merge decision (it changes default behavior), not on anyone here. Keeping ready-for-pr and tracking it through #237.
Activity
kunchenguid commentedon Oct 25, 2026
Speaking as Kun's firstmate: thanks @loxodont, this is a clear and well-evidenced report.
I confirmed it on latest main (031523d).
src/distill.jsstill inserts[... N chars elided ...]and[... middle of session elided ...]markers, andsanitizeEvidenceinsrc/analyze.jsonly checks that a quote appears in the distilled trace, so a quote made entirely of a distiller marker passes. That lets backpass's own trimming stand in for session evidence, which cuts against VISION's "Evidence is the only currency" rule and its "harder to fabricate" alignment test. Your structural direction (reject quotes overlapping distiller-synthesized spans, rather than a text-shape classifier) fits "enforced in code" and "fix at the root".Labelling this
ready-for-pr. A PR should include a test that fails when the overlap check is removed.kunchenguid commentedon Nov 8, 2026
Speaking as Kun's firstmate: re-checked against latest main (
c8d60c2). This is still real there, and the open PR #237 by @WTF-Am-ID is the fix in progress. That PR is CLEAN with green CI and live evidence, and is waiting on Kun's merge decision (it changes default behavior), not on anyone here. Keepingready-for-prand tracking it through #237.