From 5f2226e87e23c873314d86b9a95deb4059367642 Mon Sep 17 00:00:00 2001 From: Adam Wright Date: Sat, 19 Sep 2026 04:24:09 +0000 Subject: [PATCH] Record what the source-list stripper does not defend against, and why The website session found that their client-side equivalent truncates mid-stream at a partially typed heading, and recovers only because it recomputes over accumulated text -- once the rest of the heading arrives it stops matching and the answer comes back. Ours cannot recover: it is a state machine, and once it commits to a heading it has already dropped the tokens. That asymmetry leaves one way to lose an answer here that does not exist there: a complete `## Sources` line in the middle of an answer with real prose after it. Measured over six real answers across the reactome and userguide prompts rather than argued about. The heading was `## Sources` every time, last every time, with nothing but list items following. Zero mid-answer occurrences. So the guard is not built. Defending it would mean holding the text after a heading until a list item confirmed the verdict, which is more state than an unobserved failure justifies -- and this file already carries the rule that when in doubt it should not match. What is recorded is the risk, the measurement, and what the fix would be if a mid-answer heading is ever seen. Comment only; no behaviour change, so this needs no deploy. Co-Authored-By: Claude Opus 5 --- src/util/sources_section.py | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/src/util/sources_section.py b/src/util/sources_section.py index 02d4df8..6436fc6 100644 --- a/src/util/sources_section.py +++ b/src/util/sources_section.py @@ -13,7 +13,20 @@ phrasings of a contract, it is five samples from an unconstrained generator. The prompts now pin the heading to `## Sources`. This is the second half: strip -it here so no caller has to pattern-match model output at all. The pattern below +it here so no caller has to pattern-match model output at all. + +**One risk is known, measured, and deliberately not defended against.** This +stripper is a state machine: once it commits to a heading it drops everything +after it, and unlike a function that recomputes over accumulated text it cannot +change its mind. So a *complete* `## Sources` line in the middle of an answer, +with real prose after it, would cost the reader that prose. + +Measured 2026-09-19 over six real answers spanning the reactome and userguide +prompts: the heading was `## Sources` every time, last every time, with nothing +but list items after it. Zero mid-answer occurrences. Guarding against it would +mean holding the text after a heading until a list item confirms the verdict -- +buildable, and more state than an unobserved failure justifies. If a mid-answer +heading is ever seen, that is the fix; it is recorded here rather than built. The pattern below is still tolerant, because the prompt is an instruction and not a guarantee, but it is bounded -- a heading or bold-only line, at most five words, naming sources -- rather than any line mentioning the word.