Strip a Markdown code fence before json.loads on model replies - #79
Open
rogermsc wants to merge 1 commit into
Open
Strip a Markdown code fence before json.loads on model replies#79rogermsc wants to merge 1 commit into
rogermsc wants to merge 1 commit into
Conversation
gpt_generate_multi_summary parses the model's reply with a bare
json.loads. Many current models wrap JSON in a ```json ... ``` fence by
default, even when the prompt asks for bare JSON. That reply is a correct
answer, but json.loads rejects it, the except branch fires, and the
caller loses the structure it asked for.
For the multi-summary path the cost is more than the warning. When it
returns no themes, updater.py files the whole batch as one session under
a constant fallback summary with an empty keyword list. Those two fields
are the basis of two later decisions -- search_sessions scores a session
on semantic_sim + keyword_alpha * s_topic_keywords, and
insert_pages_into_session picks which existing session a new batch joins
with the same shape -- so every batch that hits the fallback shares one
identical summary embedding and carries no keywords. It becomes both
harder to retrieve and liable to be merged by a similarity it did not
earn, and nothing fails while it happens.
Adds strip_code_fence() and applies it at all six bare json.loads sites
on model output, across all five copies of utils.py (memoryos-pypi,
memoryos-playground, memoryos-mcp, memoryos-chromadb, eval). Fixing one
copy would leave four shipped variants with the same defect.
Text that is not fenced is returned unchanged, so a deployment whose
model does not fence sees no behaviour change at all. The helper carries
six doctests:
python -m doctest memoryos-pypi/utils.py -v
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi — following up on the email exchange with Prof. Bai, who kindly pointed me
here to submit security-related fixes. This is the one of my three findings that
has a patch to submit; the other two are design trade-offs in a mechanism
working as intended, and I am not filing those as issues.
The problem
gpt_generate_multi_summaryparses the model's reply with a barejson.loads.Many current models wrap JSON in a
```json ... ```fence by default, evenwhen the prompt asks for bare JSON — the prompt here does ask ("Please directly
output the JSON array, without adding any other content"). Such a reply is a
correct answer, but
json.loadsrejects it, theexceptbranch fires, and thecaller loses the structure it asked for.
Why it is worth more than the warning line
When the multi-summary returns no themes,
updater.pyfiles the entire batch asone session under a constant fallback:
Those two fields are the basis of two later decisions:
search_sessionsscores a session onsemantic_sim_score + keyword_alpha * s_topic_keywordsinsert_pages_into_sessionpicks which existing session a new batch joins with the same shapeSo every batch that hits the fallback shares one identical summary embedding and
carries no keywords. It is both harder to retrieve and liable to be merged by a
similarity it did not earn — and a deployment can lose the topic structure of
its mid-term memory for a whole run without anything failing. The trigger is an
ordinary model default, not an adversarial input.
The change
Adds
strip_code_fence()and applies it at all six barejson.loadssiteson model output, across all five copies of
utils.py(memoryos-pypi,memoryos-playground,memoryos-mcp,memoryos-chromadb,eval). Fixing onecopy would leave four shipped variants with the same defect. The extra site in
memoryos-chromadbis the profile-analysis parse, which has the same exposure.Text that is not fenced is returned unchanged, so a deployment whose model
does not fence sees no behaviour change at all.
Checking it
The helper carries six doctests, so no test infrastructure is needed:
I verified before pushing: all five files compile, 30 doctests pass, the helper
is byte-identical across the five copies, and
eval/utils.py's CRLF lineendings are preserved so the diff stays at the changed lines.
Happy to split this per-directory, drop the
evalorchromadbcopies, orrename the helper if you would prefer any of those.