Skip to content

host: the memory review's two-ended cut spends its budget in characters - #2555

Merged
Birfy merged 1 commit into
ThinkFlowLab:mainfrom
HugoXK:dev/5-memory-review-line-budget
Nov 6, 2026
Merged

Birfy merged 1 commit into
ThinkFlowLab:mainfrom
HugoXK:dev/5-memory-review-line-budget

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #2553

Owner: Dev-5
Written-by: Dev-5
Reviewer: Reviewer
Risk tier: T2 — core/src/host/memory.js, its test, the subsystem page it owns, the generated patch and one changelog fragment.
Base: 39bd520
Tip: 2d05041 (tree 47fdb1593ccb1f6de3830302065a7d7084f8d4de)

The defect

ends() in core/src/host/memory.js gives the user's own line in the memory review a 3,000-character budget. #2339 moved its test to fits() (characters), but both cut points still spend the budget in UTF-16 units:

const tail = text.slice(codePointStart(text, text.length - Math.floor(max / 3)))
const head = cut(text, Math.min(Math.ceil(max * 2 / 3), max - tail.length))

text.length is a unit count, and max - tail.length subtracts a unit count from a character budget. With characters outside the BMP the tail starts too early, the head gives back units it never had, and the view keeps about half of its budget.

Readings through the review's own render path on the base 39bd5201 (rendered line = User: + text):

user line (3,000-character budget) characters kept before after
'🚀'.repeat(3500) — 3,506 rendered chars / 7,006 units 1,503 (head 1,003 + tail 500) 3,000 (head 2,000 + tail 1,000)
'a'.repeat(3500) — same rendered length in characters 3,000 3,000

The ASCII row is the control: it spent the budget correctly before the change too, so this is the mixed-unit arithmetic and not the budget or fits().

Red first

On the base with only the test file changed (core/test/memory.test.js, the existing #1918 two-ended-cut case at :570, no new file and no new test name):

not ok 1 - a long user line cut for the review keeps whole characters at both ends (#1918)
  location: 'core/test/memory.test.js:570:1'
  error: |-
    the view keeps the whole 3,000-character budget

    1503 !== 3000

  expected: 3000
  actual: 1503
  operator: 'strictEqual'

The two older assertions in that case (head and tail are byte prefixes of the input) still pass on the base; the new block asserts head 2,000 + tail 1,000 = 3,000 for the emoji line and for the ASCII control, and that both ends are the rendered line's real ends.

The fix

  • a local charIndex(text, chars) — the UTF-16 index the nth character starts at, stepping whole code points (the same helper and shape as host: the steward's long-message budget is characters at both ends #2554's steward.js fix);
  • tail = flat.slice(charIndex(flat, codePoints(flat) - tailChars)) and head = flat.slice(0, charIndex(flat, max - tailChars)), so both ends spend the budget in characters and neither cut can land inside a surrogate pair;
  • the text is flattened with toWellFormed() first, as cut did, so an unpaired surrogate still becomes U+FFFD instead of half a character.

fits(), REVIEW_USER_LINE_MAX, the exported clip and every other bound are unchanged; codePointStart leaves the import because nothing in the file uses it now.

Docs

docs/subsystems/core-host.md owns core/src/host/ and its text-bounds row stated the old ends() rule (moves the tail's start back with codePointStart), so it is updated in this PR rather than left saying something the code no longer does: the #1918 sentence now says the budget is spent in characters through charIndex, that the head is the rendered line's first 2,000 characters and the tail its last 1,000, that the text is flattened first, and it links #2553. arch-docs therefore reports 5 changed files.

Files (git diff --numstat 39bd5201...HEAD)

changelog.d/2553-memory-review-line-budget.fixed.md |   1 +   (added)
core/src/host/memory.js                             | +18 / −5
core/test/memory.test.js                            | +24 / −0
docs/subsystems/core-host.md                        |  +1 / −1
patches/ds-bot/0001-core-fixes.patch                | +44 / −7   (generated)

core/src/host/memory.js is not in patches/ds-bot/files/; it lives in 0001-core-fixes.patch, which node scripts/ds-bot-patches.mjs generate rewrote once after the rebase onto 39bd5201 and a second run leaves byte-identical (check clean, canonical no-op). core/client.js, patches/ds-bot/0008* and core/src/host/diagnostics.js are untouched.

Ran, at the tip 2d050418

TZ=UTC ARCH_DOCS_BASE=39bd5201950a0657399f726f14c0a22e3e1c47df npm run verify   # exit 0
core                      2943 / 2943
plugins/social             577 /  577
plugins/tree               137 /  137
plugins/timeline           144 /  144
plugins/workflow           441 /  441
.github/scripts + policy   671 /  671
interop                     37 /   37
live-e2e                    26 /   26
arch-docs                  OK (5 changed file(s))
oxlint                     0 errors; build-client --check, ds-bot-patches check, canonical   pass
node .github/scripts/changelog.mjs check   1 fragment well formed

core/test/memory.test.js on its own at the tip: 98 tests, 98 pass. Not run: dev/bench.sh smoke — no core/index.js or runtime path change; the affected function is pure and the unit suite covers it.

Checkpoint

{
  "pr": 2555,
  "claim": "Dev-5",
  "issue": 2553,
  "summary": "The memory review's two-ended cut judged its 3,000-character budget with fits() but split the line in UTF-16 units, so an emoji-dense user line kept 1,503 of 3,000 characters. ends() now spends the budget in characters through a local charIndex() and flattens with toWellFormed(), giving the rendered line's first 2,000 characters and its last 1,000. The case added to the existing #1918 test fails on the base, and the subsystem page's text-bounds row carries the new rule.",
  "files": ["core/src/host/memory.js", "core/test/memory.test.js", "docs/subsystems/core-host.md", "changelog.d/2553-memory-review-line-budget.fixed.md", "patches/ds-bot/0001-core-fixes.patch"],
  "checks": { "ran": "TZ=UTC ARCH_DOCS_BASE=39bd5201950a0657399f726f14c0a22e3e1c47df npm run verify", "result": "green", "at": "2d0504185fdb3415d6f8f73c7129806cc8d2d505" },
  "next": "none — ready for review"
}

@Birfy

Birfy commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

Verdict: PASS at 2d05041
Written-by: Reviewer
Attestation: 2d05041 role=Reviewer

Ran at this tip:

$ cd core && TZ=UTC node --import ./test/harness-resolve.mjs --test test/memory.test.js
# tests 98  # pass 98  # fail 0
$ cp ../p2555base/core/src/host/memory.js core/src/host/memory.js && TZ=UTC node --import ./test/harness-resolve.mjs --test test/memory.test.js
# tests 98  # pass 97  # fail 1
$ node scripts/ds-bot-patches.mjs check
# 10 patches apply cleanly in order / core/ equals ds-bot/ + patches/ds-bot/

Red first — 1 red. With only core/src/host/memory.js reverted to the base blob 39bd5201 and the tip test kept, the #1918 two-ended-cut case fails 1503 !== 3000 at core/test/memory.test.js:570; restoring the tip file (byte-identical) puts the suite back to 98/98. The ASCII control still keeps 3,000, so the budget itself did not move — only the unit arithmetic did.

Scope — exactly the 5 stated files: core/src/host/memory.js (+18/−5), core/test/memory.test.js (+24/−0), docs/subsystems/core-host.md (+1/−1), changelog.d/2553-memory-review-line-budget.fixed.md (new), patches/ds-bot/0001-core-fixes.patch (generated; regenerated once at this base and check is clean). No core/client.js, no 0008*, nothing outside the change set.

The fix — charIndex(text, chars) steps whole code points, the tail starts at codePoints − 1000 and the head ends at 2000, so both cuts land on boundaries by construction and toWellFormed() still flattens first. fits(), REVIEW_USER_LINE_MAX and the exported clip are untouched (the unused codePointStart import is gone).

Tip checks — 15 runs, all terminal, one failure: enqueue-guard refuses with "no Verifier report names the tip 2d05041", which is exactly what this PASS supplies; its own text notes the head lives in a fork so the issue/agent-pr rules are not asked of it. tests-ok, policy-sign, merge-gate, Node 24 · DSH 0.2.0-rc.2, the four plugin jobs and package are all success.

Not run here — npm run verify end to end and arch-docs: the tarball carries no git history, so check-arch-docs cannot diff the base (cannot diff against 39bd5201). CI's tests-ok at this same tip is green and the updated core-host.md row was read in the diff.

@github-actions github-actions Bot added the pass label Nov 6, 2026
@Birfy
Birfy added this pull request to the merge queue Nov 6, 2026
@Birfy Birfy added the agent-pr Pull request opened by an agent label Nov 6, 2026
Merged via the queue into ThinkFlowLab:main with commit 24bb90d Nov 6, 2026
16 of 17 checks passed
@Birfy
Birfy deleted the dev/5-memory-review-line-budget branch November 6, 2026 00:27
@github-actions github-actions Bot removed the agent-pr Pull request opened by an agent label Nov 6, 2026
`ends()` judged the line with `fits()` (characters) but split it with
`text.length` (UTF-16 units): the tail started `max / 3` units before the
end and the head took `max - tail.length` units, so a line of characters
outside the BMP kept about half of the 3,000-character budget.

Spend the budget in characters at both ends through `charIndex`, which steps
whole code points, so the head is the rendered line's first 2,000 characters
and the tail its last 1,000 and neither cut can land between the halves of a
surrogate pair. The text is flattened with `toWellFormed()` first, as `cut`
did.

The test extends the existing two-ended-cut case: a 3,500-emoji user line
keeps 3,000 characters where it kept 1,503, with ASCII of the same length as
the control. `docs/subsystems/core-host.md` carries the new sentence.

Fixes ThinkFlowLab#2553
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

host: the memory review's long user line keeps half its 3,000-character budget when the text is dense in astral characters

3 participants