Fix quadratic blowup in newmm's BFS ambiguity resolution

[kamthorn]

What do these changes do

Fixes a quadratic-time blowup in word_tokenize() (engine="newmm", the
default) for text that stays ambiguous over a long stretch. It rebuilds a
BFS path once, from a predecessor map, instead of copying it at every BFS
step, and tracks the position queue's membership in a set instead of
scanning a list.

Related to #893 and #326, both closed: those fixed the exponential/hang
case in this same BFS ambiguity resolution, but left the quadratic-time
case (still slow, no longer hanging) described below.

What was wrong

_onecut() builds a graph of possible word boundaries while a position is
ambiguous (more than one dictionary match leads forward from it), and
resolves it with BFS once only one candidate boundary is left. Two things
compound:

1. _MAX_GRAPH_SIZE = 50 (added for Report: newmm bug #893, "newmm BFS path explosion")
   only limits how many edges are added from a single position in one
   pass (the break inside the for word in custom_dict.prefixes(...)
   loop). It does not bound how large the graph grows across many
   positions while the text stays ambiguous. For text with many
   overlapping dictionary matches over a long stretch, the graph keeps
   accumulating one batch at a time and can reach the size of the whole
   stretch before the first resolution.
2. _bfs_paths_graph() copied the whole path-so-far at every BFS step:
   queue.append((pos, path + [pos])). path + [pos] builds a new list,
   copying everything already in path. Once the graph above has grown
   large, resolving it this way costs O(V x path length) instead of the
   O(V + E) the comment above the function claims.

Together, tokenizing text with a long run of ambiguous, overlapping
dictionary matches — e.g. "กรรมกร" * 8000 (48,000 characters; "กรรมกร"
has several dictionary substrings: กร, รม, มก, กรรม, กรรมกร) — made
word_tokenize() quadratic in text length:

Repetitions	Characters	Time (before, median of 3)
2,000	12,000	0.07 s
4,000	24,000	0.27 s
8,000	48,000	1.33 s

Doubling the input roughly quadruples the time. Traced directly: for
"กรรมกร" * 3000 (18,000 characters), the graph grows to 18,022 edges
across 17,998 nodes before it resolves for the first (and only) time in
that text.

Any service that tokenizes text it does not control (a search box, a
chat message, an uploaded document) is exposed to this: input of a few
tens of thousands of characters, not an unusually large size, can cost
seconds of CPU per request and get worse for longer input, which is a
low-effort denial-of-service vector against the default tokenizer.

newmm-safe (safe_mode=True) already limits the damage as a side
effect of chunking text into ~140-character windows before tokenizing
each one, which keeps the graph from growing past the chunk. But
safe_mode is off by default, and its docstring only mentions using it
"for very long texts (hundreds of kilobytes or more)" — nothing about
this being a mitigation for persistently ambiguous text well under that
size.

How this fixes it

_bfs_paths_graph() (renamed _bfs_shortest_path()) now records one
predecessor per visited node during the BFS traversal (O(1) per step,
same as before) and rebuilds the path by walking predecessors backward
once, only when the goal is reached, instead of on every step:

if pos == goal:
    predecessor[pos] = vertex
    path = [goal]
    while path[-1] != start:
        path.append(predecessor[path[-1]])
    path.reverse()
    return path

The traversal order and the path returned are unchanged — same BFS,
same first-reached (shortest) path — only how the path is built changes.
The function now returns list[int] directly rather than a
Generator[list[int], None, None], since the only caller did
next(_bfs_paths_graph(...)) to get exactly one path; the queue is a
collections.deque (popleft(), O(1)) instead of a plain list
(pop(0), O(n)).

Separately, end_pos_candidate not in pos_list scanned the whole
position list (used as a heap) on every candidate. A pos_set mirrors
pos_list's contents (added and discarded alongside every push/pop) so
this check is O(1) instead of O(n). Tested this in isolation: on its own
it made little difference next to the path-copying fix above, but it is
still a correctness-preserving O(n) -> O(1) change worth keeping.

_MAX_GRAPH_SIZE's deeper limitation (it does not bound total graph size
across positions) is not addressed here — doing that safely needs
picking a resolution strategy for a graph that is deliberately cut off
mid-ambiguity, which risks changing tokenization output for the text
that triggers it and needs care and testing on its own. This change
brings the cost of resolving whatever graph does accumulate back down to
what the existing cutoff assumed it would be.

Verification

Compared against the unmodified tokenizer (newmm.py from dev) on:

• Every word in thai_words() (62,106), through _onecut() directly
• 20,000 pairwise concatenations of dictionary words
• 51 known edge cases: mixed Thai/Latin/numeric text, the existing
  DANGER_TEXT_* / LONG_TEXT samples, empty string, every single Thai
  consonant
• Both segment() and segment(..., safe_mode=True), not just _onecut()

Check	Result
_onecut() output differs from unmodified tokenizer	0 / 82,157 samples
segment() / safe_mode=True output differs	0 / 14 samples
"".join(word_tokenize(text)) != text	0

Performance, same adversarial pattern, before vs. after (median of 3 runs
at the smaller sizes; single measured run at 96,000/192,000 — both take
long enough on the unfixed tokenizer that the quadratic trend is already
unambiguous):

Characters	Before	After	Speedup
12,000	0.07 s	0.02 s	4.6x
24,000	0.27 s	0.04 s	7.4x
48,000	1.33 s	0.09 s	14.9x
96,000	6.97 s	0.25 s	28.0x
192,000	37.62 s	0.80 s	47.1x

New test: TokenizeTestCase.test_newmm_persistent_ambiguity_performance
in tests/core/test_tokenize.py, alongside the existing
test_newmm_ambiguous_performance (#893) regression test that this one
complements rather than replaces (that one already passed before this
fix; this one did not, at the sizes tested above).

Your checklist for this pull request

• Passed code styles and structures
• Passed code linting checks and unit test

Ruff passes and mypy reports no errors in newmm.py.
tests.core.test_tokenize (including the two new/existing performance
regression tests) and tests.core.test_tokenize_thread_safety pass — the
fix does not introduce shared mutable state (pos_set is a local
variable per _onecut() call, same lifetime as the graph/pos_list it
already had). Ruff's formatting flags on test_tokenize.py are
pre-existing on unmodified dev and unrelated to this change (confirmed
by running the same check against the unmodified file).

🤖 Generated with Claude Code

https://claude.ai/code/session_01MLAScAmHFQoCy5oDHoGoo1

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud