Conversation
romanize(..., engine="thai2rom" or "thai2rom_onnx") could return a
~100-character string of repeated characters instead of a valid
romanization, e.g. romanize("กรุงเทพฯ", engine="thai2rom") produced
"krungtheppaaaa...aaaa" (100 chars). Same for "ฯลฯ" and long
Pali/Sanskrit-derived compounds such as "ราษฎรบำรุง" and
"สตรีเศรษฐบุตรบำเพ็ญ".
Both engines decode greedily (argmax/topk(1)) with no repetition
penalty or n-gram blocking. When the attention mechanism gets
trapped on certain syllable transitions or out-of-vocabulary
sequences, it keeps re-attending to the same input position and
never emits <end>, so decoding runs to the hard _maxlength=100 cap
and returns the truncated, meaningless repeat.
Add pythainlp/transliterate/_repetition.py with
find_trailing_repeat_period(), which detects a short cycle (1-12
target characters) repeating 3+ times in a row at the end of the
tokens generated so far. Both Seq2Seq.forward() (thai2rom.py) and
Seq2Seq_ONNX.run() (thai2rom_onnx.py) now check for this after each
inference step and, if found, stop decoding and truncate the output
to keep only the cycle's first occurrence rather than running to
max_len.
Adds regression tests using the reproduction cases above to
tests/noauto_torch/testn_transliterate_torch.py and
tests/noauto_onnx/testn_transliterate_onnx.py, plus unit tests for
find_trailing_repeat_period() in tests/core/test_transliterate.py.
Closes PyThaiNLP#1403
While fixing PyThaiNLP#1403 (thai2rom's romanize() looping to a 100-character repeat under greedy decoding), pythainlp/transliterate/thaig2p.py was found to share the exact same vulnerable Seq2Seq.forward() decode loop: argmax/topk(1) greedy decoding, no repetition penalty, and no guard beyond checking for the <end> token. Reproduced the same failure mode on thaig2p.transliterate(): transliterate("สตรีเศรษฐบุตรบำเพ็ญ") # -> 's a ˨˩ . t r iː ˧ . s u ˧ . s u ˧ . s u ˧ ...' (100 chars) transliterate("เอ็มเอฟซีบัญชีเพื่อการชำระค่ารับซื้อคืน") # -> 'ʔ e m ˧ . b r i ˨˩ . b aː ˧ . b aː ˧ . b aː ˧ ...' (100 chars) Reuses find_trailing_repeat_period() from pythainlp/transliterate/_repetition.py (added on the thai2rom fix branch) in thaig2p.py's Seq2Seq.forward(), the same way it was wired into thai2rom.py and thai2rom_onnx.py: after each inference step, stop decoding and truncate to the cycle's first occurrence as soon as a short cycle repeats 3+ times, instead of running to max_len. Adds a regression test with the reproduction cases above to tests/noauto_torch/testn_transliterate_torch.py. Note: this branch is stacked on fix/thai2rom-repetition-loop (PyThaiNLP#1500) to reuse _repetition.py; this diff will shrink to just the thaig2p.py change once that PR merges. Refs PyThaiNLP#1403 Refs PyThaiNLP#1500
bact
requested changes
Sep 16, 2026
| def test_single_char_cycle(self): | ||
| # e.g. the "aaaa..." tail seen for "กรุงเทพฯ" | ||
| self.assertEqual(find_trailing_repeat_period([9, 1, 1, 1]), 1) | ||
| self.assertEqual(find_trailing_repeat_period([1, 1]), None) |
Member
There was a problem hiding this comment.
Suggested change
| self.assertEqual(find_trailing_repeat_period([1, 1]), None) | |
| self.assertIsNone(find_trailing_repeat_period([1, 1])) |
bact requested this change on PR PyThaiNLP#1500: prefer assertIsNone() over assertEqual(x, None) for the unittest idiom.
…on-loop Pull in the assertIsNone fix requested by bact on PR PyThaiNLP#1500 so PR PyThaiNLP#1501 (stacked on top) picks it up too.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #1500 / #1403.
While fixing #1403 (
thai2rom'sromanize()looping to a 100-characterrepeat under greedy decoding),
pythainlp/transliterate/thaig2p.pywasfound to share the exact same vulnerable
Seq2Seq.forward()decode loop:argmax/
topk(1)greedy decoding, no repetition penalty, and no guard beyondchecking for the
<end>token.Reproduced the same failure mode on
thaig2p.transliterate():Fix: reuse
find_trailing_repeat_period()frompythainlp/transliterate/_repetition.py(added on thethai2romfixbranch, #1500) in
thaig2p.py'sSeq2Seq.forward(), the same way it waswired into
thai2rom.pyandthai2rom_onnx.py: after each inference step,stop decoding and truncate to the cycle's first occurrence as soon as a
short cycle repeats 3+ times, instead of running to
max_len.After the fix:
สตรีเศรษฐบุตรบำเพ็ญ->s a ˨˩ . t r iː ˧ . s u(24 chars)เอ็มเอฟซีบัญชีเพื่อการชำระค่ารับซื้อคืน->ʔ e m ˧ . b r i ˨˩ . b aː ˧(27 chars)บัญชีเพื่อการชำระค่าขายคืนหน่วยลงทุน->b a n ˧ . t͡ɕʰ iː ˧ . kʰ aː n ˧ . m(35 chars)Normal words are unaffected (
สวัสดี,แมว,กรุงเทพฯunchanged).Note on branch base: this branch is stacked on
fix/thai2rom-repetition-loop(#1500) to reuse
_repetition.pywithout duplicating it. Once #1500 merges,this PR's diff will shrink to just the
thaig2p.pyand test changes shownbelow; please review/merge #1500 first, or let me know if you'd prefer this
rebased to duplicate the small helper instead of depending on the other PR.
Test plan
original
thai2romreproduction strings) totests/noauto_torch/testn_transliterate_torch.pyruff checkpasses on all changed filesbuild_tools/analysis/type-analyzer.pyreports 0 functions withincomplete type hints repo-wide (unchanged from before this PR)
tests/noauto_torch/testn_transliterate_torch.pypasses (except thepre-existing, unrelated
thaig2p_v2failure in this environment dueto a missing
sentencepiecedependency)CHANGELOG.mdunder[Unreleased]/### Fixed🤖 Generated with Claude Code