Conversation
romanize(..., engine="thai2rom" or "thai2rom_onnx") could return a
~100-character string of repeated characters instead of a valid
romanization, e.g. romanize("กรุงเทพฯ", engine="thai2rom") produced
"krungtheppaaaa...aaaa" (100 chars). Same for "ฯลฯ" and long
Pali/Sanskrit-derived compounds such as "ราษฎรบำรุง" and
"สตรีเศรษฐบุตรบำเพ็ญ".
Both engines decode greedily (argmax/topk(1)) with no repetition
penalty or n-gram blocking. When the attention mechanism gets
trapped on certain syllable transitions or out-of-vocabulary
sequences, it keeps re-attending to the same input position and
never emits <end>, so decoding runs to the hard _maxlength=100 cap
and returns the truncated, meaningless repeat.
Add pythainlp/transliterate/_repetition.py with
find_trailing_repeat_period(), which detects a short cycle (1-12
target characters) repeating 3+ times in a row at the end of the
tokens generated so far. Both Seq2Seq.forward() (thai2rom.py) and
Seq2Seq_ONNX.run() (thai2rom_onnx.py) now check for this after each
inference step and, if found, stop decoding and truncate the output
to keep only the cycle's first occurrence rather than running to
max_len.
Adds regression tests using the reproduction cases above to
tests/noauto_torch/testn_transliterate_torch.py and
tests/noauto_onnx/testn_transliterate_onnx.py, plus unit tests for
find_trailing_repeat_period() in tests/core/test_transliterate.py.
Closes PyThaiNLP#1403
6 tasks
bact
requested changes
Sep 16, 2026
| def test_single_char_cycle(self): | ||
| # e.g. the "aaaa..." tail seen for "กรุงเทพฯ" | ||
| self.assertEqual(find_trailing_repeat_period([9, 1, 1, 1]), 1) | ||
| self.assertEqual(find_trailing_repeat_period([1, 1]), None) |
Member
There was a problem hiding this comment.
Suggested change
| self.assertEqual(find_trailing_repeat_period([1, 1]), None) | |
| self.assertIsNone(find_trailing_repeat_period([1, 1])) |
bact requested this change on PR PyThaiNLP#1500: prefer assertIsNone() over assertEqual(x, None) for the unittest idiom.
|
kamthorn
added a commit
to kamthorn/pythainlp
that referenced
this pull request
Sep 17, 2026
…on-loop Pull in the assertIsNone fix requested by bact on PR PyThaiNLP#1500 so PR PyThaiNLP#1501 (stacked on top) picks it up too.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1403.
romanize(..., engine="thai2rom")andromanize(..., engine="thai2rom_onnx")could return a ~100-character string of repeated characters instead of a
valid romanization, e.g.:
Same for
ฯลฯand long Pali/Sanskrit-derived compounds such asราษฎรบำรุงandสตรีเศรษฐบุตรบำเพ็ญ(see the issue for full analysis).Root cause: both engines decode greedily (argmax/
topk(1)) with norepetition penalty or n-gram blocking. When the attention mechanism gets
trapped on certain syllable transitions or out-of-vocabulary sequences, it
keeps re-attending to the same input position and never emits
<end>, sodecoding runs to the hard
_maxlength = 100cap and returns the truncated,meaningless repeat.
Fix: add
pythainlp/transliterate/_repetition.pywithfind_trailing_repeat_period(), a small pure-Python helper (no torch/onnxdependency) that detects a short cycle (1-12 target characters) repeating 3+
times in a row at the end of the tokens generated so far. Both
Seq2Seq.forward()(thai2rom.py) andSeq2Seq_ONNX.run()(
thai2rom_onnx.py) now check for this after each inference step and, iffound, stop decoding and truncate the output to keep only the cycle's first
occurrence rather than running to
max_len.กรุงเทพฯ->krungtheppaฯลฯ->pailaราษฎรบำรุง->ratsadotbสตรีเศรษฐบุตรบำเพ็ญ->satrisetthabutbaNormal words are unaffected (verified
แมว->maeo,สวัสดี->sawatdi,etc. are unchanged).
pythainlp/transliterate/thaig2p.pyhas a near-identical decode loop andlikely the same failure mode, but it's out of scope for this PR/issue and
tracked separately.
Test plan
find_trailing_repeat_period()(branch coverage:no cycle, single-char cycle, multi-char cycle, period > default
max_period,min_repeatsthreshold) intests/core/test_transliterate.pythai2romandthai2rom_onnx, intests/noauto_torch/testn_transliterate_torch.pyand
tests/noauto_onnx/testn_transliterate_onnx.pyruff checkpasses on all changed filesbuild_tools/analysis/type-analyzer.pyreports 0 functions withincomplete type hints repo-wide (unchanged from before this PR)
tests.coresuite (209 tests) passesthe issue, on both the torch and ONNX engines
CHANGELOG.mdunder[Unreleased]/### Fixed🤖 Generated with Claude Code