Upstramming main into dev - #1513
Merged
Merged
Conversation
- Generate an SBOM for PyThaiNLP package and embed it inside the wheel, following PEP 770 guidelines - Upload the wheel and the SBOM to GitHub Releases too Signed-off-by: Arthit Suriyawongkul <arthit@gmail.com>
Signed-off-by: Arthit Suriyawongkul <arthit@gmail.com>
Signed-off-by: Arthit Suriyawongkul <arthit@gmail.com>
Bumps [budoux](https://github.com/google/budoux) from 0.7.0 to 0.9.0. - [Release notes](https://github.com/google/budoux/releases) - [Commits](google/budoux@v0.7.0...v0.9.0) --- updated-dependencies: - dependency-name: budoux dependency-version: 0.9.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [transformers](https://github.com/huggingface/transformers) from 5.14.1 to 5.15.0. - [Release notes](https://github.com/huggingface/transformers/releases) - [Commits](huggingface/transformers@v5.14.1...v5.15.0) --- updated-dependencies: - dependency-name: transformers dependency-version: 5.15.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.37.6 to 4.37.7. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v4.37.6...v4.37.7) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.37.7 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [EndBug/add-and-commit](https://github.com/endbug/add-and-commit) from 10.0.0 to 11.0.0. - [Release notes](https://github.com/endbug/add-and-commit/releases) - [Commits](EndBug/add-and-commit@290ea2c...645ecc0) --- updated-dependencies: - dependency-name: EndBug/add-and-commit dependency-version: 11.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Arthit Suriyawongkul <arthit@gmail.com>
…Bug/add-and-commit-11.0.0 Bump EndBug/add-and-commit from 10.0.0 to 11.0.0
Bump budoux from 0.7.0 to 0.9.0
…hub/codeql-action-4.37.7 Bump github/codeql-action from 4.37.6 to 4.37.7
….15.0 Bump transformers from 5.14.1 to 5.15.0
Bumps [epitran](https://github.com/dmort27/epitran) from 1.26.0 to 1.35.2. - [Release notes](https://github.com/dmort27/epitran/releases) - [Changelog](https://github.com/dmort27/epitran/blob/master/CHANGELOG.md) - [Commits](dmort27/epitran@v1.26.0...v1.35.2) --- updated-dependencies: - dependency-name: epitran dependency-version: 1.35.2 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bump epitran from 1.26.0 to 1.35.2
Bumps [transformers](https://github.com/huggingface/transformers) from 5.15.0 to 5.15.1. - [Release notes](https://github.com/huggingface/transformers/releases) - [Commits](huggingface/transformers@v5.15.0...v5.15.1) --- updated-dependencies: - dependency-name: transformers dependency-version: 5.15.1 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [EndBug/add-and-commit](https://github.com/endbug/add-and-commit) from 11.0.0 to 11.1.1. - [Release notes](https://github.com/endbug/add-and-commit/releases) - [Commits](EndBug/add-and-commit@645ecc0...cc9c08b) --- updated-dependencies: - dependency-name: EndBug/add-and-commit dependency-version: 11.1.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
…Bug/add-and-commit-11.1.1 Bump EndBug/add-and-commit from 11.0.0 to 11.1.1
….15.1 Bump transformers from 5.15.0 to 5.15.1
Bumps [transformers](https://github.com/huggingface/transformers) from 5.15.1 to 5.16.1. - [Release notes](https://github.com/huggingface/transformers/releases) - [Commits](huggingface/transformers@v5.15.1...v5.16.1) --- updated-dependencies: - dependency-name: transformers dependency-version: 5.16.1 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.37.7 to 4.37.9. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v4.37.7...v4.37.9) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.37.9 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
…hub/codeql-action-4.37.9 Bump github/codeql-action from 4.37.7 to 4.37.9
….16.1 Bump transformers from 5.15.1 to 5.16.1
Bumps [budoux](https://github.com/google/budoux) from 0.9.0 to 0.9.1. - [Release notes](https://github.com/google/budoux/releases) - [Commits](google/budoux@v0.9.0...v0.9.1) --- updated-dependencies: - dependency-name: budoux dependency-version: 0.9.1 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
romanize(..., engine="thai2rom" or "thai2rom_onnx") could return a
~100-character string of repeated characters instead of a valid
romanization, e.g. romanize("กรุงเทพฯ", engine="thai2rom") produced
"krungtheppaaaa...aaaa" (100 chars). Same for "ฯลฯ" and long
Pali/Sanskrit-derived compounds such as "ราษฎรบำรุง" and
"สตรีเศรษฐบุตรบำเพ็ญ".
Both engines decode greedily (argmax/topk(1)) with no repetition
penalty or n-gram blocking. When the attention mechanism gets
trapped on certain syllable transitions or out-of-vocabulary
sequences, it keeps re-attending to the same input position and
never emits <end>, so decoding runs to the hard _maxlength=100 cap
and returns the truncated, meaningless repeat.
Add pythainlp/transliterate/_repetition.py with
find_trailing_repeat_period(), which detects a short cycle (1-12
target characters) repeating 3+ times in a row at the end of the
tokens generated so far. Both Seq2Seq.forward() (thai2rom.py) and
Seq2Seq_ONNX.run() (thai2rom_onnx.py) now check for this after each
inference step and, if found, stop decoding and truncate the output
to keep only the cycle's first occurrence rather than running to
max_len.
Adds regression tests using the reproduction cases above to
tests/noauto_torch/testn_transliterate_torch.py and
tests/noauto_onnx/testn_transliterate_onnx.py, plus unit tests for
find_trailing_repeat_period() in tests/core/test_transliterate.py.
Closes #1403
While fixing #1403 (thai2rom's romanize() looping to a 100-character repeat under greedy decoding), pythainlp/transliterate/thaig2p.py was found to share the exact same vulnerable Seq2Seq.forward() decode loop: argmax/topk(1) greedy decoding, no repetition penalty, and no guard beyond checking for the <end> token. Reproduced the same failure mode on thaig2p.transliterate(): transliterate("สตรีเศรษฐบุตรบำเพ็ญ") # -> 's a ˨˩ . t r iː ˧ . s u ˧ . s u ˧ . s u ˧ ...' (100 chars) transliterate("เอ็มเอฟซีบัญชีเพื่อการชำระค่ารับซื้อคืน") # -> 'ʔ e m ˧ . b r i ˨˩ . b aː ˧ . b aː ˧ . b aː ˧ ...' (100 chars) Reuses find_trailing_repeat_period() from pythainlp/transliterate/_repetition.py (added on the thai2rom fix branch) in thaig2p.py's Seq2Seq.forward(), the same way it was wired into thai2rom.py and thai2rom_onnx.py: after each inference step, stop decoding and truncate to the cycle's first occurrence as soon as a short cycle repeats 3+ times, instead of running to max_len. Adds a regression test with the reproduction cases above to tests/noauto_torch/testn_transliterate_torch.py. Note: this branch is stacked on fix/thai2rom-repetition-loop (#1500) to reuse _repetition.py; this diff will shrink to just the thaig2p.py change once that PR merges. Refs #1403 Refs #1500
Signed-off-by: Arthit Suriyawongkul <arthit@gmail.com>
Bump budoux from 0.9.0 to 0.9.1
bact suggested passing just the next token instead of threading the full token list and current index through _check_is_thainum; thainum still needed to detect "ศูนย์" coming after "จุด". Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
fix(util): handle Thai zero ("ศูนย์") in text_to_num floating-point parsing
Generate SBOM for wheel
bact requested this change on PR #1500: prefer assertIsNone() over assertEqual(x, None) for the unittest idiom.
Bumps [transformers](https://github.com/huggingface/transformers) from 5.16.1 to 5.17.0. - [Release notes](https://github.com/huggingface/transformers/releases) - [Commits](huggingface/transformers@v5.16.1...v5.17.0) --- updated-dependencies: - dependency-name: transformers dependency-version: 5.17.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.37.9 to 4.38.0. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v4.37.9...v4.38.0) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.38.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
…hub/codeql-action-4.38.0 chore(deps): bump github/codeql-action from 4.37.9 to 4.38.0
….17.0 chore(deps-dev): bump transformers from 5.16.1 to 5.17.0
Hatchling 1.32.3 breaks Pitloom 0.18.0. Fix by update to Pitloom 0.18.1. Signed-off-by: Arthit Suriyawongkul <arthit@gmail.com>
Signed-off-by: Arthit Suriyawongkul <arthit@gmail.com>
Update Pitloom to 0.18.1
fix(transliterate): stop runaway repetition in thaig2p g2p decoding
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upstramming main into dev