fix: prevent ReDoS in text normalizer (URL/email/acronym patterns)#489
Open
Fieldnote-Echo wants to merge 3 commits into
Open
fix: prevent ReDoS in text normalizer (URL/email/acronym patterns)#489Fieldnote-Echo wants to merge 3 commits into
Fieldnote-Echo wants to merge 3 commits into
Conversation
normalize_text() runs synchronously in the request path, so a single
request of adversarial text could pin the event loop for tens of seconds
via catastrophic regex backtracking:
- URL_PATTERN: the domain branch '[a-zA-Z0-9.-]+' rescanned the whole
tail from every position of a long word-char run (100KB of 'a' -> the
normalizer hung). Added a token-start negative lookbehind and bounded
the domain run to 253 chars (DNS limit).
- EMAIL_PATTERN: unbounded local/domain runs scanned for '@' from every
word boundary of an 'a.a.a...' flood. Bounded to RFC lengths (64/253).
- Dotted-acronym handler '(?:[A-Za-z]\.){2,} [a-z]': unbounded '{2,}'
backtracked O(n) per position on an 'a.a...' flood. Bounded to {2,12}
(real acronyms are short); 'U.S.A. b' -> 'U-S-A- b' behavior preserved.
Worst cases drop from 10-65s to <0.1s at 100KB. Adds
api/tests/test_normalizer_redos.py covering each vector plus a
linear-scaling guard; existing normalizer tests unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reviewer flagged (https?://|www\.|)+ — a '+' quantifier over a group with an empty alternative, a ReDoS smell and needlessly non-deterministic. Rewrite as two optionals (?:https?://)?(?:www\.)? — deterministic, same accepted URLs. handle_url uses only group(0), so dropping the capturing group is safe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
|
Good catch; will want to trim the exposition comments, but will merge in for the next release, just going to check some edge cases. |
Condense the ReDoS-fix comments to one terse line each — keep the load-bearing 'why bounded' so the limits aren't removed later, drop the exposition. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Author
|
Trimmed, thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
normalize_text()runs synchronously inside the async request path, so a single request containing adversarial text can pin the event loop for tens of seconds via catastrophic regex backtracking — a one-request denial of service. This PR bounds the three vulnerable patterns; normalization behavior is preserved.Repro: send ~100KB of adversarial text (e.g.
"a" * 100_000or"a." * 50_000) throughnormalize_text(). Before this fix, the worst cases take 10–65 seconds; after, all complete in under 0.1s.Root cause and fixes
URL_PATTERN— the domain branch[a-zA-Z0-9.-]+re-scanned the tail from every position of a long word-char run, O(n²) (100KB ofahung the normalizer). Fix: a token-start negative lookbehind(?<![a-zA-Z0-9.-])so matching is only attempted at token starts, plus the domain run bounded to 253 chars (the DNS name length limit).EMAIL_PATTERN— unbounded local-part/domain runs scanned for@from every word boundary of ana.a.a...flood. Fix: bounded to the RFC limits (64 for local part, 253 for domain).Dotted-acronym handler
(?:[A-Za-z]\.){2,} [a-z]— the unbounded{2,}backtracked O(n) from every position of ana.a.a...flood. Fix: bounded to{2,12}(real dotted acronyms are short);"U.S.A. b"→"U-S-A- b"behavior is preserved.Verification
uv run pytest api/tests/test_normalizer_redos.py api/tests/test_normalizer.py— 17 passed (new regression tests + existing normalizer suite unchanged).ruff checkclean on both changed files.Timing at 100KB adversarial input (before → after)
"a" * 100_000URL_PATTERNdomain branch"a." * 50_000".a" * 50_000"a@" * 50_000EMAIL_PATTERNlocal/domain runsNotes
🤖 Generated with Claude Code