Skip to content

fix: prevent ReDoS in text normalizer (URL/email/acronym patterns)#489

Open
Fieldnote-Echo wants to merge 3 commits into
remsky:masterfrom
Fieldnote-Echo:security/normalizer-redos
Open

fix: prevent ReDoS in text normalizer (URL/email/acronym patterns)#489
Fieldnote-Echo wants to merge 3 commits into
remsky:masterfrom
Fieldnote-Echo:security/normalizer-redos

Conversation

@Fieldnote-Echo

Copy link
Copy Markdown

Summary

normalize_text() runs synchronously inside the async request path, so a single request containing adversarial text can pin the event loop for tens of seconds via catastrophic regex backtracking — a one-request denial of service. This PR bounds the three vulnerable patterns; normalization behavior is preserved.

Repro: send ~100KB of adversarial text (e.g. "a" * 100_000 or "a." * 50_000) through normalize_text(). Before this fix, the worst cases take 10–65 seconds; after, all complete in under 0.1s.

Root cause and fixes

  1. URL_PATTERN — the domain branch [a-zA-Z0-9.-]+ re-scanned the tail from every position of a long word-char run, O(n²) (100KB of a hung the normalizer). Fix: a token-start negative lookbehind (?<![a-zA-Z0-9.-]) so matching is only attempted at token starts, plus the domain run bounded to 253 chars (the DNS name length limit).

  2. EMAIL_PATTERN — unbounded local-part/domain runs scanned for @ from every word boundary of an a.a.a... flood. Fix: bounded to the RFC limits (64 for local part, 253 for domain).

  3. Dotted-acronym handler (?:[A-Za-z]\.){2,} [a-z] — the unbounded {2,} backtracked O(n) from every position of an a.a.a... flood. Fix: bounded to {2,12} (real dotted acronyms are short); "U.S.A. b""U-S-A- b" behavior is preserved.

Verification

  • uv run pytest api/tests/test_normalizer_redos.py api/tests/test_normalizer.py17 passed (new regression tests + existing normalizer suite unchanged).
  • New test file covers each adversarial vector, a linear-scaling guard (4x input must not cost ~16x time), and behavior-preservation checks for URLs, emails, and dotted acronyms.
  • ruff check clean on both changed files.
Timing at 100KB adversarial input (before → after)
Input (100KB) Pattern hit Before After
"a" * 100_000 URL_PATTERN domain branch ~65s <0.1s
"a." * 50_000 URL/email domain scans + acronym handler ~30s <0.1s
".a" * 50_000 URL/email domain scans ~10s <0.1s
"a@" * 50_000 EMAIL_PATTERN local/domain runs ~10s <0.1s

Notes

  • No behavior change for legitimate input: real URLs, emails, and dotted acronyms normalize exactly as before (asserted in the new tests).
  • Scope is intentionally minimal: 2 files, +70/−3.

🤖 Generated with Claude Code

normalize_text() runs synchronously in the request path, so a single
request of adversarial text could pin the event loop for tens of seconds
via catastrophic regex backtracking:

- URL_PATTERN: the domain branch '[a-zA-Z0-9.-]+' rescanned the whole
  tail from every position of a long word-char run (100KB of 'a' -> the
  normalizer hung). Added a token-start negative lookbehind and bounded
  the domain run to 253 chars (DNS limit).
- EMAIL_PATTERN: unbounded local/domain runs scanned for '@' from every
  word boundary of an 'a.a.a...' flood. Bounded to RFC lengths (64/253).
- Dotted-acronym handler '(?:[A-Za-z]\.){2,} [a-z]': unbounded '{2,}'
  backtracked O(n) per position on an 'a.a...' flood. Bounded to {2,12}
  (real acronyms are short); 'U.S.A. b' -> 'U-S-A- b' behavior preserved.

Worst cases drop from 10-65s to <0.1s at 100KB. Adds
api/tests/test_normalizer_redos.py covering each vector plus a
linear-scaling guard; existing normalizer tests unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reviewer flagged (https?://|www\.|)+ — a '+' quantifier over a group with an
empty alternative, a ReDoS smell and needlessly non-deterministic. Rewrite as
two optionals (?:https?://)?(?:www\.)? — deterministic, same accepted URLs.
handle_url uses only group(0), so dropping the capturing group is safe.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@remsky

remsky commented Jul 19, 2026

Copy link
Copy Markdown
Owner

Good catch; will want to trim the exposition comments, but will merge in for the next release, just going to check some edge cases.

Condense the ReDoS-fix comments to one terse line each — keep the load-bearing
'why bounded' so the limits aren't removed later, drop the exposition.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Fieldnote-Echo

Copy link
Copy Markdown
Author

Trimmed, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants