Translate format: regexps into patterns valid under Ajv's u flag

[VSN2015]

A bug fix. It closes #47 and fixes the rest of the Ruby → ECMA-262 pattern translation, which had the same problem in more places than #47 listed.

The bug

Ajv compiles every pattern as new RegExp(pattern, "u"), and Unicode mode is much stricter than Ruby. The translation copied Ruby's source and swapped \A/\z for ^/$, so:

/\A\d{3}\-\d{4}\z/     ->  ^\d{3}\-\d{4}$      SyntaxError: Invalid escape
:email / :url presets  ->  ...\#...            SyntaxError: Invalid escape (#47)
/\Aa{,3}\z/            ->  ^a{,3}$             SyntaxError (and a literal "{,3}" without u)
/\A[a-z&&[^q]]\z/      ->  ^[a-z&&[^q]]$       compiles without u, means something else
/\A[^@\s]+@[^@\s]+\z/  ->  ^[^@\s]+@[^@\s]+$   accepts "jo\u00a0x@example.com"; the server rejects it
/\A\\A\z/              ->  ^\^$                corrupted: the \\A was an escaped backslash + A

Every one of these is a real pattern in the published document, not an x-permittable-pattern fallback.

The fix: one tokenizer pass that only lets through what it knows

JsonSchema::EcmaPattern (new file) reads the source one token at a time. Escape pairs and character classes are read as whole units. Every token is either rewritten into something both engines read the same way, or the whole regexp exports as x-permittable-pattern:

Ruby	Exported
\A / \z	^ / $ (a real anchor only, so \\A stays a backslash and an A)
\s / \S	[ \t\n\v\f\r] / [^ \t\n\v\f\r], and [^@\s] becomes [^@ \t\n\v\f\r]
[\S] / [\s\S]	[^ \t\n\v\f\r] / [\s\S] (together they cover every character, whatever \s means). \S beside anything else in a class is refused
[\b]	kept: a backspace in both
.	[^\n], because ECMA-262's . also stops at \r, U+2028 and U+2029
\- outside a class, \#, \ , \_	written without the backslash, which means the same thing
\- inside a class, and a bare - right after a completed range ([a-c-e])	kept, because both dialects read either one the same way
a bare { / } that Ruby reads as a literal	\{ / \}
(?>…) (?#…) (?'n'…) {,n} {n}? && nested classes, octal, \e \x7 \g<>, backreferences, \b/\B outside a class, && or [ straight after a range hyphen, a stacked quantifier, a quantified lookaround, a repeated group name	x-permittable-pattern

It is a whitelist on purpose. If nobody thought of a construct, it is refused rather than published, and that keeps the old rule: a wrong pattern in published docs is worse than a missing one. The comment on the module lists each refusal and the reason for it.

The decisions I'd most like reviewed

1. Presets no longer skip the check. They used to skip the regexp scan (vouched:), because that scan read the *+ inside :email's character class as a possessive quantifier. The tokenizer reads a class as a unit, so the skip is no longer needed and I removed it. Presets now take the same path as an app's own regexp. That shared path is also what fixes #47: :email's \# is written as a plain # when it is exported, and the definition does not change. It is still URI::MailTo::EMAIL_REGEXP itself.

2. Backreferences are now refused. You asked me to decide between \1 as a backreference and \1 as octal. I went further and refused both. A backreference to a group that took no part in the match fails in Ruby, but in ECMA-262 it matches the empty string. So /\A(a)?\1b\z/ rejects "b" on the server while its exported pattern accepts it, which is the unsafe direction. Also, \12 can be a backreference or an octal escape depending on how many groups come before it. Backreferences in validation regexps are rare, and before this change they exported with that looser meaning.

3. . → [^\n] is beyond what the issue asked for. It is the same kind of bug as \s. Ruby's . accepts \r and U+2028 and ECMA-262's does not, so the old export was stricter than the server. The translation now matches Ruby exactly.

4. Golden fixture. The email pattern in spec/fixtures/openapi.json changes from ^[^@\\s]+@[^@\\s]+$ to ^[^@ \\t\\n\\v\\f\\r]+@[^@ \\t\\n\\v\\f\\r]+$. That is the \s fix and nothing else. ^\\d{5}$ does not change.

Also

• \p{...} is refused, because Ruby's property names differ from ECMA-262's. In practice these regexps never reach the translation: Ruby marks them encoding-fixed, and the existing options.zero? guard already exports them as x-permittable-pattern. That was true before this PR too.
• The Unreleased Added entry for presets (Add format: presets, which export the JSON Schema format keyword #20) still says presets skip the scan. I left that entry alone and put the correction in the new Fixed entries, so the two PRs don't conflict in the CHANGELOG.

Review round 1

• \b/\B are refused outside a class. Ruby's word boundary counts é as a word character (although its \w does not). Unicode mode's is ASCII-only, so /\Aa\b/ rejects "aé" on the server while ^a\b accepts it. \B is off the other way. This was also exported as-is on master. [\b] (backspace) still translates.
• && or [ straight after a range hyphen is refused. [$-&&%] matches nothing in Ruby but accepts % in ECMA-262, and [a-&&z] is a SyntaxError in Unicode mode.
• [\s\S] exports again. This was a regression against master. A class with both \s and \S exports as [\s\S] (or [^\s\S]), and a lone [\S] / [^\S] becomes the exact complement of Ruby's \s.
• The \u spec case now really reaches the translation. Ruby rewrites \u0041 to A in the source, even through Regexp.new, so the case uses Regexp.new('\A\u{41}\x42\z').
• The node spec's cases live in an EcmaPatternSpec module instead of top-level constants.
• origin/master is merged in. The golden fixture's email pattern for the new master operations now uses the spelled-out \s too.
• A leading-] case and the new hyphen cases are built with warnings off, so the suite prints no regexp warnings.

Review round 2 (final regression pass)

• A bare - right after a completed range was refused — a regression this PR itself introduced. Fixing "&& or [ straight after a range hyphen" (round 1) over-tightened the same guard: it treated "the previous member was a range" the same as "the previous member was a class escape," and refused both. It shouldn't have. [a-c-e] is not "a different range in Unicode mode" — both Ruby and ECMA-262 read that hyphen as an ordinary member once a range has just closed, so [a-c-e] is the set {a, b, c, -, e} (rejecting "d") in both. A format: as ordinary as /\A[a-zA-Z0-9-_]+\z/ therefore published no pattern at all, where origin/master translated it correctly. The hyphen is now carried as a literal member, and may itself reopen a further range ([a-z--x], which is exactly how Ruby's own parser reads a second hyphen). The class-escape half of the guard stays, but I checked it against real Ruby first rather than assuming: Regexp.new('[\d-z]') and Regexp.new('[a-\d]') both raise RegexpError — a class escape can never sit beside a range hyphen in a real Regexp, on either side — so that refusal is a defensive backstop, not something a live disagreement between the two engines depends on.
• Two doc mismatches, found while re-checking this. The CHANGELOG said "\S beside anything else is refused," but the code actually collapses a class holding both \s and \S — plus any other members — to [\s\S]/[^\s\S], since the pair already covers every character whatever either escape means; only \S without \s is refused when anything else shares the class. And the README's fallback rationale implied every refusal is either a Ruby-only construct or one ECMA-262 reads differently, which stopped being true once the over-refusal above needed fixing — there's a third case, a fragment with no ECMA-262 spelling once folded into the rest of its own class (\S beside a member other than \s). Both are reworded to state the actual rules.
• TDD: added a failing spec reproducing the regression first (spec/json_schema_spec.rb), watched it fail against the pre-fix code, then fixed char_class. The node-verification spec's corpus (spec/ecma_pattern_spec.rb) gains the same cases plus the chained-hyphen case, each checked to compile under node -u and to agree with Ruby on samples that include the negative case ("d" for [a-c-e]).

Verification

• 657 examples, 0 failures. 16 of them are new across all three rounds, and each was written before its fix. RuboCop is clean.
• spec/ecma_pattern_spec.rb runs every exported pattern through node with the u flag in a single process: every preset, a table of app-style regexps, and every pattern in the golden fixture. It also checks that each translated pattern accepts exactly what Ruby accepts on edge-case samples: NBSP, U+FEFF, U+2028 and U+3000 for \s; \r and U+2028 for .; # inside :email's local part; \\A; {5}; and now the hyphen-after-range corpus. If node is not on PATH it skips with a message. GitHub's runners have node, so it runs in CI.
• I also ran a throwaway fuzz outside the suite: 24,000 random regexps built from the constructs above, all of them ones Ruby compiles and the tokenizer translates. After the backreference decision, 0 failed to compile under u and 0 disagreed with Ruby on the samples. Before that decision it found exactly one disagreement, the unset-backreference case above.
• After review round 1 I re-ran a wider fuzz. It adds \b, \B, \W, [\s\S], $-&, \u{e9} and \& tokens and é, éa, %, NUL and an emoji as samples. Across 80,000 translated regexps (16 seeds, lengths 1–14): 0 compile errors and 0 disagreements with Ruby. Pointed at the pre-review tokenizer, the same harness finds 105 \b/\B disagreements on one seed, so it does catch that class of bug.
• After round 2's fix, I re-ran that same fuzz harness (fuzz_F66.rb, unchanged) against the fixed code: same 16 seeds, lengths 1–14, 80,000 translated regexps, 0 compile errors, 0 disagreements with Ruby — still clean at the same scale.

Closes #47

🤖 Generated with Claude Code

[VSN2015]

Review fixes pushed in bfc9812, a fast-forward on top of a merge of origin/master:

1. \b/\B are refused outside a class. [\b] (backspace) still translates.
2. && or [ straight after a range hyphen is refused ([$-&&%], [a-&&z]).
3. [\s\S] exports again. A lone [\S] / [^\S] becomes the exact complement of Ruby's \s, and \S beside anything else is refused.
4. I re-checked the CHANGELOG/README wording and added a \b entry. The "read identically" claim now holds.
5. The \u case uses Regexp.new('\A\u{41}\x42\z'), since Ruby rewrites A to A.
6. The spec cases are scoped in an EcmaPatternSpec module.

655 examples, 0 failures, including the node spec. RuboCop is clean. A wider fuzz ran 80,000 translated regexps with 0 compile errors and 0 disagreements. The same harness finds 105 \b disagreements against the pre-review tokenizer. The PR body is updated under "Review round 1".

🤖 Generated with Claude Code

[VSN2015]

Round 2 (final regression pass)

A regression review at bfc9812 found that fixing "&&/[ straight after a range hyphen" (round 1) over-tightened the same guard: a bare - right after a completed range (previous == :range) was refused too, on the theory that [a-c-e] means a different range under the u flag. It doesn't — both Ruby and ECMA-262 read that hyphen as an ordinary member once a range has just closed, so [a-c-e] is {a, b, c, -, e} (rejecting "d") in both. A format: as common as /\A[a-zA-Z0-9-_]+\z/ therefore exported no pattern at all, where origin/master translated it correctly.

Fix: that hyphen is now carried as a literal class member, and may itself reopen a further range ([a-z--x], matching Ruby's own reading of a repeated hyphen). The class-escape half of the guard (\d/\w/\s beside a hyphen) stays, but only after checking it against real Ruby rather than assuming: Regexp.new('[\d-z]') and Regexp.new('[a-\d]') both raise RegexpError on either side of the hyphen, so that combination can never reach the translator as a real Regexp — the refusal is a defensive backstop, not something a live divergence depends on.

Also fixed two doc mismatches found in the same pass:

• CHANGELOG said "\S beside anything else is refused"; the code actually collapses a class holding both \s and \S (plus other members) to [\s\S]/[^\s\S], since the pair already covers every character. Reworded to state that only \S without \s is refused when anything else shares the class.
• README's fallback rationale implied every refusal is either Ruby-only or something ECMA-262 reads differently — no longer true once the over-refusal above is fixed. Reworded to name the third case (no ECMA-262 spelling once folded into the rest of a class) and to state the hyphen fix explicitly.

Process: TDD — added a failing spec reproducing [a-zA-Z0-9-_] and friends first, watched it fail against bfc9812, then fixed char_class in lib/permittable/json_schema/ecma_pattern.rb. Added the same corpus (plus negations like [a-c-e]'s "d", and the chained [a-z--x] case) to the node-verification spec, each checked to compile under node -u and agree with Ruby on samples.

Verification:

• 657 examples, 0 failures (was 655; 2 new specs this round). RuboCop clean.
• Re-ran the fuzz harness from the previous round (fuzz_F66.rb, unchanged) against the fixed code: 16 seeds, lengths 1–14, 80,000 translated regexps, 0 compile errors, 0 disagreements with Ruby — same scale as round 1, still clean.

Commit: caaf6f5

🤖 Generated with Claude Code