Escape control characters in client-sent names in prose

[VSN2015]

A bug fix: client-sent key names could forge log entries. Updated after three review rounds: see "Review rounds" at the end.

The bug

The unknown: :log warn line, the monitor-mode warn line and InvalidParameters#message all interpolate the names of the keys a request sent, raw. 0.8.0's prose bounds cap how many names are listed (10) and how long each is (120 chars), but they escape nothing. So one key:

{ "evil\nE, [2026-09-23] ERROR -- : forged admin login" => "v" }

W, [...] WARN -- : Permittable: unknown parameter(s) ignored by the #create contract: evil
E, [2026-09-23] ERROR -- : forged admin login

That second line is not from the app. Anyone reading or grepping the log sees a fake ERROR entry.

The fix

permittable_prose_item now quotes and escapes a name only when it needs it:

Permittable: unknown parameter(s) ignored by the #create contract: "evil\nE, [2026-09-23] ERROR -- : forged admin login"
Invalid parameters: "evil\nE, [2026-09-23] ERROR -- : forged admin login" (unknown)

  ordinary name (café, 名前, a\nb as literal text, x,y)       unchanged (now always UTF-8)
  name with \n / \r / \t                                      "…\n…"  (short escape)
  any other unsafe character                                  "…\u00A0…", "…\u{E0041}…"
  name with a quote, a lookalike comma, or "and N more"       quoted; those characters as they are
  cp1252 key "caf" 0xE9 0x81 (0x81 has no mapping)           "café\x81"
  developer message: / I18n copy                              printed as is, never escaped
  details / problem+json errors / instrumentation payload     unchanged: the name as sent

InvalidParameters#message is also what API clients read: the envelope's message and the problem+json detail. A client that sent such a key sees the escaped rendering there too. The CHANGELOG says so.

Which characters are escaped (PROSE_UNSAFE)

/[\p{Cc}\p{Cf}\p{Zl}\p{Zp}\p{Zs}&&[^ ]]/, one Unicode property per reason:

• Cc is C0, DEL and C1. C1 is in because U+0085 is NEL (a line break to many readers) and U+009B is the 8-bit CSI that starts a terminal escape.
• Zl/Zp (U+2028/U+2029): JSON-lines and JavaScript readers split a line on them.
• Cf covers two groups:
  • the bidi embeddings, overrides and isolates, which can visually reorder a line and hide where a quoted name ends
  • the zero-width and marker characters (U+200B, U+200E/U+200F, U+061C, U+FEFF, …), which make two different names print identically
• Zs except U+0020, so x,<NBSP>and 49990 more cannot pass for the , separator.

When a name is quoted without escaping (PROSE_AMBIGUOUS)

These characters print as they are. Only the quoting marks the name:

• A quote or a quote lookalike: ", the fullwidth ＂, or any Pi/Pf mark (curly quotes, guillemets). Otherwise x, "evil\nE, … forged" sent with a literal backslash would print like two names, one with a real newline.
• , or a fullwidth, small or ideographic comma (， ﹐ 、). Otherwise the name could pass for two names.
• A name beginning and N more. Otherwise keys b and and 49990 more would print as b, and 49990 more, a fake overflow count. This applies in all three prose outputs.

Every check looks only at the part of the name that the 120-char truncation shows. A character past the cut is never printed, so it does not quote an otherwise ordinary long name.

Decision I'd most like reviewed: the ellipsis may overrun

The rule is that a quoted name is never cut when it fits the limit on its own. When name + suffix is too long, the suffix is cut to whatever room is left. For a quoted name of 118–120 chars, nothing is left, and the ... that marks the dropped suffix then runs up to 3 characters past 120.

The alternatives were worse:

• cutting a name that fit
• dropping the suffix silently, so nothing shows the item was truncated

The overrun is bounded and happens only on this path. A quoted name that does not fit is cut between whole escapes, to exactly 120 with the ... outside the closing quote. The spec pins every boundary, by quoted-name length: 110 (the item is exactly 120), 111, 117, 118, 120 and 121.

Other decisions

• Only the name is escaped. The violation summary passes each item to the prose helper as [param, suffix]. The suffix is the developer's message: or the (code), and it is printed as is. Ordinary items truncate exactly as before.
• Escape format. \n \r \t use the short form. Other characters are \uXXXX, and \u{XXXXX} beyond the BMP (the Cf tag characters). I did not use String#dump/#inspect: dump turns every non-ASCII character into \u, and inspect depends on Encoding.default_external.
• Legacy encodings are transcoded per character with Encoding::Converter#primitive_convert:
  • Characters that map are converted, so a Windows-1252 é prints as é.
  • Only a byte with no mapping (Windows-1252 0x81 0x8D 0x8F 0x90 0x9D) is kept as an invalid byte, which the escaper shows as \xNN.
  • A binary key is read as UTF-8, since Rack hands UTF-8 over as binary.
  • A dummy encoding with no converter falls back to reading the bytes as UTF-8.
• Bounded work. Only the first PROSE_SCAN_LIMIT (4 × 120) characters of a name are converted or scanned.
• Mixed encodings. Every item is UTF-8 before the prose list is joined, so the join cannot raise. Building a nested key's path with "#{path}.#{key}" is left to Turn three client inputs that raised into 422 violations #61, as agreed. The CHANGELOG claim is limited to the join and points at Turn three client inputs that raised into 422 violations #61.

Round 3: an honest limit

This guard defeats literal-text lookalikes: case variants of "and N more", and characters that visually stand in for a quote or comma. It does not attempt general Unicode confusable detection — a Cyrillic а (U+0430) substituted for Latin a in "and N more" still passes unescaped. The CHANGELOG says so plainly, so nobody mistakes this for a complete guarantee. Two concrete gaps were closed instead:

1. Case. \Aand \p{Nd}+ more matched only lowercase; "And 49990 more" printed unescaped and was visually indistinguishable from the real overflow suffix. The match is now \A(?i:and \p{Nd}+ more), scoped to just that alternative so the rest of the pattern stays case-sensitive (nothing else in it is a letter).
2. CJK quote/comma marks. The quote-lookalike set was ["\u{FF02}\p{Pi}\p{Pf}] — but the CJK corner brackets 「/」 (U+300C/U+300D), real quotation marks in Japanese and Chinese text, are punctuation category Ps/Pe, not Pi/Pf, so they were missed entirely. Ruby/Onigmo supports the broader, correct fix: \p{Quotation_Mark}, a binary Unicode property that already covers plain ", fullwidth ＂, every curly quote and guillemet, and the CJK corner brackets — so it replaces the whole enumerated set. (U+3008/U+3009 angle brackets are not in Quotation_Mark — Unicode itself doesn't classify them as quotation marks, and they're used in CJK for titles more than quoting, so I left them out rather than guessing.) Separately, U+FE51 (the small-form ideographic comma, sibling to U+3001 the way U+FE50 is the plain comma's) was missing from the comma set and is now added.

Verification

• 684 examples, 0 failures. The new block has 25 examples; the 2 from this round were written first and failed first. They cover:
  • the CJK corner-bracket and small-form-ideographic-comma cases specifically
  • the overflow phrase in mixed case ("And 49990 more", "AND 1 MORE")
  • the forged line in the :log warn line, the exception message and the monitor warn line
  • every Cc/Cf/Zl/Zp/Zs character escaped, including an astral tag character
  • quote, comma and overflow-count lookalikes quoted in all three outputs; band 4 more stays raw
  • ordinary, non-ASCII, backslash and bare-comma names unchanged
  • the developer message: left alone
  • visible-prefix judging
  • the truncation boundaries above
  • Windows-1252 with every unmapped byte, cp1252 mojibake, and a truncated Shift_JIS key
  • Latin-1 and binary keys
  • details and the instrumentation payload carrying the raw name
• All pre-existing message assertions pass without changes
• RuboCop clean

Review rounds

Round 1 (9e27f54): quote on " anywhere and , ; UTF-8 output for every item; escape only the name, not the developer message; judge the visible prefix and read a bounded one; bidi characters; CHANGELOG notes the client-facing message/detail.

Round 2 (139d3b9):

1. Quote a name beginning and N more.
2. Escape Zs except U+0020; quote on fullwidth, small and ideographic commas, ＂, and Pi/Pf.
3. A quoted name that fits is no longer cut for its suffix.
4. Transcode legacy encodings per character, so only unmappable bytes print as \xNN.
5. Add Cf.
6. CHANGELOG: the mixed-encoding claim is limited to the prose join and references Turn three client inputs that raised into 422 violations #61.

Round 3 (7890786, this update):

1. Make the overflow-phrase match case-insensitive.
2. Replace the enumerated Pi/Pf quote set with \p{Quotation_Mark} (covers the CJK corner brackets too); add U+FE51 to the comma set; state the homoglyph limit in the CHANGELOG.

🤖 Generated with Claude Code

[VSN2015]

Review addressed in 9e27f54 (merged origin/master first; CHANGELOG kept master's entries first). All six points: quote on " anywhere or , ; UTF-8 output for every item; only the client name is escaped (message passed as a separate suffix); judged by the visible prefix, bounded read; bidi override/isolate chars added; CHANGELOG notes the client-facing message/detail. 656 examples, 0 failures; RuboCop clean. PR body updated.

🤖 Generated with Claude Code

[VSN2015]

Regression pass addressed in 139d3b9: the and N more phrase is quoted in all three outputs; non-ASCII spaces (Zs except U+0020) and Cf characters are escaped; fullwidth, small and ideographic commas, ＂ and Pi/Pf quotes trigger quoting. A quoted name that fits the limit is no longer cut for its suffix. That means the ... can run up to 3 chars past 120 for quoted names of 118–120 chars, so please review that decision in the body. Legacy encodings are now transcoded per character, so only unmappable bytes print as \xNN. permittable_path is untouched and the CHANGELOG points at #61. 662 examples, 0 failures; RuboCop clean. PR body updated.

🤖 Generated with Claude Code

[VSN2015]

Final regression round addressed in 7890786 (rebased on origin/master; CHANGELOG kept master's entries first). Two concrete gaps closed: the and N more overflow-phrase match is now case-insensitive; the quote-lookalike set now uses Unicode's \p{Quotation_Mark} property instead of enumerating Pi/Pf, which also covers the CJK corner brackets 「/」 that Pi/Pf missed, plus U+FE51 added to the comma set. CHANGELOG now states plainly that this guard defeats case and punctuation lookalikes but not Unicode homoglyph substitution (e.g. Cyrillic а for Latin a) — no confusable-detection is attempted. 684 examples, 0 failures; RuboCop clean. PR body updated.

🤖 Generated with Claude Code