Turn three client inputs that raised into 422 violations

[VSN2015]

A bug fix. Client input raised out of the contract, which turned a 422 into a 500. Each case below is reproduced and now produces an ordinary violation. Revised after three review rounds. The encoding handling is inspect, never convert, so controllers using skip_parameter_encoding keep their bytes.

The bugs

  array :ids, of: :integer, validate: ->(a) { a.sum < 100 }
    sent ["x", 2]                     TypeError: nil can't be coerced into Integer
  required :n, :integer
    sent Float::NAN                   FloatDomainError: NaN
    sent "12".encode("UTF-16LE")      Encoding::CompatibilityError
  required :d, :decimal
    sent "12".encode("UTF-16LE")      accepted as 1   (read byte by byte)
  required :s, :string, normalize: :squish    (or format:, :strip, :email, :downcase)
    sent "caf\xC3"                    ArgumentError: invalid byte sequence in UTF-8
    sent UTF-16 / binary text         Encoding::CompatibilityError   (format:, :squish)
  Contract.define(unknown: :error) { ... }
    sent key "caf\xC3"                the 422 itself: JSON::GeneratorError on to_json
    sent a UTF-16 key                 Encoding::CompatibilityError building the path

A standalone Contract#call on a webhook payload has nothing in front of it. A controller using skip_parameter_encoding / param_encoding receives binary or other-encoding Strings on purpose. JSON parsed with allow_nan reaches the NaN case.

The fix

  ["x", 2] against validate: a.sum              ids[0] invalid_type        (validate: not run)
  [1, -4], validate: all positive, transform: sqrt   ids invalid            (transform: not run)
  NaN / Infinity for :integer                   invalid_type
  UTF-16 "12" for :integer / :decimal           12 / 12                    (parsed from a UTF-8 copy)
  Windows-1252 "1\x81" for :integer             invalid_type               (no UTF-8 reading)
  binary "caf\xE9", Shift_JIS, UTF-16 :string   returned unchanged, own encoding
  "caf\xC3" (UTF-8), lone UTF-16 surrogate      invalid_type               (invalid in its OWN encoding)
  UTF-16 " café " with normalize: :squish       left as it is
  UTF-16 / binary "café" against /\Acafé\z/     format
  undeclared key "caf\xC3"                      { param: "caf�", code: "unknown" }

1. Array validate: runs only when no element violated. An undeclared key inside an element does not count. transform: runs only when nothing violated, validate: included, as on master and on the scalar path.
2. :integer returns invalid_type for a non-finite Float, as :float and :decimal already did.
3. Other encodings are inspected, never converted:
   • A String invalid in its own encoding is invalid_type for every scalar type.
   • Number, boolean and date casts parse a UTF-8 copy of the text. If there is no UTF-8 reading, that is invalid_type.
   • A :string is returned in its original encoding, bytes unchanged.
   • When format: cannot be applied to a String's encoding, that is a format violation.
   • When a built-in normalize: preset cannot handle the encoding, the value is left as it is. An app's own normalize: Proc always runs and always raises its own errors, on any encoding — this round's fix; see below.
4. Violation params are always valid UTF-8. Undeclared keys are converted for reporting, and anything that still cannot be read becomes U+FFFD.

No new error codes, so the ERROR_SCHEMA enumeration is unchanged.

Where the checks live

• Coercion.cast is the one step every scalar value goes through: a field, an of: element, a sub-field of an array of hashes, and an authored default:/example:. It refuses a String that fails valid_encoding?, returns a :string unchanged, and gives the other types a utf8_text copy — reusing the encoding check already done a line above when the value is already UTF-8, rather than scanning it twice (see Regression pass 3 below).
• Coercion.format_match? turns an EncodingError/ArgumentError from the regexp into "does not match".
• apply_normalize skips a String that fails valid_encoding?, so the cast refuses it. For a String that passes that check, only a built-in preset (identified by object identity against NORMALIZERS' values) gets the encoding-failure leniency; an app-supplied Proc's raise always propagates.
• permittable_unknown_key_violation is the one place a client key enters a path, and the only place reportable_text runs. Declared keys are not converted, and a spec asserts that a clean nested request calls reportable_text zero times. The entry is also remembered by identity. That is how permittable_check_array tells an undeclared key from a failed sub-field, because the code alone cannot: a sub-field's validate: may return :unknown, and a spec covers that case.

Regression pass 3 (final)

Two issues found on re-review of round 2's diff, both fixed here:

1. Real regression: a custom normalize: Proc's own raise was silently swallowed on non-UTF-8 input. apply_normalize's leniency for an encoding a preset can't handle was written as a method-level rescue EncodingError, ArgumentError, so it also caught an early return normalizer.call(value) for an app-supplied Proc — normalize: ->(v) { raise ArgumentError, "business rule violated" if v.include?("FAKE") } propagated on master and on UTF-8 input, but silently returned the unvalidated value for Shift_JIS, Windows-1252 or binary-with-high-byte input. That made normalize: an unintended bypass channel for exactly the apps this PR is meant to help (skip_parameter_encoding). Fixed by scoping the rescue to the preset call alone, gated on NORMALIZERS.value?(normalizer) — object identity, since resolve_normalizer! replaces field[:normalize] with the exact Proc from that frozen Hash. A custom Proc's ArgumentError or EncodingError now always propagates, on any encoding, exactly as on master. New spec: a custom Proc raising both error classes on Shift_JIS, Windows-1252 and binary input.
2. Minor perf: a redundant full-string encoding scan on every non-:string scalar field. Coercion.cast called value.valid_encoding? directly, then utf8_text re-checked text.valid_encoding? on the same object when its fast path returned ordinary UTF-8 input unchanged. cast now skips straight to the object it already validated when the encoding is UTF-8, and only calls utf8_text for the encodings where it produces a genuinely new object (US-ASCII, binary, anything needing encode). New spec spies on valid_encoding? via a String subclass and asserts exactly one call per cast.

Behaviour changes worth reviewing

• The array validate: trade-off. An array-level violation is no longer reported alongside element violations. [1, "x", 1] against a uniqueness validator reports only ids[1], and the duplicate shows up after the client fixes that and resends. The CHANGELOG states this plainly, and a spec pins it.
• skip_parameter_encoding apps behave as on master, except that former crashes (format:, a built-in preset like :squish) are now violations, and a preset that can't handle the encoding now leaves the value as it is instead of raising. An app's own normalize: Proc is unaffected either way — see Regression pass 3.
• README claims narrowed. It now documents what Contract#call still raises: ArgumentError for input that is not a Hash, nil (read as {}) or to_unsafe_h-able, plus the app's own validate:/transform:/normalize:/finalize code. It no longer claims that the app receives UTF-8.

:json: deliberately not examined

A :json field's nested strings are neither checked nor converted, and a spec pins that down. The gem runs no string operation inside a :json value, so nothing there can raise in the gem. Walking every leaf on every request would add the cost the opaque type exists to avoid.

Also

• Left as it is: keys are converted only for reporting, not for matching. A UTF-16 "name" key does not match a declared :name.
• origin/master is merged into the branch, twice across the review rounds. CHANGELOG conflicts are resolved with master's entries first.

Verification

• 693 examples, 0 failures. Each review round's specs were written before its implementation: 10, 9, 9, and 2 failing respectively. The rest pin unchanged behaviour: other scalar types, well-formed non-ASCII text, the :json pass-through, the validate: trade-off, the :unknown-returning sub-field, and an app normalizer bug on ordinary UTF-8 still raising. The Contract#call specs check not_to raise_error and the exact violation, and call to_json on a violation for a non-UTF-8 key.
• RuboCop clean

🤖 Generated with Claude Code

[VSN2015]

Review addressed in 25416d2 (merged master first; fast-forward push):

1. Every String is read as UTF-8 before any cast: binary via force_encoding on a copy, other encodings via encode, and anything not valid UTF-8 afterwards is invalid_type. UTF-16 "12" is now 12 for :integer and :decimal. Specs cover UTF-16LE, binary with high bytes, and Windows-1252.
2. Violation params are always valid UTF-8. permittable_path converts client keys and replaces what it cannot read with U+FFFD. Specs call to_json via Contract#call and check the message and the unknown: :log line.
3. The CHANGELOG now states the array validate: trade-off plainly, and a spec pins it.
4. The CHANGELOG and README claims now match the code, including a README paragraph on what #call still raises.

663 examples, 0 failures; RuboCop clean. PR body updated.

🤖 Generated with Claude Code

[VSN2015]

Regression pass addressed in 3ae9495 (fast-forward push):

1. An array's transform: no longer runs after its validate: failed. It runs only when nothing violated, as on master and on the scalar path.
2. Redesigned to inspect other encodings, never convert them. A String invalid in its own encoding is invalid_type. Numeric, boolean and date casts parse a UTF-8 copy. A :string comes back in its original encoding. An unapplicable format: is a format violation, and an unapplicable normalizer leaves the value as it is. skip_parameter_encoding apps behave as on master, except that former crashes are now violations.
3. CHANGELOG and README claims rewritten to match.
4. An undeclared key inside an element no longer stops the array's validate:. Undeclared-key violations are recognised by identity, not by code, because a sub-field's own validate: may return :unknown.
5. Only undeclared keys go through reportable_text. A spec asserts zero calls for a clean nested request.
6. README now says Contract#call(nil) reads as {}, and lists which inputs raise ArgumentError.

671 examples, 0 failures; RuboCop clean. PR body updated.

🤖 Generated with Claude Code

[VSN2015]

Final regression pass addressed in f7ccbb4 (merged master; fast-forward push):

1. Regression fixed: apply_normalize's leniency for an encoding a normalizer can't handle is now scoped to the gem's own presets only, identified by object identity against NORMALIZERS' values. An app-supplied normalize: Proc's own raise (ArgumentError or EncodingError) always propagates, on any encoding, exactly as on master. The bug was a method-level rescue that also caught an early-return call to a custom Proc; the rescue is now scoped to the preset call alone. New spec: a custom Proc raising both error types on Shift_JIS, Windows-1252 and binary input — both propagate.
2. Perf fixed: Coercion.cast no longer re-scans a UTF-8 String's encoding inside utf8_text when the fast path returns the same object — the check just above already answered it. New spec spies on valid_encoding? via a String subclass and asserts exactly one call per cast for ordinary UTF-8 input.

693 examples, 0 failures; RuboCop clean.

🤖 Generated with Claude Code