Reject malformed hexadecimal in machine:readmem, machine:writemem and machine:debugreg

[chrisgleissner]

Cherry-pick of #884 onto master. That PR was merged to test-merge
(bce4535) but belongs on master, so the same three commits are replayed here
with their original authorship. They applied without conflict, and all four
files are byte-identical to the merged test-merge result.

What it fixes

machine:readmem, machine:writemem (PUT and POST) and machine:debugreg
parsed their hexadecimal parameter with strtol(..., NULL, 16) and never
checked the end pointer. strtol skips leading whitespace, takes an optional
sign, and stops at the first character it cannot use, so input outside the
documented grammar parsed to something and was acted on:

• readmem/writemem: ZZZZ, 0xZZZZ, gggg, -0, +1 and 12GG all
  answered HTTP 200. For writemem that is a write to $0000 reported as
  success.
• machine:debugreg (machine:debugreg accepts malformed and out-of-range hexadecimal values #885): with the register set to 1F first, ZZ, 0xZZ
  and -0 wrote 00, 1G wrote 01, and 1FF was truncated to FF, each
  answered HTTP 200.

The change

One parser for all four call sites, built on the chartohex helper already at
the top of route_machine.cc. It implements the documented grammar and nothing
else: hex digits, no larger than the caller's limit, no sign, no whitespace, no
0x prefix, no trailing characters. The limit is 0xFFFF for an address and
0xFF for the debug register, and it is checked inside the loop, so an
over-long value is refused as it is read rather than truncated.

machine:debugreg's API doc gains the 400 Invalid value it now returns, and
doc/api/rest_api_openapi_u64.yaml is regenerated to match. openapi_check is
a build gate, so the committed document cannot drift from the source.

The one form that used to be accepted and now is not is the 0x prefix. No
client in this repository sends it: the test harness formats %04X, and the
firmware's own web UI uses toString(16).padStart(4, "0") and literal
four-digit strings, with its typed-address path validating as a 16-bit hex
number first.

Tests

Two stages in tests/e2e/api/readmem_writemem_test.py, each gated through the
FIXES table so a machine whose released firmware predates the fix skips with a
named reason:

• bad-address covers the six address forms on all three endpoints.
• bad-debugreg covers the five value forms. machine:debugreg has a GET as
  well as a PUT, so "the refused request wrote nothing" is asserted directly:
  the register is read before and after each rejected write and has to be
  unchanged. It skips where the route does not exist, both debugreg routes
  being inside #if U64.

Verification on this branch

Built from master and JTAG-deployed to an Ultimate 64 Elite:

readmem_writemem_test: OK (78 checks, 23.3s)   --test all, all eight stages

and on an Ultimate II+L in a C64 Ultimate, flashed from this branch's CI
artifact (git_commit_hash 27cfe033):

readmem_writemem_test: OK (38 checks, 16.9s)   --test all

bad-debugreg skips on the cartridge, both debugreg routes being inside
#if U64. freezer-audio, which master gained in #886, also passes on both
machines against this build (u64 6/6 in 7.7s, u2@c64u 6/6 in 9.4s), so the
change sits cleanly on top of it.

Offline: lint_test, registry_test, openapi_contract_test,
check_transport_usage, stale_gates_test, make openapi_check and
make openapi_test (191 tests) all pass.

The red measurements above were taken on #884 by reverting each part of the
firmware in turn and redeploying. The code here is byte-identical to what was
verified there.