[codezzzsleep]

What this adds

skills/splash-app-verify/: a development skill for OctoSense Splash script apps (main.splash). This PR contains the author's v3.3 skill files unchanged; the runner's scenario protocol remains version 3. It moves verification to the front, the way backend code is tested:

1. Write a contract (screens and states, actions, host services, storage, sizes) and turn it into scenario files before the code.
2. Keep the app in a narrow, testable shape (references/SHAPE.md).
3. After every edit, run every scenario in card-host with scripts/uitest.py. Host services are answered from the scenario, so every failure path (no service, an error, no answer, an authorization that ended, a refusal) is reproducible without a real host.
4. A failed scenario writes FEEDBACK.md: steps, expected, actual and evidence, in the shape docs/MODEL-VALIDATION.md asks for, so a model or a person fixes one thing at a time.
5. A final section covers acceptance on the real host, which the runner does not replace.

The runner is Python 3.9+, standard library only. It drives card-host through the remote routes (/snap, /click, /t, /k, /m), never edits the bundle under test, and starts its own Xvfb on a headless Linux machine.

Also in this PR:

• a link from AGENTS.md (step 6) and from both READMEs;
• a CI step that runs the skill's own unit tests (no card-host needed);
• two examples: examples/notes tests the App Flow template app; examples/titled-notes is a small app in the narrow shape with ten scenario files and twelve runs across sizes.

Current host policy

Capability names are disclosures on current OctoSense hosts (App Hub e1fdf28, makepad d653bc44), while Rinx still grants a mini-app only the services it was granted. A scenario picks the host with "host_profile":

• octosense (default): an undeclared service is answered, and the run reports it as undisclosed;
• rinx: it is refused with Rinx's own text, Mini app was not granted <service> (Rinx crates/miniapp-core/src/lib.rs:103).

The test copy keeps the bundle's capabilities, storage and quota as written. The fake host reports calls on card-host's log with a per-run code, so missing or unreadable call evidence is an error, never "no calls".

Evidence

The recorded example and OnCue scenario runs used card-host built from App Hub main 8d3853c, with the Splash runtime Octoscript-Makepad 6cf2d90a, on Linux x86_64 (2 cores, Xvfb, Mesa llvmpipe). Unit tests and tool/link checks do not need card-host.

What	Result
Skill unit tests (scripts/test_uitest.py)	58 / 58 on Python 3.11, 3.12 and 3.13, also under a cp1252 locale (Windows' default encoding) and with every default-encoding use turned into an error
examples/notes against templates/script-app/bundle, recorded	6 / 6 runs
examples/titled-notes, recorded	12 / 12 runs
A real app, OnCue, a Rinx mini-app of 2,500 to 3,200 lines of Splash, recorded	0.5.2: 43 / 43 runs; 0.5.3: 67 / 67 runs (OnCue 96ce6ee, with this PR's runner); 0.5.4: 85 / 85 runs (OnCue 6dab6a1, with the revised runner)
App Flow tool tests and link check, current patch on main 15b1da01	48 tests, OK (skipped=7); 0 broken links
Malformed input	0 crashes in 24,000 mutated scenario files and 18,000 mutated call logs; 51,000 steps driven over 12,000 mutated scenarios

Ordinary recorded scenario runs take about 0.5 to 2 seconds; timeout scenarios wait for the app's own timer (20 seconds in the titled-notes "never answers" case). card-host's recorded peak memory was 240 to 265 MB.

How it found real bugs: on OnCue 0.5.2 the first run of 38 scenarios failed 6 of 43 runs, on 5 real defects:

• an assistant error reported as "could not read the room";
• an untranslated rate limit;
• a send error carrying the wrong subject;
• a consent button still saying "allowed" after the grant expired;
• a status line asking to confirm a confirmation that had been invalidated.

Each was fixed red to green with the scenarios unchanged. Three of the shape rules in references/SHAPE.md come from these fixes.

The runner itself was reviewed independently four times. The following fixes have regression coverage:

• a storage name that escaped the jail;
• unknown keys that skipped checks;
• the test copy gaining storage;
• unreadable call logs read as zero calls;
• cleanup skipped on an evidence error;
• setup.sh not persisting the card-host path;
• a flush widget reported as clipped;
• budget-stop and parse-error log lines missed;
• file checks matching a same-named file elsewhere;
• a stale summary.json after a failed rerun;
• text matched across two widgets;
• an example that wrote over a store it could not read;
• a bundle symlink the copy would have written through;
• string "false" or true accepted where a boolean or a count was meant;
• a call record cut off at the end of the log;
• a host that fails to close, or no display, leaving no final result;
• a host close failure still being counted and printed as a pass;
• count, calls or files accepted as objects, or null leaving a summary marked as running;
• malformed shared parts and host replies slipping through until the host starts;
• non-finite JSON numbers, lone surrogate escapes and excessive nesting causing failures after loading;
• unbounded waits, selector chains and scroll counts, and invalid or overlong storage paths.

Negative checks on the real card-host: a parse error fails (with a hint, since the runtime logs nothing for it); a budget stop on the older runtime fails; an undeclared service is refused under rinx and noted under octosense; a file written to the wrong path fails the exact-path check.

Not verified

• macOS and Windows: only Linux was run; the tests avoid Linux-only paths, and a cp1252 locale stood in for Windows' default encoding.
• L0 cards (page.card): out of scope.
• Real host services: by design the runner fakes them. SKILL.md §5 says how to finish on the real host.
• Phones.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[BH3GEI]

Thanks for putting this together — the runner's validation discipline is unusually thorough, and the tests are genuinely portable: I ran the full suite here, 43/43 on Python 3.14.5 and again under a non-UTF-8 locale, plus a 150-case mutation fuzz of the loader (only clean exit-2s, no tracebacks). One caveat on scope: I did not get to run the scenarios against a real card-host in this pass, so the host-driving path is reviewed at the unit/code level; everything below was reproduced or traced directly.

Two issues I'd want addressed before this lands:

1. Environment failures still write a FEEDBACK.md addressed to the app. scripts/uitest.py:1084 writes FEEDBACK.md on environment_error runs too. Reproduced here by running a valid scenario against a host binary that exits immediately: the resulting FEEDBACK.md is titled "Repair request", names the bundle's main.splash artifact, and ends with "Fix: change only what makes this step pass. Keep widget ids…", while the actual cause is card-host exited (0) — an exit-2, "not the app's fault" failure by the skill's own contract. SKILL.md step 3 tells the repair model to read FEEDBACK.md and fix only what it names; a dead Xvfb or a hung card-host would send that model editing app source for a machine problem. Could environment errors skip the app-facing template, or clearly retitle the artifact as not a repair request?

2. The close path is POSIX-only, so a hung card-host on Windows raises and leaks. scripts/uitest.py:566 reaches os.killpg(..., signal.SIGKILL) when the process ignores /quit; the surrounding handlers catch only ProcessLookupError/TimeoutExpired. os.killpg does not exist on Windows, and start_new_session=True is a no-op there, so a card-host that hangs past the 5s wait raises AttributeError into the close wrapper and the process is left running — one leaked process per scenario. If the runner is meant to be Linux/macOS-only, a platform note (or a proc.kill() fallback) would close it either way.

Smaller points, each verified against the tree:

• scripts/uitest.py:762 vs the Checks table in references/SCENARIOS.md: a calls item with only excludes passes when the service was never called (the "excludes" not in item carve-out skips the "≥1" requirement), while the doc says "without count, at least one". Doc and code should agree, whichever way you pick.
• references/SCENARIOS.md:42 documents storage values as "text, or JSON for an object", but examples/notes/tests/04-restart.json and 05-remove.json seed JSON arrays, and the runner (scripts/uitest.py:1005) accepts any JSON value. Someone following the docs would wrap or stringify where they needn't.
• examples/titled-notes/bundle/main.splash:25-26,91 uses is_array() / is_string(), which appear nowhere else in the repo — not in docs/SCRIPT-API.md, docs/RUNTIME-TYPES.md, or any other .splash — while references/SHAPE.md itself says "Use only documented APIs". The recorded runs show the predicates exist in the current runtime; could their source be cited (or the docs extended) so the flagship example meets the skill's own rule?
• The example set doesn't reach SKILL.md's own "at least" list: main.splash:33 implements the "not granted" branch ("This app may not use the model."), but no scenario sets host_profile: "rinx" or exercises it; and only 01-first-screen.json runs at two sizes — the main path (02-keep.json) runs at the default size only. Since agents will pattern-match on these examples, closing the gap (or noting it) would help.
• scripts/uitest.py:983-987: mkdtemp() and run_dir.mkdir() sit before the try whose finally removes the work dir, so an OSError there (e.g. a scenario stem long enough that stem@size exceeds 255 bytes) leaks one uitest-* temp dir and surfaces as a generic environment error rather than a scenario-name complaint.
• scripts/uitest.py:380-388: if all fifty candidate displays are taken, the loop falls through and Xvfb is spawned on :139 regardless; if the socket never appears the runner still proceeds, costing a full startup_timeout per scenario with the error pointing at card-host rather than the display.
• scripts/uitest.py:1353: use/run part names accept absolute paths and .. (base / name), escaping the scenario directory — inconsistent with the strict jail applied to storage and files names.
• scripts/test_uitest.py:273: test_examples_are_valid asserts a step-action set that lacks scroll_to (and run), both documented in SCENARIOS.md — the first example adopting them would fail this test spuriously.
• SKILL.md:52: "Rinx refuses a mini-app what it was not granted" → "the services it was not granted".
• In the same spirit as the runner's own "nothing passes silently" note: "sizes": [] validates and silently runs the default size, and {"text": [""]} passes vacuously.

For the record, these all held up under checking: the exact CI step command on the 3-OS onboarding job; tools/check-links.py 0 broken; every scenario field documented in SCENARIOS.md cross-checked against the runner's validation constants; all 14 example scenarios field-validated and traced against both apps' code (including the template texts examples/notes targets); both host profiles' refusal/reporting behavior; the storage-jail and symlink refusal; and the version/claim consistency (runner: 3). Looking forward to the next revision.

[codezzzsleep]

Thanks for the review. The revision is pushed (3de7f92). Both blocking points are fixed:

1. Environment errors are no longer repair requests. A run that ends in an environment error (no display, a card-host that exited or would not stop, call evidence the runner cannot read) now writes ENVIRONMENT.md, titled "Environment error, not a repair request", which says not to change main.splash for it. It never writes FEEDBACK.md. The console line reads "environment, not the app", and SKILL.md step 3 has the matching rule. Your reproduction, a card-host that exits at once, now ends with exit status 2 and only ENVIRONMENT.md in the run directory; a regression test runs exactly that case.
2. Stopping a hung card-host is portable. On POSIX the process group gets SIGTERM, then SIGKILL; where os.killpg does not exist (Windows), the process gets terminate(), then kill(). A host that survives both is reported as an environment error instead of being left running. Tests cover both paths with a fake process.

The smaller points, in your order:

• calls: without count, at least one call, with or without excludes; "count": 0 requires none. Code and SCENARIOS.md now agree.
• Storage values: SCENARIOS.md says any JSON value; a string is written as the file's text, anything else as its JSON. A test seeds an array, an object, a number and a text.
• is_array() / is_string(): documented in SCRIPT-API (Data and strings, "Type checks") with the runtime source (MP/platform/script/src/native.rs, the types list and the loop after it, which gives every value every predicate), and cited in the example.
• Examples: titled-notes/tests/10-model-not-granted.json sets "host_profile": "rinx" and exercises the "not granted" branch (it fails when that branch is removed), and 02-keep.json runs at two sizes.
• The work directory is now created right before the try that removes it, and a scenario file name too long for its <stem>@<size> evidence directory is a scenario error when the scenarios load.
• Xvfb: no free display number, or a server that exits or never makes its socket, is now a display error before any scenario runs.
• use/run part names follow the storage name rules and must resolve inside the directory of the file that names them, links included.
• test_examples_are_valid takes the action set from the runner, so scroll_to counts.
• SKILL.md:52 now reads "the services it was not granted".
• "sizes": [] and an empty text in a check or a selector are refused when the scenarios load.

Evidence for this revision: 58 skill unit tests pass on Python 3.11, 3.12 and 3.13, also under a cp1252 locale and with every default-encoding use turned into an error. 15 of them are new: 12 fail on the previous runner, and the other 3 pin behavior that must not change. The App Flow tool tests pass (48, 6 skipped), and tools/check-links.py reports 0 broken links. On Linux card-host (App Hub 8d3853c, Octoscript-Makepad 6cf2d90a), titled-notes passed 12/12, notes 6/6 and OnCue 0.5.4 85/85 (75 scenarios). Still not run: card-host on macOS or Windows; the Windows close path is covered by unit tests only.