diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4fa818dd..f3755da9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -26,6 +26,8 @@ jobs: python-version: '3.12' - name: Starter and executable discovery regressions run: python -m unittest discover -s tools -p "test_*.py" -v + - name: Script app verify skill (runner logic, no card-host) + run: python -m unittest discover -s skills/splash-app-verify/scripts -p "test_*.py" -v rust-sdk: # The Rust SDK for app components (OctoSense ADR 0014): the macro, the diff --git a/AGENTS.md b/AGENTS.md index bdcf453a..75813b4b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -65,6 +65,9 @@ says who runs each one and when. loaded, not only that it opens. Judge behavior, layout and appearance separately. Keep the evidence of every failure, and say which checks a person ran and which you ran. + For a script app, use [the script app verify skill](skills/splash-app-verify/SKILL.md): + the contract and scenarios before the code, and the scenarios run after + every edit. For new Glance cards or card UX acceptance, also use [the app-card UX skill](skills/octoscript-app-card-ux/SKILL.md): summary-to-workspace transitions, shared editing state, keyboard/scrolling checks and source-bound @@ -343,4 +346,4 @@ The full list is in [docs/SCRIPT-API.md](docs/SCRIPT-API.md#gotchas). - Loop with `for i in n`; there is no `range()`. - Write `name := Widget{}` to address a widget as `ui.name`. - `draw_bg +: {…}` merges your fields into the existing `draw_bg`. -- In `on_render`, keep `if` and `for` separate (no `else for`). +- In `on_render`, keep `if` and `for` separate (no `else for`, braced or not). diff --git a/README.md b/README.md index 3643ff11..f84a1202 100644 --- a/README.md +++ b/README.md @@ -201,6 +201,17 @@ release tag, submission) are listed in [AGENTS.md](AGENTS.md#how-to-work); explains the ones every flow shares. An agent never fabricates an approval, a review result or a submission. +### Script app verify skill + +Use [splash-app-verify](skills/splash-app-verify/SKILL.md) when writing or +changing a script app (`main.splash`). It moves the work to the front: write the +contract and the scenarios before the code, keep the app in a narrow testable +shape, and run the scenarios in `card-host` after every edit, with host +services answered from the scenario. A failed scenario writes a repair request +in the shape [MODEL-VALIDATION](docs/MODEL-VALIDATION.md) asks for. It checks +behavior and placement, not looks, and does not replace the gate or the visual +review. + ### App card UX skill Use [octoscript-app-card-ux](skills/octoscript-app-card-ux/SKILL.md) when designing diff --git a/README.zh-CN.md b/README.zh-CN.md index 79f4219f..380bd209 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -107,6 +107,10 @@ macOS 上的原生公开签名目录安装及 0.2.0 → 0.2.1 更新保留了本 需要人来把关的节点(Release 工作流、发布者身份与隐私文本、平台声明、付费图像生成、视觉确认、Release 的 tag、提交)都列在 [AGENTS.md](AGENTS.md#how-to-work) 中;[flows/README.zh-CN.md](flows/README.zh-CN.md#每个流程都遵循同一份约定) 说明所有流程共有的那些节点。Agent 绝不伪造人的确认、评审结果或提交记录。 +### 脚本应用验证技能 + +编写或修改脚本应用(`main.splash`)时,使用 [splash-app-verify](skills/splash-app-verify/SKILL.md)。它把工作前移:先写契约和测试场景再写代码,让应用保持可测试的收窄结构,每次修改后在 `card-host` 中运行全部场景,宿主服务的回复由场景提供。场景失败时,按 [MODEL-VALIDATION](docs/MODEL-VALIDATION.zh-CN.md) 要求的格式写出修复请求。它检查行为和位置,不评判外观,也不替代准入检查和视觉评审。 + ### AppCard UX 技能 设计或验收速览卡片时,使用 [octoscript-app-card-ux](skills/octoscript-app-card-ux/SKILL.md)。它补充摘要 → 展开卡片 → 完整工作区的任务流程、Chat/编辑/审阅共用状态、键盘和滚动检查,以及保留指定模型创作归属的修复循环。验收矩阵区分本地行为、真实 Agent 执行、外部业务效果和视觉批准,历史分数不能代替新卡片的验收。 diff --git a/docs/MODEL-VALIDATION.md b/docs/MODEL-VALIDATION.md index 7115765d..cfdc4f32 100644 --- a/docs/MODEL-VALIDATION.md +++ b/docs/MODEL-VALIDATION.md @@ -8,6 +8,11 @@ from the final source: a successful tool call or a model's summary is not enough Use this guide alongside the selected [flow](../flows/README.md), not as a replacement for its admission, visual-review or publishing gates. +For script apps, the [script app verify skill](../skills/splash-app-verify/SKILL.md) +turns this loop into scenario files that run in `card-host` after every edit, +with host services answered from the scenario, and writes the executable +feedback below when one fails. + For Glance card work, the [app-card UX skill](../skills/octoscript-app-card-ux/SKILL.md) adds a task journey and acceptance matrix for workspace expansion, shared editing state, phone keyboard reachability and retained context. It reuses this model diff --git a/docs/MODEL-VALIDATION.zh-CN.md b/docs/MODEL-VALIDATION.zh-CN.md index 227c9e02..70a449a6 100644 --- a/docs/MODEL-VALIDATION.zh-CN.md +++ b/docs/MODEL-VALIDATION.zh-CN.md @@ -4,6 +4,8 @@ 宿主提供相应工具后,编码模型可以编写应用、查看实际渲染结果、注入输入并修复失败。完成与否取决于最终源码的验证证据,不能仅凭工具调用成功或模型自己的总结。本指南配合所选[设计流程](../flows/README.zh-CN.md)使用,不替代准入、视觉评审或发布检查。 +脚本应用可使用[脚本应用验证技能](../skills/splash-app-verify/SKILL.md):它把这个循环写成场景文件,每次修改后在 `card-host` 中运行,宿主服务的回复由场景提供;场景失败时写出下文的可执行反馈。 + 速览卡片可进一步使用 [AppCard UX 技能](../skills/octoscript-app-card-ux/SKILL.md),补充工作区展开、共享编辑状态、手机键盘可达性和上下文保留的任务流程与验收矩阵。该技能复用本文的模型验证循环,不另建模型服务或截图协议。 ## 选择测试方式 diff --git a/docs/SCRIPT-API.md b/docs/SCRIPT-API.md index 3339375b..83c2eb31 100644 --- a/docs/SCRIPT-API.md +++ b/docs/SCRIPT-API.md @@ -260,6 +260,7 @@ host.request("mail.list", {account: account.id folder: folder.id offset: 0 limit | Missing | `to_lower to_upper contains starts_with ends_with` | use `search(p) == 0`, `search(p) >= 0`, or a regex | | Arrays | `push pop clear len remove(i) retain(fn)` | no `sort map filter join insert index_of slice` | | Objects | `o.k`, `o[k]`, `delete`, `extend`, `o += {k: v}` | | +| Type checks | `v.is_string()`, `is_array()`, `is_object()`, `is_number()`, `is_bool()`, `is_nil()`; also `is_nan is_color is_regex is_err is_id is_opcode` | Every value has every one of these (`MP/platform/script/src/native.rs`: the `types` list and the loop after it), so they are safe on parsed JSON of unknown shape. `is_array` and `is_string` **✓ run** in `card-host` on Linux ([splash-app-verify example](../skills/splash-app-verify/examples/titled-notes/bundle/main.splash)). | | Numbers → text | `"" + n` | | | Loops | `for i in n` (0..n-1), `for v in array`, `for i v in array`, `for k v in object`, `a..b` ranges, `loop {}`, `while` | there is **no `range()`** (**✓ run**: `for i in 3` sums to 3; `for k v in {a:1 b:2}` gives `ab`) | @@ -328,7 +329,10 @@ installed apps; disclosure changes do not expose the host profile. an exponent. - **`on_render` empty state:** write `if list.len() == 0 { Empty } for i in list.len() { Row }`. Do not write `if … {…} else for …`: the empty branch draws nothing and stale - rows stay on screen (**✓ run**). + rows stay on screen (**✓ run**). The braced form `if … {…} else { for … }` + fails the same way: the empty label is never drawn (**✓ run**, a scenario in + [splash-app-verify](../skills/splash-app-verify/SKILL.md) against a copy of its + example with only that change). - **Backgrounds:** `View{show_bg: true draw_bg.color: …}` draws no background in `card-host`, whether always visible or shown later with `set_visible(true)` (**✓ run**); `SolidView{draw_bg.color: …}` does. Use diff --git a/skills/splash-app-verify/SKILL.md b/skills/splash-app-verify/SKILL.md new file mode 100644 index 00000000..2d3d1ad5 --- /dev/null +++ b/skills/splash-app-verify/SKILL.md @@ -0,0 +1,190 @@ +--- +name: splash-app-verify +description: Build or change an OctoSense Splash script app (main.splash) so it is right by construction and verified like backend code - write observable scenarios before the code, keep the app in a narrow testable shape, and run the scenarios in card-host with faked host services after every edit. Use when creating or changing a script app, when a model keeps breaking its UI, or before hand-off. +--- + +# Splash app verify + +A UI app is hard for a model to check: its logic lives in the screen, it calls +host services that a development machine does not have, and seeing it means +starting a whole window. So models write blind and a person finds the bugs +later, when they cost the most. This skill moves the work to the front: + +- **Prevent first.** Fix what "done" looks like, and the shape the code may + take, before writing it. Most UI bugs come from an open-ended process: a + layout nobody budgeted, a host reply nobody planned for, logic spread through + click handlers. A narrow process leaves less room for them. +- **Test like a backend.** Each behavior is a scenario file. A runner starts the + real app in `card-host`, answers its `host.request` calls from the scenario, + clicks and types through the remote bridge, and checks what the screen, the + storage jail and the log show. Run it after every edit. +- **Repair one thing at a time.** A failed scenario writes `FEEDBACK.md`: the + steps, what was expected, what the screen had, the evidence and the scope of + the fix, in the shape [MODEL-VALIDATION](../../docs/MODEL-VALIDATION.md) + asks for. Hand it to the author, a person or a model, unchanged. + +This skill adds to the chosen flow ([script-app](../../flows/script-app/FLOW.md)); +it does not replace the gate, the visual review or publishing. + +## 0. Set up once + +```sh +skills/splash-app-verify/scripts/setup.sh # Linux: Xvfb + Mesa; builds card-host if missing +python3 skills/splash-app-verify/scripts/uitest.py doctor +``` + +`doctor` must print `[ok]` for `card-host` and the display. `setup.sh` records +the `card-host` it finds or builds, so later shells need no variable; +`--card-host` and `OCTO_CARD_HOST` still win. Build `card-host` from current +App Hub `main`: it carries the Splash runtime the current hosts run. On a +machine with no display the runner starts its own Xvfb; nothing appears on +anyone's screen. + +## 1. Write the contract before any code + +Write `BRIEF.md` (outside `bundle/`) with these tables. Every later step checks +against them, so a gap here becomes a gap in the app. + +| Table | One row per | Columns | +| --- | --- | --- | +| Screens and states | screen × state: first open with data, empty, loading, ready, each failure | what the person sees: the exact texts that must and must not show | +| Actions | button, input or gesture | widget id (`name :=`), visible text, what changes, what is saved | +| Host services | `host.request` service | capability, the documented reply shape, every failure the app must show: not granted, no service, an error, no answer, an authorization that ended. Name the host: on OctoSense a capability is a disclosure; Rinx refuses a mini-app the services it was not granted | +| Storage | file in the jail | format, version, what happens when it is missing or corrupt | +| Sizes | size the app is given: a phone, and the rectangle each shell gives it inside its own window | what must be on the first screen without scrolling; what may need a scroll | + +Cite where each host service and API comes from ([SCRIPT-API](../../docs/SCRIPT-API.md), +[HOST-SERVICES](../../docs/HOST-SERVICES.md), the runtime source). If nothing +documents it, the app cannot use it. + +Then turn the tables into scenarios, **before** `main.splash` exists or before +you change it. At least: + +1. the first screen with its data loaded; +2. the main path, end to end; +3. the empty state; +4. each host-service failure from the table; +5. a restart: storage seeded, the saved state comes back; +6. the main path at the smallest and the largest size. + +The format is in [references/SCENARIOS.md](references/SCENARIOS.md). Put the +normal host replies and repeated sequences (load, confirm) in shared `_parts`, +so each scenario states only what differs; a Rinx mini-app's shared part sets +`"host_profile": "rinx"`. Run them once on the old source, +or on the bare template. A new scenario must fail first; one that passes +before the feature exists tests nothing. + +## 2. Keep the app in a narrow shape + +Write the app in the shape [references/SHAPE.md](references/SHAPE.md) shows. Each +rule there closes off a class of bug: + +- State in top-level `let`s. Decisions in plain functions that take and return + data. The screen updated only from a few `sync_*` functions. +- One function per host service, handling every reply in the contract, and + ignoring a reply that arrives after its request was replaced. +- A `name :=` id on every widget a scenario or a function touches. +- Lists drawn with the documented empty-state form: `if list.len() == 0 { … }` + then `for …`, never in an `else`. +- Layout from `Fill`/`Fit`, wrapping rows and one scroll container, with a + budget per region for the smallest size. No fixed heights around text. +- Only documented APIs; `//` comments; `#x` colors; no reserved words as names. + +## 3. Build one path at a time; test after every edit + +1. Pick the next red scenario in contract order: first screen, then the main + action, then its failures, then the next action. +2. Make the smallest change that can turn it green. +3. Run all scenarios: + ```sh + python3 skills/splash-app-verify/scripts/uitest.py run /bundle /tests + ``` +4. Green: commit, take the next one. Red: read `FEEDBACK.md`, fix only what it + names, run again. If a scenario that was green turns red, undo the last edit + before anything else. +5. `ENV` (exit status 2): the machine failed, not the app. The run writes + `ENVIRONMENT.md` instead of `FEEDBACK.md`; it is not a repair request. Do not + change the app for it: fix the display, the card-host or the disk + (`uitest.py doctor`), then run again. + +Keep exactly one scenario you are working on red. Do not edit a scenario to +make it pass unless the brief changed, and then say so in the commit. + +## 4. Sizes, then look at it + +When all scenarios pass at every size, take the screenshots the flow asks for +(`tools/octo shot`) and have a person, or a model that can read images, review +them against the brief. The runner checks text, placement inside the viewport, +enablement, counts, host calls, files and runtime errors. It does not judge +looks, and a clipped glyph inside a widget that is itself in view can pass. + +## 5. Real host, last + +The fixtures cover every reply in the contract, including the ones a real host +makes hard to produce: another person posting, a grant that expired, a +switched account, a rate limit. A real host is for acceptance, not discovery: +one pass of the main path and the failures you can trigger safely. + +- **Prepare it before you need it.** List the accounts, rooms and permissions + each real check needs. Set them up a day ahead and try each once with a + request that changes nothing. +- **Diagnose before you work around.** When the host does something the + contract does not say, shrink it to the smallest request that shows it, read + the client and server source for that request, and write the cause down with + file and line before changing anything. Report it under "Gaps found" with + the reproduction. If it is the host's bug, keep the app's contract. +- **No unattended effects.** A scheduled check may probe and notify. Sending, + creating rooms, inviting and anything else that changes the world waits for + a person. +- **Bind evidence to source.** Record the `main.splash` SHA-256 each real run + used. A run on an earlier source is history, not proof of the current one. + +## 6. Hand off + +Report, from `build/uitest/summary.json` and the flow's checks: + +- which scenarios pass, at which sizes, against which `main.splash` SHA-256; +- what the fixtures stood in for and was not run for real: the host services, + permission sheets, a phone keyboard, provider behavior; +- what ran on a real host, against which source; +- anything a person must still look at. + +## What the runner does + +`scripts/uitest.py run `: + +1. copies the bundle and prepends a fake `host` to the copy's `main.splash`, so + `host.request` gets the scenario's replies; the code under test is otherwise + unchanged, and the bundle under test is never edited; +2. starts `card-host` on the copy (hidden; under a private Xvfb on a headless + Linux machine) at each size the scenario lists; +3. performs the steps through the remote routes and checks each expectation; + a runtime error in the log fails the step; +4. stops at the first failure and writes `snap.json`, `screen.png`, + `card-host.log`, `calls.json` and `FEEDBACK.md` under `build/uitest/`; + every run also keeps the app's jail files there. When the environment + failed instead (no display, a card-host that exited or would not stop, + evidence it cannot read), it writes `ENVIRONMENT.md`, never `FEEDBACK.md`. + +Exit status: 0 all passed, 1 a scenario failed, 2 the environment, the +evidence or a scenario file is broken (not the app's fault). Scenario files +are checked when they load: an unknown key, a value of the wrong kind or a +file name that leaves the jail stops the run before any app starts. The test +copy keeps the bundle's capabilities and storage as written; the fake host +reports calls through card-host's log. Under each result the runner notes what +is true of the run but not a failure, such as a service called without being +declared. `summary.json` is rewritten by every run and names its `run_id` and +`status`. + +It sees what `/snap` lists, which is what is on screen; scroll to a part of +the app before checking it ([what the runner sees](references/SCENARIOS.md#what-the-runner-sees)). + +A card-host that ignores `/quit` is stopped: on Linux and macOS its whole +process group (terminate, then kill), on Windows the process itself. One that +survives that is reported as an environment error, never left running +silently. The recorded runs are on Linux; the macOS and Windows paths are +covered by unit tests only. + +Examples: [examples/notes](examples/notes) tests the App Flow template app; +[examples/titled-notes](examples/titled-notes) is a small app in the shape, +with its scenarios. diff --git a/skills/splash-app-verify/agents/openai.yaml b/skills/splash-app-verify/agents/openai.yaml new file mode 100644 index 00000000..4947f7d9 --- /dev/null +++ b/skills/splash-app-verify/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Splash App Verify" + short_description: "Scenarios first, then code: test script apps in card-host after every edit" + default_prompt: "Use $splash-app-verify to write the contract and scenarios for this script app, then build or change main.splash one path at a time and run every scenario in card-host after each edit." diff --git a/skills/splash-app-verify/examples/notes/tests/01-first-screen.json b/skills/splash-app-verify/examples/notes/tests/01-first-screen.json new file mode 100644 index 00000000..262fbbe0 --- /dev/null +++ b/skills/splash-app-verify/examples/notes/tests/01-first-screen.json @@ -0,0 +1,8 @@ +{ + "name": "first screen: empty list and the add row in view", + "sizes": ["412x892", "990x613"], + "steps": [ + {"expect": {"text": ["My Notes", "No notes yet.", "Tap a note to remove it."], + "in_view": [{"id": "entry"}, {"text": "Add"}]}} + ] +} diff --git a/skills/splash-app-verify/examples/notes/tests/02-add-note.json b/skills/splash-app-verify/examples/notes/tests/02-add-note.json new file mode 100644 index 00000000..5717a916 --- /dev/null +++ b/skills/splash-app-verify/examples/notes/tests/02-add-note.json @@ -0,0 +1,10 @@ +{ + "name": "add a note: it shows, the empty text goes, it is saved", + "steps": [ + {"type": {"into": {"id": "entry"}, "text": "Buy milk"}}, + {"click": {"text": "Add"}}, + {"wait": {"text": ["Buy milk"], "absent": ["No notes yet."], + "files": [{"name": "notes.json", "contains": ["Buy milk"]}]}}, + {"expect": {"count": [{"of": {"text": "Buy milk"}, "n": 1}]}} + ] +} diff --git a/skills/splash-app-verify/examples/notes/tests/03-empty-input.json b/skills/splash-app-verify/examples/notes/tests/03-empty-input.json new file mode 100644 index 00000000..102143a2 --- /dev/null +++ b/skills/splash-app-verify/examples/notes/tests/03-empty-input.json @@ -0,0 +1,7 @@ +{ + "name": "an empty input adds nothing", + "steps": [ + {"click": {"text": "Add"}}, + {"expect": {"text": ["No notes yet."], "files": [{"name": "notes.json", "missing": true}]}} + ] +} diff --git a/skills/splash-app-verify/examples/notes/tests/04-restart.json b/skills/splash-app-verify/examples/notes/tests/04-restart.json new file mode 100644 index 00000000..d6c40d15 --- /dev/null +++ b/skills/splash-app-verify/examples/notes/tests/04-restart.json @@ -0,0 +1,7 @@ +{ + "name": "restart: saved notes come back", + "storage": {"notes.json": ["First", "Second"]}, + "steps": [ + {"wait": {"text": ["First", "Second"], "absent": ["No notes yet."]}} + ] +} diff --git a/skills/splash-app-verify/examples/notes/tests/05-remove.json b/skills/splash-app-verify/examples/notes/tests/05-remove.json new file mode 100644 index 00000000..c8a0072f --- /dev/null +++ b/skills/splash-app-verify/examples/notes/tests/05-remove.json @@ -0,0 +1,10 @@ +{ + "name": "tap a note to remove it", + "storage": {"notes.json": ["Keep", "Drop"]}, + "steps": [ + {"wait": {"text": ["Keep", "Drop"]}}, + {"click": {"text": "Drop"}}, + {"wait": {"text": ["Keep"], "absent": ["Drop"]}}, + {"expect": {"files": [{"name": "notes.json", "contains": ["Keep"]}]}} + ] +} diff --git a/skills/splash-app-verify/examples/titled-notes/README.md b/skills/splash-app-verify/examples/titled-notes/README.md new file mode 100644 index 00000000..64bd4363 --- /dev/null +++ b/skills/splash-app-verify/examples/titled-notes/README.md @@ -0,0 +1,34 @@ +# Titled Notes + +A small script app in the [narrow shape](../../references/SHAPE.md): notes kept +on the device, and a "Suggest a title" button that asks the host's +`model.complete` once. Its scenarios cover the first screen and the main action +at two sizes, an empty input, a restart, a corrupt store, a title from the +model, the model refusing on budget, no model service at all, a model that +never answers, and a host that has not granted the model (`10-model-not-granted`, +written the way a Rinx mini-app's scenario is: `"host_profile": "rinx"`). + +```sh +python3 skills/splash-app-verify/scripts/uitest.py run \ + skills/splash-app-verify/examples/titled-notes/bundle \ + skills/splash-app-verify/examples/titled-notes/tests --out /tmp/uitest-titled --keep-going +``` + +Recorded on 2026-10-13 (Linux x86_64, 2 cores, Xvfb with Mesa llvmpipe, card-host +built from App Hub `8d3853c` with the Splash runtime Octoscript-Makepad `6cf2d90a`): +all 12 runs pass. After the first launch each takes 0.6–1.1 s, except the "never +answers" scenario, which waits out the app's own 20-second timer. card-host's peak +resident memory was 241–244 MB in every run. With the `not granted` line removed +from `model_message`, `10-model-not-granted` fails, as it should. + +`05-corrupt-store` keeps a note after the unreadable file is reported and checks +that the file is still the original. Before the app locked writes, that step +replaced the file with the new note; the scenario failed then and passes now. + +## The braced `else` in `on_render` + +Change the list to `if notes.len() == 0 { Label{…} } else { for … }` and run the +same scenarios: `01-first-screen` fails at both sizes and `05-corrupt-store` +fails, each with `text "No notes yet." is shown` / `not on screen`. That is the +documented `else for` gotcha in its braced form; see +[SCRIPT-API](../../../../docs/SCRIPT-API.md#gotchas). diff --git a/skills/splash-app-verify/examples/titled-notes/bundle/assets/icon.svg b/skills/splash-app-verify/examples/titled-notes/bundle/assets/icon.svg new file mode 100644 index 00000000..4978331a --- /dev/null +++ b/skills/splash-app-verify/examples/titled-notes/bundle/assets/icon.svg @@ -0,0 +1,4 @@ + + + + diff --git a/skills/splash-app-verify/examples/titled-notes/bundle/main.splash b/skills/splash-app-verify/examples/titled-notes/bundle/main.splash new file mode 100644 index 00000000..02b9eec0 --- /dev/null +++ b/skills/splash-app-verify/examples/titled-notes/bundle/main.splash @@ -0,0 +1,134 @@ +// Titled Notes: the narrow shape from references/SHAPE.md, small enough to read +// in one sitting. Notes are kept on the device; "Suggest a title" asks the +// host's model service once, and every reply the contract lists has a visible +// result. + +// ---- 1. State: top-level lets only. +let notes = [] // [{text, title}] +let status = "Write a note and keep it." +let asking = false +let ask_token = 0 // the reply to an older request is ignored +let store_file = "notes-v1.json" +let store_locked = false // the file could not be read: never write over it + +// ---- 2. Decisions: plain functions, data in, data out, no ui. +fn clean(text){ text.trim() } +fn note_ok(text){ text != "" && text.len() <= 500 } +fn with_note(list, text){ + let out = [] + for n in list { out.push(n) } + out.push({text: text, title: ""}) + out +} +// is_array() and is_string() exist on every value (makepad platform/script/src/native.rs, +// the `types` list and the loop after it; docs/SCRIPT-API.md, Data and strings). +fn store_valid(v){ + try { + if v == nil || v.version != 1 || !v.notes.is_array() { return false } + for n in v.notes { if !n.text.is_string() || !n.title.is_string() { return false } } + true + } catch { false } +} +fn model_message(error){ + // `: ` from model.complete (docs/AI-SERVICES.md). + if error.search("no service answers") >= 0 { return "Titles need the OctoSense model service; notes still work." } + if error.search("not granted") >= 0 { return "This app may not use the model. Notes still work." } + if error.search("budget") == 0 || error.search("rate") == 0 { return "Today's AI budget is used up. Try tomorrow." } + if error.search("no_provider") == 0 { return "No AI provider is set up in OctoSense." } + "No title: " + error +} + +// ---- 3. Storage: one versioned file, validated on read. +fn load(){ + if fs.exists(store_file) { + let v = fs.read(store_file).parse_json() + if store_valid(v) { notes = v.notes } else { + // Keep the unreadable file as it is: the person may still recover it. + store_locked = true + status = "The saved notes could not be read; nothing was changed, and nothing will be saved over them." + } + } + sync() +} +fn save(){ + if store_locked { return false } + fs.write(store_file, {version: 1, notes: notes}.to_json()) + true +} + +// ---- 4. Actions: change state, then sync once. +fn keep(){ + let text = clean(ui.entry.text()) + if store_locked { status = "Not kept: the saved notes could not be read, and keeping would overwrite them." sync() return } + if !note_ok(text) { status = "Write something first (up to 500 characters)." sync() return } + notes = with_note(notes, text) + save() + ui.entry.set_text("") + status = "Kept." + sync() +} + +// ---- 5. Host service: one function, every reply handled, stale replies ignored. +fn suggest_title(){ + if notes.len() == 0 { status = "Keep a note first." sync() return } + if asking { return } + ask_token += 1 + let token = ask_token + let index = notes.len() - 1 + asking = true + status = "Asking for a title…" + sync() + host.request("model.complete", { + task: "Give the note a short title." + input: {note: notes[index].text} + schema: {type: "object" required: ["title"] additionalProperties: false + properties: {title: {type: "string" maxLength: 40}}} + class: "fast" + }, fn(r){ + if token != ask_token { return } + asking = false + if !r.is_ok { status = model_message(r.error) sync() return } + let title = "" + try { title = r.data.output.title } catch { title = "" } + if !title.is_string() || title.trim() == "" { status = "The reply had no title." sync() return } + notes[index].title = title.trim() + save() + status = "Titled." + sync() + }) + start_timeout(20.0, || { + if token == ask_token && asking { asking = false ask_token += 1 status = "No answer in 20 seconds. Try again." sync() } + }) +} + +// ---- 6. Sync: the only place that writes to the screen. +fn sync(){ + ui.status.set_text(status) + ui.list.render() +} + +start_timeout(0.05, || load()) + +let ink = #x1c1c1e +let secondary = #x8e8e93 +let Chip = ButtonFlat{height: 40 padding: Inset{left: 16 right: 16} + draw_bg +: {border_radius: 20.0 color: #x007aff color_hover: #x0a84ff color_down: #x0062cc color_focus: #x007aff border_size: 1.0 border_color: #x007aff border_color_hover: #x0a84ff border_color_down: #x0062cc border_color_focus: #x007aff} + draw_text +: {color: #xffffff color_hover: #xffffff color_down: #xffffff color_focus: #xffffff}} +SolidView{width: Fill height: Fill flow: Down padding: 16 spacing: 12 draw_bg.color: #xffffff + Label{width: Fill text: "Titled Notes" draw_text.color: ink draw_text.text_style: theme.font_bold{font_size: 24}} + status := Label{width: Fill text: "" draw_text.color: secondary draw_text.text_style.font_size: 13} + View{width: Fill height: Fit flow: Right{wrap: true} spacing: 8 align: Align{y: 0.5} + entry := TextInput{width: Fill height: 40 empty_text: "Write a note" + draw_bg +: {color: #xf2f2f7 color_hover: #xf2f2f7 color_focus: #xf2f2f7 color_empty: #xf2f2f7 border_radius: 10.0} + draw_text +: {color: ink color_hover: ink color_focus: ink color_empty: secondary color_empty_hover: secondary}} + keep_button := Chip{text: "Keep" on_click: || keep()} + title_button := Chip{text: "Suggest a title" on_click: || suggest_title()} + } + list := ScrollYView{width: Fill height: Fill flow: Down spacing: 8 on_render: || { + if notes.len() == 0 { Label{text: "No notes yet." draw_text.color: secondary} } + for i in notes.len() { + if notes[i].title != "" { Label{width: Fill text: notes[i].title draw_text.color: ink draw_text.text_style: theme.font_bold{font_size: 14}} } + Label{width: Fill text: notes[i].text draw_text.color: ink draw_text.text_style.font_size: 15} + } + }} +} diff --git a/skills/splash-app-verify/examples/titled-notes/bundle/manifest.json b/skills/splash-app-verify/examples/titled-notes/bundle/manifest.json new file mode 100644 index 00000000..6dbf9a57 --- /dev/null +++ b/skills/splash-app-verify/examples/titled-notes/bundle/manifest.json @@ -0,0 +1,13 @@ +{ + "capabilities": [ + "storage", + "model" + ], + "id": "titled-notes-demo", + "integrity": { + "bundle_blake3": "" + }, + "name": "Titled Notes", + "schema": 1, + "version": "0.1.0" +} diff --git a/skills/splash-app-verify/examples/titled-notes/tests/01-first-screen.json b/skills/splash-app-verify/examples/titled-notes/tests/01-first-screen.json new file mode 100644 index 00000000..4772a23b --- /dev/null +++ b/skills/splash-app-verify/examples/titled-notes/tests/01-first-screen.json @@ -0,0 +1,8 @@ +{ + "name": "first screen: empty list, hint, both buttons in view", + "sizes": ["412x892", "990x613"], + "steps": [ + {"expect": {"text": ["Titled Notes", "No notes yet.", "Write a note and keep it."], + "in_view": [{"id": "entry"}, {"text": "Keep"}, {"text": "Suggest a title"}]}} + ] +} diff --git a/skills/splash-app-verify/examples/titled-notes/tests/02-keep.json b/skills/splash-app-verify/examples/titled-notes/tests/02-keep.json new file mode 100644 index 00000000..0e293b39 --- /dev/null +++ b/skills/splash-app-verify/examples/titled-notes/tests/02-keep.json @@ -0,0 +1,10 @@ +{ + "name": "keep a note: shown, saved, input cleared", + "sizes": ["412x892", "990x613"], + "steps": [ + {"type": {"into": {"id": "entry"}, "text": "Call the plumber"}}, + {"click": {"text": "Keep"}}, + {"wait": {"text": ["Call the plumber", "Kept."], "absent": ["No notes yet."], + "files": [{"name": "notes-v1.json", "contains": ["Call the plumber", "\"version\":1"]}]}} + ] +} diff --git a/skills/splash-app-verify/examples/titled-notes/tests/03-keep-empty.json b/skills/splash-app-verify/examples/titled-notes/tests/03-keep-empty.json new file mode 100644 index 00000000..355bc3c4 --- /dev/null +++ b/skills/splash-app-verify/examples/titled-notes/tests/03-keep-empty.json @@ -0,0 +1,7 @@ +{ + "name": "keep with an empty input: a hint, nothing saved", + "steps": [ + {"click": {"text": "Keep"}}, + {"expect": {"text": ["Write something first"], "files": [{"name": "notes-v1.json", "missing": true}]}} + ] +} diff --git a/skills/splash-app-verify/examples/titled-notes/tests/04-restart.json b/skills/splash-app-verify/examples/titled-notes/tests/04-restart.json new file mode 100644 index 00000000..77d903c5 --- /dev/null +++ b/skills/splash-app-verify/examples/titled-notes/tests/04-restart.json @@ -0,0 +1,7 @@ +{ + "name": "restart: saved notes and titles come back; a corrupt file is left alone", + "storage": {"notes-v1.json": {"version": 1, "notes": [{"text": "Pay rent", "title": "Rent"}]}}, + "steps": [ + {"wait": {"text": ["Rent", "Pay rent"], "absent": ["No notes yet."]}} + ] +} diff --git a/skills/splash-app-verify/examples/titled-notes/tests/05-corrupt-store.json b/skills/splash-app-verify/examples/titled-notes/tests/05-corrupt-store.json new file mode 100644 index 00000000..3c2a39f8 --- /dev/null +++ b/skills/splash-app-verify/examples/titled-notes/tests/05-corrupt-store.json @@ -0,0 +1,12 @@ +{ + "name": "a corrupt store: say so, change nothing, and never save over it", + "storage": {"notes-v1.json": "{not json"}, + "steps": [ + {"wait": {"text": ["could not be read", "No notes yet."], "files": [{"name": "notes-v1.json", "contains": ["{not json"]}]}}, + {"type": {"into": {"id": "entry"}, "text": "Buy milk"}}, + {"click": "Keep"}, + {"expect": {"text": ["Not kept", "Buy milk"], "absent": ["Kept."], + "files": [{"name": "notes-v1.json", "contains": ["{not json"]}]}, + "note": "keeping would have replaced the unreadable file, which the person may still recover"} + ] +} diff --git a/skills/splash-app-verify/examples/titled-notes/tests/06-title-ok.json b/skills/splash-app-verify/examples/titled-notes/tests/06-title-ok.json new file mode 100644 index 00000000..d693416f --- /dev/null +++ b/skills/splash-app-verify/examples/titled-notes/tests/06-title-ok.json @@ -0,0 +1,12 @@ +{ + "name": "suggest a title: the model's title is shown and saved", + "storage": {"notes-v1.json": {"version": 1, "notes": [{"text": "Call the plumber about the sink", "title": ""}]}}, + "host": {"model.complete": {"data": {"output": {"title": "Fix the sink"}, "meta": {"class": "fast"}}, "delay": 0.3}}, + "steps": [ + {"wait": {"text": ["Call the plumber about the sink"]}}, + {"click": {"text": "Suggest a title"}}, + {"wait": {"text": ["Fix the sink", "Titled."], + "calls": [{"service": "model.complete", "count": 1, "contains": "Call the plumber about the sink"}], + "files": [{"name": "notes-v1.json", "contains": ["Fix the sink"]}]}} + ] +} diff --git a/skills/splash-app-verify/examples/titled-notes/tests/07-title-budget.json b/skills/splash-app-verify/examples/titled-notes/tests/07-title-budget.json new file mode 100644 index 00000000..87346c77 --- /dev/null +++ b/skills/splash-app-verify/examples/titled-notes/tests/07-title-budget.json @@ -0,0 +1,55 @@ +{ + "name": "the model refuses (budget): a plain message, the stored note unchanged", + "storage": { + "notes-v1.json": { + "version": 1, + "notes": [ + { + "text": "Buy stamps", + "title": "" + } + ] + } + }, + "host": { + "model.complete": { + "error": "budget: today's budget for this app is used up" + } + }, + "steps": [ + { + "wait": { + "text": [ + "Buy stamps" + ] + } + }, + { + "click": { + "text": "Suggest a title" + } + }, + { + "wait": { + "text": [ + "budget is used up" + ] + } + }, + { + "expect": { + "absent": [ + "Titled." + ], + "files": [ + { + "name": "notes-v1.json", + "contains": [ + "\"title\": \"\"" + ] + } + ] + } + } + ] +} diff --git a/skills/splash-app-verify/examples/titled-notes/tests/08-no-model-service.json b/skills/splash-app-verify/examples/titled-notes/tests/08-no-model-service.json new file mode 100644 index 00000000..dbe42d34 --- /dev/null +++ b/skills/splash-app-verify/examples/titled-notes/tests/08-no-model-service.json @@ -0,0 +1,9 @@ +{ + "name": "no model service on this host (card-host as it is)", + "storage": {"notes-v1.json": {"version": 1, "notes": [{"text": "Water plants", "title": ""}]}}, + "steps": [ + {"wait": {"text": ["Water plants"]}}, + {"click": {"text": "Suggest a title"}}, + {"wait": {"text": ["Titles need the OctoSense model service"]}} + ] +} diff --git a/skills/splash-app-verify/examples/titled-notes/tests/09-title-no-answer.json b/skills/splash-app-verify/examples/titled-notes/tests/09-title-no-answer.json new file mode 100644 index 00000000..33f8f36d --- /dev/null +++ b/skills/splash-app-verify/examples/titled-notes/tests/09-title-no-answer.json @@ -0,0 +1,11 @@ +{ + "name": "the model never answers: the app stops waiting by itself", + "storage": {"notes-v1.json": {"version": 1, "notes": [{"text": "Book dentist", "title": ""}]}}, + "host": {"model.complete": {"never": true}}, + "steps": [ + {"wait": {"text": ["Book dentist"]}}, + {"click": {"text": "Suggest a title"}}, + {"expect": {"text": ["Asking for a title"]}}, + {"wait": {"text": ["No answer in 20 seconds"]}, "timeout": 25} + ] +} diff --git a/skills/splash-app-verify/examples/titled-notes/tests/10-model-not-granted.json b/skills/splash-app-verify/examples/titled-notes/tests/10-model-not-granted.json new file mode 100644 index 00000000..883d01e9 --- /dev/null +++ b/skills/splash-app-verify/examples/titled-notes/tests/10-model-not-granted.json @@ -0,0 +1,17 @@ +{ + "name": "the host refuses the model: a plain message, and notes still work", + "host_profile": "rinx", + "note": "Rinx refuses a service it did not grant with this text (Rinx crates/miniapp-core/src/lib.rs:103). This app declares the model, which the profile alone would answer, so the fixture sends the refusal.", + "storage": {"notes-v1.json": {"version": 1, "notes": [{"text": "Renew passport", "title": ""}]}}, + "host": {"model.complete": {"error": "Mini app was not granted model.complete"}}, + "steps": [ + {"wait": {"text": ["Renew passport"]}}, + {"click": {"text": "Suggest a title"}}, + {"wait": {"text": ["This app may not use the model. Notes still work."], + "calls": [{"service": "model.complete", "count": 1}]}}, + {"type": {"into": {"id": "entry"}, "text": "Buy bread"}}, + {"click": {"text": "Keep"}}, + {"wait": {"text": ["Kept.", "Buy bread"], + "files": [{"name": "notes-v1.json", "contains": ["Renew passport", "Buy bread"]}]}} + ] +} diff --git a/skills/splash-app-verify/references/SCENARIOS.md b/skills/splash-app-verify/references/SCENARIOS.md new file mode 100644 index 00000000..b6ee90dc --- /dev/null +++ b/skills/splash-app-verify/references/SCENARIOS.md @@ -0,0 +1,209 @@ +# Scenario files + +One JSON file per behavior, kept in `/tests/` beside `bundle/`. The runner +reads every `*.json` in the directory, in name order. Name them by contract +row: `01-first-screen.json`, `02-add-note.json`, `03-read-fails.json`. Files +whose names start with `_` are shared parts, not scenarios. + +```json +{ + "name": "a message that arrived before sending is shown, and sending waits for a second confirmation", + "use": "_room.json", + "sizes": ["412x892", "954x448"], + "host": { + "matrix.read_messages": [ + {"data": {"messages": [{"sender": "ann", "sender_id": "@ann:example.org", "event_id": "$1", "body": "noon works", "ts": 1, "msgtype": "m.text"}]}}, + {"data": {"messages": []}, "delay": 1.5}, + {"error": "Mini-app authorization expired"} + ], + "matrix.send_message": {"data": {}} + }, + "steps": [ + {"run": "_load.json"}, + {"scroll_to": {"id": "draft"}}, + {"type": {"into": {"id": "draft"}, "text": "see you at noon", "clear": true}}, + {"scroll_to": "Send"}, + {"click": "Send"}, + {"wait": {"text": ["Check the room and the text"]}, "timeout": 5}, + {"scroll_to": [{"contains": "see you at noon"}, {"text": "Confirm", "after": {"contains": "Room: Trip"}}]}, + {"expect": {"calls": [{"service": "matrix.send_message", "count": 0}]}} + ] +} +``` + +## Top-level fields + +| Field | Meaning | Default | +| --- | --- | --- | +| `name` | Shown in results and in `FEEDBACK.md` | the file name | +| `use` | A part (`"_room.json"`) or a list of them: their `host` replies, `storage` and shared fields fill in what this scenario does not set | none | +| `sizes` (or `size`) | Window sizes, `WxH` in layout points; the scenario runs once per size from a fresh launch. An empty list is refused: leave the key out for the default | `412x892`, card-host's default | +| `viewport` | `[x, y, w, h]` that must hold what `in_view` lists | the whole window | +| `storage` | Files written into the app's jail before it starts. A string is written as the file's text; any other JSON value (an object, an array, a number) is written as its JSON. A name is a path inside the jail: no absolute path, `..`, backslash or glob character | none | +| `host` | Replies for `host.request`, by service name; see below | every call answers `no service answers "" on this device`, as card-host does | +| `host_profile` | What a request for a service the manifest does not declare gets: `octosense` answers it, as the current OctoSense hosts do (capability names are disclosures there), and the run lists it as undisclosed; `rinx` refuses it with Rinx's `Mini app was not granted ` | `octosense` | +| `fake_host` | `false` runs the bundle as it is, with card-host's real (empty) services | `true` | +| `startup_timeout` | Seconds to wait for the first widgets, at most 600 | 60 | +| `allow_runtime_errors` | `true` stops a runtime error in the log from failing the scenario | `false` | +| `description`, `note` | Free text for people; the runner ignores it | none | + +Every key in a scenario, a step, a check and a selector is checked when the +scenarios load. An unknown key (`texts` for `text`, `servce` for `service`) or a +value of the wrong kind (`"text": "Kept."` instead of a list, `"false"` in quotes +where `false` is meant, `true` where a count is meant, a size that is not +`WxH`) stops the run with exit status 2 before any app starts: a check the +runner skipped, or a switch it read the wrong way, would otherwise pass +without testing anything. The same holds for what JSON allows but nothing +after it can use: `NaN` or `Infinity`, a lone surrogate escape such as +`\ud800`, nesting more than 64 deep, an empty text in a check or a selector +(every widget contains it). Every time (`sleep`, `timeout`, +`startup_timeout`, `delay`) is seconds from 0 to 600, since a larger number is +almost always milliseconds; a file name must fit the filesystem (255 bytes a +part) and hold no control character, and `@`, the run's evidence +directory, must fit too. A bundle that contains a symbolic link is +refused too: the runner never writes through a link. + +Size the app the way its hosts do. A shell that gives the app part of its +window gives it a smaller rectangle than the window: measure that rectangle +from a screenshot of the shell and test that size. + +## Shared parts + +A part is a JSON file whose name starts with `_`. A scenario takes its fixtures +with `"use"`, and its steps with a `{"run": "_part.json"}` step, which the +runner replaces by the part's `steps`. Keep one part for the host replies of +the normal path (`_room.json`) and one for each sequence many scenarios repeat +(`_load.json`, `_confirm.json`). A scenario then states only what differs: +the one reply that fails, the steps after it, the checks. A part may hold both +fixtures and steps; it cannot `use` another part, and its keys are checked like +a scenario's. A part's name is a path inside the directory of the file that +names it, with the `storage` name rules; a part that resolves outside it, +through `..`, an absolute path or a link, is refused. + +## Host replies + +A service maps to one reply, used for every call, or to a list, used in call +order with the last one repeating. `"*"` answers any service not listed. + +| Reply | The app's callback gets | +| --- | --- | +| `{"data": …}` | `{is_ok: true, data: …, error: ""}` | +| `{"error": "text"}` | `{is_ok: false, data: nil, error: "text"}` | +| `{"never": true}` | No callback: test the app's own timeout or stop button | +| `"delay": seconds` | Wait before the callback (default 0.05, at most 600); use it to test a second click while a request is out | + +A reply has exactly one of `data`, `error` and `never`, and may add `delay` and +a `"note"` for people. `error` is the text the app gets; `never` is `true`. + +A host answers each request once ([SCRIPT-API](../../../docs/SCRIPT-API.md#host-services-hostrequest)), +so a reply is one callback. + +A service the manifest does not declare is treated as `host_profile` says. On +current OctoSense hosts a capability name is a disclosure, not a gate (see +[CAPABILITIES](../../../docs/CAPABILITIES.md#current-source-policy-and-release-boundary)), +so the fixture answers and the run prints `called without declaring: …`. Rinx +still grants a mini-app only the services it was granted, so a Rinx mini-app's +scenarios set `"host_profile": "rinx"` (in their shared part), and an undeclared +call gets `Mini app was not granted `, whatever the fixture says. + +Copy reply shapes and error texts from the host's source or documentation, and +name the source in the contract. A fixture the real host could never send +tests nothing. Write the failure texts the host really produces, such as +`Mini-app authorization expired` or `no service answers "matrix" on this device`. + +The fake `host` keeps `has` from the real one; `capabilities` returns the +manifest's list as written. The test copy differs from the bundle only in +`main.splash` (and loses a signature its new bytes would not match): the +runner adds no capability and writes nothing into the app's jail beyond the +scenario's `storage`. + +Every call is printed to card-host's log as one line, with whether it was +declared; the `calls` check reads those lines. Each line carries a code that is +new for every run, so the app cannot print a line the runner would take for a +call, and the fake host prints a ready line before the app's first line runs. +A missing ready line, a call line that does not parse, a gap in the call +numbers or a record cut off at the end of the log is broken evidence (exit +status 2), never "no calls": a `"count": 0` or an `excludes` check cannot pass +on evidence the runner could not read. While the app runs, a record still being +written makes a check wait for it. + +## Steps + +| Step | Does | +| --- | --- | +| `{"click": SELECTOR}` | Clicks the centre of the matching widget's visible part. Fails when none is on screen or it lies outside the viewport | +| `{"type": {"into": SELECTOR, "text": "…", "clear": true}}` | Clicks the field, optionally deletes its current text, types | +| `{"key": "ReturnKey"}` | Presses and releases a key (`Escape`, `Tab`, `Backspace`, …) | +| `{"scroll_to": SELECTOR}` | Scrolls until the widget is wholly inside the viewport, at any size; a list scrolls to each in turn. Fails when it never appears or does not fit. `"on": SELECTOR` scrolls over that widget instead of the viewport's centre | +| `{"scroll": {"on": SELECTOR, "dy": 300}}` | Scrolls over a widget by a fixed amount; prefer `scroll_to` | +| `{"expect": CHECKS, "timeout": 2}` | The checks must hold within the timeout | +| `{"wait": CHECKS, "timeout": 10}` | The same, for slower changes | +| `{"sleep": 0.5}` | Waits; prefer `wait` | +| `{"run": "_part.json"}` | The part's steps, in place | + +Any step but `run` may add `"allow_runtime_errors": true` (the log is checked +after every step), and any step a `"note"` that says why it is there. +`timeout` and `sleep` are seconds, at most 600; `scroll_to` tries at most +`"max"` scrolls (default 25, at most 1000). + +A `SELECTOR` is a string, matched against a widget's whole text, or an object: + +| Key | Matches | +| --- | --- | +| `id` | The widget's `name :=` id | +| `text` | Its whole text, trimmed | +| `contains` | Part of its text or of an input's value | +| `type` | Its widget type, such as `TextInput` | +| `after` | Only widgets listed after the one this selector picks (its own `nth` counts): the button in the same card as a text, in a list whose rows have no ids. At most 8 deep | +| `nth` | Which match, from 0, among those on screen now | + +## Checks + +| Check | Holds when | +| --- | --- | +| `"text": ["…"]` | Each text appears within one widget on screen, in its text or input value; a text is never matched across two widgets | +| `"absent": ["…" or SELECTOR]` | No widget on screen shows the text, or matches the selector (`{"type": "Button", "text": "Send"}` when the same words may appear in a sentence) | +| `"in_view": [SELECTOR]` | The widget is wholly inside the viewport and no scroll view cuts it off (see below) | +| `"enabled"` / `"disabled": [SELECTOR]` | The widget exists and is (not) enabled | +| `"count": [{"of": SELECTOR, "n": 3}]` | Exactly n widgets on screen match | +| `"calls": [{"service": "…", "count": 1, "contains": "…", "excludes": "…"}]` | The app made that many calls to the service; `contains` filters by the JSON of the arguments; without `count`, at least one, with or without `excludes`; `excludes` fails if any of those calls sends the text (a name that must not reach a model, say). To require that no call was made, write `"count": 0` | +| `"files": [{"name": "…", "contains": ["…"]}]` | The file at exactly that path in the jail exists and contains each text; `"missing": true` requires its absence. The name follows the `storage` rules; a file of the same name elsewhere (a backup) or a link does not count | + +## What the runner sees + +The runner reads `/snap`, which lists the widgets on screen, the way a person +sees the app: + +- A widget inside a scroll view is listed only while it is scrolled into view, + and a half-hidden one is reported by its visible part, so its cut edge lies + on the scroll view's edge. A whole widget can also sit flush against that + edge. When a widget touches a container's edge, `in_view` and `scroll_to` + settle it by looking: they scroll that container a little toward the edge + and snap again. If the widget grows, part of it was hidden; if it only moves, + or nothing moves because the content ends there, it was whole. The probe is + scrolled back afterwards. +- `text`, `absent` and `count` judge what is on screen now. Scroll to the part + of the app first: an `absent` over an area that is off screen passes without + testing anything. +- `nth` counts what is on screen, so it shifts when the view scrolls; use + `after` or an `id`. +- A `TextInput`'s text is its value, or its placeholder while it is empty. + +A widget can be drawn yet not readable this way (text clipped inside its own +box, colour, overlap); that is for the visual review. + +A step also fails when the log shows the script failing: an error, a panic, a +parse error, or a stop on its instruction budget (older runtimes stop a script +there; current ones only count), or when card-host has exited. + +After every run the runner copies the app's jail to `/@/jail/`: +evidence of what the app wrote, and the seed for a restart scenario's `storage`. + +`/summary.json` belongs to one invocation. It is rewritten when a run +starts (`"status": "running"`) and after every scenario, and always ends as +`passed`, `failed`, `environment_error` (no display, a host that exited or would not +close, unreadable evidence; such a run writes `ENVIRONMENT.md`, not `FEEDBACK.md`) or `aborted` (a scenario file or the card-host +could not be used, or the run was interrupted), with a `finished` time. It carries a `run_id`, the runner version, digests of the +bundle and of the card-host binary, and each result carries the digest of the +scenario as it ran. A summary left by an earlier run is never mistaken for this +one. diff --git a/skills/splash-app-verify/references/SHAPE.md b/skills/splash-app-verify/references/SHAPE.md new file mode 100644 index 00000000..3abd75fe --- /dev/null +++ b/skills/splash-app-verify/references/SHAPE.md @@ -0,0 +1,49 @@ +# The narrow shape of a script app + +Write every script app in this shape. Each rule closes off one class of bug +that otherwise turns up late: on a phone, in review, or after a person has lost +their text. The worked example is [examples/titled-notes](../examples/titled-notes/bundle/main.splash); +read it once, it is short. + +```text +1. state top-level lets only +2. decisions plain functions: data in, data out, no ui +3. storage one versioned file, validated on read +4. actions change state, then sync once +5. host calls one function per service; every reply handled; stale replies ignored +6. sync the only functions that write to the screen + root view named widgets; lists in the documented empty-state form +``` + +## Rules and what each one prevents + +| Rule | Prevents | Source | +| --- | --- | --- | +| Keep state in top-level `let`s. Name the request token, the busy flag and the status text there, not in closures. | A handler that reads a value an older closure captured; a busy flag that never resets | [SCRIPT-API: shape of a program](../../../docs/SCRIPT-API.md#shape-of-a-program) | +| Put decisions in plain functions that take and return data and never touch `ui`. | Logic that can only be reached by clicking; a rule changed in one handler and not another | Scenarios can then check one rule through one action | +| Write to the screen only from a few `sync_*` functions, called after state changes. | Two handlers updating the same label differently; a list that is not re-rendered after a change | [MODEL-VALIDATION: partial start-up counted as success](../../../docs/MODEL-VALIDATION.md#recover-from-observed-model-weaknesses) | +| Keep one function per host service. It sets busy, sends, and handles each reply in the contract: `is_ok` false with each documented error, a reply after the request was replaced, no reply at all (a timer that ends the wait). | The spinner that never stops; a late reply overwriting newer work; an English host error shown to a Chinese user; "Send" that never sends | [SCRIPT-API: host services](../../../docs/SCRIPT-API.md#host-services-hostrequest), [HOST-SERVICES](../../../docs/HOST-SERVICES.md) | +| Map each host error to a sentence the person can act on, and keep the app usable without the service. | A dead end when the host has no model, no account or no permission | [AGENTS: every AI feature is optional](../../../AGENTS.md#rules-for-every-app) | +| Translate host errors in one function that takes the service as well as the text. The failing service is the sentence's subject; an error it does not know keeps a prefix that names that service. Include rate limits (`429`, `rate limit`, `too many requests`). | A model failure reported as "could not read the chat"; a raw English error or rate-limit message | OnCue 0.5.2, tested with this skill on 2026-10-10: one shared mapper with a chat-read fallback mislabelled model and send errors | +| Keep that function pure. A side effect of an error (dropping a consent, ending a session) happens in the caller, followed by a sync. | State that changed without the screen showing it: "Allowed" still displayed after the grant was dropped | OnCue 0.5.2, the same run | +| Derive every place that shows a state from the same variable, and clear them together. | A status line still saying "press Confirm" after the confirmation was cancelled | OnCue 0.5.2, the same run | +| Store one versioned file. Validate it on read; if it fails, say so, change nothing, and write nothing over it until the person decides. Write only after the in-memory state is valid. | A corrupt or older file wiping the person's data; a half-written state | — | +| Give every widget a scenario or function touches a `name :=` id, and keep button texts stable. | Tests that break on a reworded label; `ui.x` not found after a wrapper was added | [SCRIPT-API: `ui.`](../../../docs/SCRIPT-API.md#uiid-reading-and-changing-widgets) | +| Draw lists as `if list.len() == 0 { Empty }` then `for …`, two statements. | An empty state that never shows, stale rows that stay | [SCRIPT-API: gotchas](../../../docs/SCRIPT-API.md#gotchas) | +| Lay out with `Fill`/`Fit`, rows that wrap (`flow: Right{wrap: true}`) and one scroll container. Write a height budget per region for the smallest size in the brief. No fixed heights around text. | Buttons pushed off the first screen; text clipped at a narrow width; a layout that only works at the size it was written at | Scenarios with `in_view` at each size | +| Use only documented APIs; `//` comments; `#x` colors; no reserved word as a name; no top-level `tick`. | Invented methods (`contains`, `map`, `range`), init stopped halfway by a `#` line, a parse error | [SCRIPT-API: data and strings, gotchas](../../../docs/SCRIPT-API.md#data-and-strings) | +| Declare every capability the app uses, and only those; no secrets. | An undisclosed call (OctoSense treats the list as a disclosure) or a refused one (Rinx grants only what it lists); a store listing that asks for more than the app uses | [CAPABILITIES](../../../docs/CAPABILITIES.md) | + +## Order of work + +Build in the order of the contract, and keep each step green before the next: + +1. the root view and the first screen, with its data loaded (`01-first-screen`); +2. the main action and its saved result; +3. that action's failures, one scenario each; +4. restart and the stored state; +5. the next action. + +A model that writes the whole app in one pass and then debugs it spends its +turns on failures that interact. One path at a time keeps each failure small +and its `FEEDBACK.md` precise. diff --git a/skills/splash-app-verify/scripts/setup.sh b/skills/splash-app-verify/scripts/setup.sh new file mode 100755 index 00000000..a129b689 --- /dev/null +++ b/skills/splash-app-verify/scripts/setup.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +# Prepare a machine to run uitest.py: a card-host build and, on Linux without a +# display, Xvfb with Mesa's software OpenGL. Safe to run again. +# +# skills/splash-app-verify/scripts/setup.sh [path/to/OctoSense-App-Hub] +# +# It installs system packages only with your confirmation (apt-get), builds +# card-host with cargo when none is found, and finishes with `uitest.py doctor`. +set -euo pipefail + +here="$(cd "$(dirname "$0")" && pwd)" +hub="${1:-${OCTOSENSE_APP_HUB:-}}" +if [ -z "$hub" ]; then + for c in "$here/../../../../OctoSense-App-Hub" "$here/../../../../octosense-app-hub"; do + [ -d "$c" ] && hub="$(cd "$c" && pwd)" && break + done +fi + +if [ "$(uname -s)" = "Linux" ]; then + need=() + command -v Xvfb >/dev/null || need+=(xvfb xauth) + command -v pkg-config >/dev/null || need+=(pkg-config) + # Build-time headers for card-host; at run time only Xvfb and Mesa are used. + dpkg -s libwayland-dev >/dev/null 2>&1 || need+=(libwayland-dev libxkbcommon-dev libx11-dev libxcursor-dev libxrandr-dev libxi-dev libasound2-dev libpulse-dev libgl-dev libegl-dev libdrm-dev libssl-dev) + dpkg -s libgl1-mesa-dri >/dev/null 2>&1 || need+=(libgl1-mesa-dri) + if [ ${#need[@]} -gt 0 ]; then + echo "Packages to install: ${need[*]}" + read -r -p "Install them with apt-get? [y/N] " ok + if [ "${ok:-n}" = "y" ]; then + sudo apt-get update -q && sudo apt-get install -y -q "${need[@]}" + else + echo "Skipped; uitest.py doctor will say what is missing." + fi + fi +fi + +if [ -z "${OCTO_CARD_HOST:-}" ] && ! command -v card-host >/dev/null; then + if [ -n "$hub" ] && [ -x "$hub/target/release/card-host" ]; then + echo "card-host: $hub/target/release/card-host" + elif [ -n "$hub" ]; then + echo "Building card-host in $hub (first build takes a while)…" + (cd "$hub" && cargo build --release -p octosense-card-host) + else + echo "No OctoSense-App-Hub checkout found: pass its path, or set OCTO_CARD_HOST." >&2 + fi +fi + +# A script cannot set a variable in the shell that ran it, so record the +# card-host where uitest.py looks for it in every later shell. +found="${OCTO_CARD_HOST:-}" +[ -z "$found" ] && [ -n "$hub" ] && [ -x "$hub/target/release/card-host" ] && found="$hub/target/release/card-host" +[ -z "$found" ] && found="$(command -v card-host || true)" +if [ -n "$found" ]; then + conf="${XDG_CONFIG_HOME:-$HOME/.config}/splash-app-verify/card-host" + mkdir -p "$(dirname "$conf")" + printf '%s\n' "$(cd "$(dirname "$found")" && pwd)/$(basename "$found")" > "$conf" + echo "Recorded card-host in $conf; uitest.py finds it from any shell (OCTO_CARD_HOST or --card-host still win)." +fi +python3 "$here/uitest.py" doctor diff --git a/skills/splash-app-verify/scripts/test_uitest.py b/skills/splash-app-verify/scripts/test_uitest.py new file mode 100644 index 00000000..9df6b44f --- /dev/null +++ b/skills/splash-app-verify/scripts/test_uitest.py @@ -0,0 +1,1168 @@ +"""uitest regressions that need no card-host: fixtures, the fake host source, selectors, checks, feedback.""" +import json +import signal +import subprocess +import sys +import tempfile +import types +import unittest +from pathlib import Path +from unittest import mock + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import uitest # noqa: E402 + +# Any executable file stands in for card-host where the host itself is faked; +# this one exists on every system the tests run on. +FAKE_CARD_HOST = sys.executable + + +def link_or_skip(test, link, target): + """Make a symbolic link, or skip: Windows allows them only with a privilege.""" + try: + link.symlink_to(target, target_is_directory=Path(target).is_dir()) + except (OSError, NotImplementedError) as e: + test.skipTest(f"this system cannot make symbolic links here: {e}") + +WIDGETS = [ + {"i": "title", "ty": "Label", "r": [16, 16, 380, 30], "t": "Titled Notes"}, + {"i": "entry", "ty": "TextInput", "r": [16, 60, 300, 40], "t": "", "value": "Call the plumber"}, + {"i": "keep_button", "ty": "ButtonFlat", "r": [320, 60, 76, 40], "t": "Keep", "enabled": True}, + {"i": "title_button", "ty": "ButtonFlat", "r": [16, 900, 160, 40], "t": "Suggest a title", "enabled": False}, + {"i": "", "ty": "Label", "r": [16, 120, 380, 20], "t": "No notes yet."}, +] + + +class FakeApp: + """A screen that never scrolls: probing a flush widget finds nothing moves.""" + + def __init__(self, widgets, calls=(), files=None): + self.widgets, self._calls, self.files = widgets, list(calls), files or {} + + def snap(self): + return self.widgets + + def get(self, route, **params): + assert route == "/m" + + def calls(self): + return self._calls + + def find_file(self, name): + return self.files.get(name) + + +class Fixtures(unittest.TestCase): + def test_reply_forms(self): + out = uitest.normalise_replies({ + "a.b": {"data": {"x": 1}}, + "c.d": [{"error": "nope", "delay": 1}, {"never": True}], + }) + by = {e["service"]: e["replies"] for e in out} + self.assertEqual(by["a.b"][0]["reply"], {"is_ok": True, "data": {"x": 1}, "error": ""}) + self.assertEqual(by["c.d"][0]["reply"]["is_ok"], False) + self.assertEqual(by["c.d"][0]["delay"], 1.0) + self.assertTrue(by["c.d"][1]["never"]) + + def test_bad_reply_is_a_scenario_error(self): + # A host answers each request once, so a streamed reply is not a fixture. + for bad in ("yes", {"stream": [{"data": 1}]}, {"dta": 1}, []): + with self.assertRaises(uitest.ScenarioError): + uitest.normalise_replies({"a.b": bad}) + + def test_shim_profiles(self): + # OctoSense treats the manifest as a disclosure; Rinx refuses what it did not grant. + src = uitest.shim_source({}, "octosense", ["storage"], "n0nce") + head = src.split("fn uitest_answer")[0] + self.assertIn('print("UITEST_READY_n0nce\\n")', head) # ready before the app's first line + self.assertIn('print("UITEST_CALL_n0nce " + {seq: uitest_seq, service: service', src) + self.assertNotIn("fs.", src) # the app's storage is left alone + self.assertIn('\\"profile\\": \\"octosense\\"', src) + rinx = uitest.shim_source({}, "rinx", ["matrix.read_messages"], "n0nce") + self.assertIn('error: "Mini app was not granted " + service', rinx) + self.assertIn('\\"declared\\": [\\"matrix.read_messages\\"]', rinx) + self.assertIn("capabilities: fn(){ uitest_fx.declared }", rinx) + with self.assertRaises(uitest.ScenarioError): + uitest.shim_source({}, "android", []) + + def test_shim_embeds_fixtures_as_one_string(self): + src = uitest.shim_source({"x.y": {"data": {"say": 'he said "hi"\\n'}}}) + self.assertTrue(src.startswith(uitest.SHIM_BEGIN)) + self.assertIn("let host = {request: fn(service, args, callback){", src) + literal = src.split("let uitest_fx = ", 1)[1].split(".parse_json()", 1)[0] + # Undo the Splash escapes and the JSON must round-trip. + text, i, body = [], 0, literal[1:-1] + while i < len(body): + if body[i] == "\\": + text.append({"n": "\n", "r": "\r", "t": "\t"}.get(body[i + 1], body[i + 1])) + i += 2 + else: + text.append(body[i]) + i += 1 + text = "".join(text) + self.assertEqual(json.loads(text)["services"][0]["service"], "x.y") + + def test_splash_string_escapes(self): + self.assertEqual(uitest.splash_string('a"b\\c\nd'), '"a\\"b\\\\c\\nd"') + + def test_prepare_copy_leaves_the_original(self): + with tempfile.TemporaryDirectory() as t: + bundle = Path(t) / "bundle" + bundle.mkdir() + (bundle / "main.splash").write_text("let x = 1\n", encoding="utf-8") + (bundle / "manifest.json").write_text(json.dumps({"id": "a", "capabilities": ["model"], + "integrity": {"bundle_blake3": "", "signature": {"k": 1}}}), encoding="utf-8") + work = Path(t) / "work" + work.mkdir() + copy, lines = uitest.prepare_copy(bundle, work, {"host": {}}, "n0nce") + self.assertEqual((bundle / "main.splash").read_text(encoding="utf-8"), "let x = 1\n") + self.assertTrue((copy / "main.splash").read_text(encoding="utf-8").endswith("let x = 1\n")) + self.assertIn("UITEST_READY_n0nce", (copy / "main.splash").read_text(encoding="utf-8")) + self.assertGreater(lines, 10) + manifest = json.loads((copy / "manifest.json").read_text(encoding="utf-8")) + self.assertEqual(manifest["capabilities"], ["model"]) # nothing added: no storage for the runner + self.assertNotIn("signature", manifest["integrity"]) + work2 = Path(t) / "w2" + work2.mkdir() + copy2, lines2 = uitest.prepare_copy(bundle, work2, {"fake_host": False}) + self.assertEqual(lines2, 0) + self.assertEqual((copy2 / "main.splash").read_text(encoding="utf-8"), "let x = 1\n") + (bundle / "manifest.json").write_text(json.dumps({"id": "../escape", "capabilities": []}), encoding="utf-8") + with self.assertRaises(uitest.ScenarioError): + uitest.prepare_copy(bundle, Path(t) / "w3", {}) + + +class Checks(unittest.TestCase): + view = [0, 0, 412, 892] + + def test_selectors(self): + self.assertEqual(uitest.find(WIDGETS, "Keep")[1]["i"], "keep_button") + self.assertEqual(uitest.find(WIDGETS, {"id": "entry"})[1]["ty"], "TextInput") + self.assertEqual(uitest.find(WIDGETS, {"contains": "plumber"})[1]["i"], "entry") + self.assertIsNone(uitest.find(WIDGETS, "Kee")[1]) + self.assertEqual(len(uitest.find(WIDGETS, {"type": "ButtonFlat"})[0]), 2) + + def test_after_picks_the_button_in_the_same_card(self): + rows = [{"i": "", "ty": "Label", "r": [0, 0, 300, 20], "t": "Route A: noon"}, + {"i": "", "ty": "Button", "r": [0, 24, 80, 28], "t": "Use"}, + {"i": "", "ty": "Label", "r": [0, 60, 300, 20], "t": "Route B: evening"}, + {"i": "", "ty": "Button", "r": [0, 84, 80, 28], "t": "Use"}] + _, w = uitest.find(rows, {"text": "Use", "after": {"contains": "Route B"}}) + self.assertEqual(w["r"][1], 84) + self.assertIsNone(uitest.find(rows, {"text": "Use", "after": {"contains": "Route C"}})[1]) + + def test_after_counts_from_the_anchor_itself_not_an_equal_widget(self): + # Two cards with the same heading report equal widgets; "after" the second must start after the second. + head = {"i": "", "ty": "Label", "r": [0, 0, 300, 20], "t": "Today"} + rows = [head, {"i": "", "ty": "Button", "r": [0, 24, 80, 28], "t": "Use", "value": "x"}, + dict(head), {"i": "", "ty": "Button", "r": [0, 84, 80, 28], "t": "Use", "value": "y"}] + _, w = uitest.find(rows, {"text": "Use", "after": {"text": "Today", "nth": 1}}) + self.assertEqual(w["value"], "y") + + class Pane: + """A scroll view at y 50..440; /snap lists only the visible part of each child, as card-host does.""" + + def __init__(self, items, content_bottom): + self.items, self.offset = items, 0.0 + self.max_offset = max(0.0, content_bottom - 440) + self.scrolls = 0 + + def snap(self): + out = [{"i": "app_scroll", "ty": "View", "r": [8, 50, 938, 390], "t": ""}] + for i, t, (x, y, w, h) in self.items: + top, bottom = max(y - self.offset, 50), min(y - self.offset + h, 440) + if bottom > top: + out.append({"i": i, "ty": "Button", "r": [x, top, w, bottom - top], "t": t}) + return out + + def get(self, route, **params): + assert route == "/m" and params["k"] == "scroll" + self.scrolls += 1 + self.offset = min(self.max_offset, max(0.0, self.offset + float(params["dy"]) * 0.5)) + + def test_a_widget_cut_off_by_a_scroll_view_is_not_in_view(self): + pane = self.Pane([("go", "Go", (18, 342, 89, 28)), ("cue_draft", "draft", (67, 433, 869, 70))], 560) + widgets = pane.snap() + cut = uitest.find(widgets, {"id": "cue_draft"})[1] + self.assertEqual(cut["r"][3], 7) # only 7 points of it show + uitest.check_expectation(pane, {"in_view": ["Go"]}, [0, 0, 954, 448]) + with self.assertRaises(uitest.StepFailed) as e: + uitest.check_expectation(pane, {"in_view": [{"id": "cue_draft"}]}, [0, 0, 954, 448]) + self.assertIn("cuts it off", e.exception.actual) + self.assertEqual(pane.offset, 0.0) # the probe scrolled back + + def test_a_whole_widget_flush_with_the_edge_is_in_view(self): + # Content ends exactly at the scroll view's edge: nothing scrolls, so nothing is hidden. + end = self.Pane([("send", "Send", (18, 412, 89, 28))], 440) + uitest.check_expectation(end, {"in_view": ["Send"]}, [0, 0, 954, 448]) + # More content below, but this widget is whole: scrolling moves it without growing it. + more = self.Pane([("send", "Send", (18, 412, 89, 28)), ("next", "Next", (18, 460, 89, 28))], 520) + uitest.check_expectation(more, {"in_view": ["Send"]}, [0, 0, 954, 448]) + self.assertEqual(more.offset, 0.0) + + def test_calls_excludes(self): + app = FakeApp(WIDGETS, calls=[{"service": "model.complete", "args": {"text": "[[1, \"noon\"]]"}}]) + uitest.check_expectation(app, {"calls": [{"service": "model.complete", "excludes": "ann"}]}, self.view) + with self.assertRaises(uitest.StepFailed): + uitest.check_expectation(app, {"calls": [{"service": "model.complete", "excludes": "noon"}]}, self.view) + + def test_text_is_never_matched_across_two_widgets(self): + app = FakeApp([{"i": "", "ty": "Label", "r": [0, 0, 100, 20], "t": "first"}, + {"i": "", "ty": "Label", "r": [0, 30, 100, 20], "t": "second"}]) + with self.assertRaises(uitest.StepFailed): + uitest.check_expectation(app, {"text": ["first\nsecond"]}, self.view) + uitest.check_expectation(app, {"absent": ["first\nsecond"]}, self.view) + + def test_absent_takes_a_selector(self): + app = FakeApp(WIDGETS + [{"i": "", "ty": "Label", "r": [16, 150, 380, 20], "t": "Keep a note to see more"}]) + uitest.check_expectation(app, {"absent": [{"type": "Label", "text": "Keep"}]}, self.view) + with self.assertRaises(uitest.StepFailed): + uitest.check_expectation(app, {"absent": [{"type": "ButtonFlat", "text": "Keep"}]}, self.view) + + def test_a_whole_widget_flush_with_a_still_root_is_in_view(self): + root = {"i": "root", "ty": "SolidView", "r": [0, 0, 412, 892], "t": ""} + button = {"i": "go", "ty": "Button", "r": [16, 0, 80, 40], "t": "Go"} + uitest.check_expectation(FakeApp([root, button]), {"in_view": ["Go"]}, root["r"]) + + def test_text_and_absent(self): + app = FakeApp(WIDGETS) + uitest.check_expectation(app, {"text": ["No notes yet.", "plumber"], "absent": ["Kept."]}, self.view) + with self.assertRaises(uitest.StepFailed) as e: + uitest.check_expectation(app, {"text": ["Kept."]}, self.view) + self.assertIn("not on screen", e.exception.actual) + + def test_in_view_reports_the_overflow(self): + app = FakeApp(WIDGETS) + uitest.check_expectation(app, {"in_view": ["Keep"]}, self.view) + with self.assertRaises(uitest.StepFailed) as e: + uitest.check_expectation(app, {"in_view": ["Suggest a title"]}, self.view) + self.assertIn("bottom by 48", e.exception.actual) + + def test_enabled_count_calls_files(self): + with tempfile.TemporaryDirectory() as t: + f = Path(t) / "notes.json" + f.write_text('{"notes":["a"]}', encoding="utf-8") + app = FakeApp(WIDGETS, calls=[{"service": "model.complete", "args": {"input": {"note": "a"}}}], + files={"notes.json": f}) + uitest.check_expectation(app, {"enabled": ["Keep"], "disabled": ["Suggest a title"], + "count": [{"of": {"type": "ButtonFlat"}, "n": 2}], + "calls": [{"service": "model.complete", "count": 1, "contains": "note"}], + "files": [{"name": "notes.json", "contains": ["notes"]}, + {"name": "other.json", "missing": True}]}, self.view) + for bad in ({"enabled": ["Suggest a title"]}, {"calls": [{"service": "mail.send"}]}, + {"files": [{"name": "notes.json", "missing": True}]}, + {"count": [{"of": {"type": "ButtonFlat"}, "n": 3}]}): + with self.assertRaises(uitest.StepFailed): + uitest.check_expectation(app, bad, self.view) + + def test_feedback_has_the_template_parts(self): + text = uitest.feedback({"scenario": "keep", "size": "412x892", "viewport": self.view, + "source_sha256": "abc", "file": "tests/02-keep.json", + "steps_run": [{"click": "Keep"}], "expected": 'text "Kept." is shown', + "actual": "not on screen", "detail": {"similar": []}, + "evidence": {"snap": "snap.json", "log": "card-host.log"}}, "/app/bundle", "/bin/card-host") + for part in ("Artifact:", "Runtime:", "Start state:", "Steps", "Expected:", "Actual:", "Evidence:", "Fix:", "Retest:"): + self.assertIn(part, text) + + +class Scenarios(unittest.TestCase): + def test_examples_are_valid(self): + root = Path(__file__).resolve().parent.parent / "examples" + files = sorted(root.glob("*/tests/*.json")) + self.assertTrue(files) + for f in files: + for _, sc in uitest.load_scenarios(f): + uitest.normalise_replies(sc.get("host")) + for step in sc["steps"]: + # "run" steps are already expanded here; every other documented action counts. + self.assertEqual(len(set(step) & uitest.STEP_ACTIONS), 1, f) + + def test_broken_scenario_is_reported(self): + for body in ('{"name": "no steps"}', '{"steps": [{"clik": "Keep"}]}', '{"steps": [{"run": "_missing.json"}]}'): + with tempfile.TemporaryDirectory() as t: + (Path(t) / "x.json").write_text(body, encoding="utf-8") + with self.assertRaises(uitest.ScenarioError): + uitest.load_scenarios(t) + + def test_parts_share_fixtures_and_steps(self): + with tempfile.TemporaryDirectory() as t: + d = Path(t) + (d / "_room.json").write_text(json.dumps({ + "sizes": ["412x892", "954x448"], + "host": {"a.read": {"data": {"rows": [1]}}, "a.info": {"data": {"id": "r"}}}, + "storage": {"s.json": "{}"}, + "steps": [{"click": "Load"}, {"wait": {"text": ["1 row"]}}]}), encoding="utf-8") + (d / "02-x.json").write_text(json.dumps({ + "use": "_room.json", + "host": {"a.read": {"error": "expired"}}, + "steps": [{"run": "_room.json"}, {"expect": {"text": ["expired"]}, "note": "why"}]}), encoding="utf-8") + (d / "01-plain.json").write_text('{"steps": [{"sleep": 0}]}', encoding="utf-8") + scenarios = uitest.load_scenarios(d) + self.assertEqual([f.name for f, _ in scenarios], ["01-plain.json", "02-x.json"]) + sc = scenarios[1][1] + self.assertEqual(sc["sizes"], ["412x892", "954x448"]) + self.assertEqual(sc["host"]["a.read"], {"error": "expired"}) # the scenario wins + self.assertEqual(sc["host"]["a.info"], {"data": {"id": "r"}}) # the part fills in + self.assertEqual(sc["storage"], {"s.json": "{}"}) + self.assertEqual([list(s)[0] for s in sc["steps"]], ["click", "wait", "expect"]) + + +class Scrolling(unittest.TestCase): + class Scroller: + """A column of widgets under a 400-point viewport; /m scroll moves them by dy * 0.5.""" + + def __init__(self, target_y): + self.offset, self.target_y, self.scrolls = 0.0, target_y, 0 + + def snap(self): + return [{"i": "top", "ty": "Label", "r": [0, 10 - self.offset, 100, 20], "t": "Top"}, + {"i": "go", "ty": "ButtonFlat", "r": [0, self.target_y - self.offset, 80, 28], "t": "Send"}] + + def get(self, route, **params): + assert route == "/m" and params["k"] == "scroll" + self.scrolls += 1 + self.offset = max(0.0, self.offset + float(params["dy"]) * 0.5) + + def test_scrolls_until_in_view_whatever_the_scroll_unit(self): + s = self.Scroller(target_y=1300) + uitest.scroll_into_view(s, "Send", None, [0, 0, 400, 400], 10) + y = s.target_y - s.offset + self.assertTrue(0 <= y and y + 28 <= 400, y) + self.assertLessEqual(s.scrolls, 4) + + def test_reports_when_scrolling_stops(self): + s = self.Scroller(target_y=1300) + s.get = lambda route, **p: None # nothing moves + with self.assertRaises(uitest.StepFailed) as e: + uitest.scroll_into_view(s, "Send", None, [0, 0, 400, 400], 10) + self.assertIn("no longer moves", e.exception.actual) + + +class Safety(unittest.TestCase): + """Scenario files come from the repository under test: they must not reach outside the run.""" + + def test_storage_names_cannot_leave_the_jail(self): + with tempfile.TemporaryDirectory() as t: + jail = Path(t) / "data" / "app" + jail.mkdir(parents=True) + self.assertEqual(uitest.jail_path(jail, "notes.json"), (jail / "notes.json").resolve()) + self.assertEqual(uitest.jail_path(jail, "sub/notes.json"), (jail / "sub/notes.json").resolve()) + for bad in ("../escape.txt", "a/../../escape.txt", "/tmp/escape.txt", "C:\\x", "a\\b", "*.json", "", None): + with self.assertRaises(uitest.ScenarioError, msg=repr(bad)): + uitest.jail_path(jail, bad) + self.assertFalse((Path(t) / "data" / "escape.txt").exists()) + # Refused when the scenario loads, before anything is written. + d = Path(t) / "tests" + d.mkdir() + (d / "01.json").write_text(json.dumps({"storage": {"../../x": "1"}, "steps": [{"sleep": 0}]}), encoding="utf-8") + with self.assertRaises(uitest.ScenarioError): + uitest.load_scenarios(d) + (d / "01.json").write_text(json.dumps({"steps": [{"wait": {"files": [{"name": "/etc/passwd"}]}}]}), encoding="utf-8") + with self.assertRaises(uitest.ScenarioError): + uitest.load_scenarios(d) + + def test_a_link_in_the_bundle_is_refused_before_anything_is_written(self): + with tempfile.TemporaryDirectory() as t: + outside = Path(t) / "outside-source.splash" + outside.write_text("let x = 1\n", encoding="utf-8") + bundle = Path(t) / "bundle" + bundle.mkdir() + link_or_skip(self, bundle / "main.splash", outside) + (bundle / "manifest.json").write_text(json.dumps({"id": "a", "capabilities": []}), encoding="utf-8") + with self.assertRaises(uitest.ScenarioError): + uitest.prepare_copy(bundle, Path(t) / "work", {"host": {}}, "n0nce") + self.assertEqual(outside.read_text(encoding="utf-8"), "let x = 1\n") # untouched + (bundle / "main.splash").unlink() + (bundle / "main.splash").write_text("let x = 1\n", encoding="utf-8") + (bundle / "assets").mkdir() + link_or_skip(self, bundle / "assets" / "up", Path(t)) # a directory link counts too + with self.assertRaises(uitest.ScenarioError): + uitest.prepare_copy(bundle, Path(t) / "work2", {"host": {}}, "n0nce") + + def test_values_of_the_wrong_type_are_refused(self): + bad = [ + {"allow_runtime_errors": "false", "steps": [{"sleep": 0}]}, + {"fake_host": "false", "steps": [{"sleep": 0}]}, + {"sizes": ["412*892"], "steps": [{"sleep": 0}]}, + {"viewport": [0, 0, 412], "steps": [{"sleep": 0}]}, + {"startup_timeout": True, "steps": [{"sleep": 0}]}, + {"steps": [{"click": "Keep", "allow_runtime_errors": "false"}]}, + {"steps": [{"wait": {"files": [{"name": "a.json", "missing": "false", "contains": ["x"]}]}}]}, + {"steps": [{"wait": {"files": [{"name": "a.json", "missing": True, "contains": ["x"]}]}}]}, + {"steps": [{"wait": {"count": [{"of": "Keep", "n": True}]}}]}, + {"steps": [{"wait": {"calls": [{"service": "a.b", "count": False}]}}]}, + {"steps": [{"click": {"text": "Use", "nth": True}}]}, + {"steps": [{"click": {"id": 3}}]}, + {"steps": [{"type": {"into": "entry", "text": "x", "clear": "yes"}}]}, + {"steps": [{"sleep": float("inf")}]}, + {"steps": [{"wait": {"text": ["x"]}, "timeout": "5"}]}, + ] + for body in bad: + with tempfile.TemporaryDirectory() as t: + (Path(t) / "x.json").write_text(json.dumps(body), encoding="utf-8") + with self.assertRaises(uitest.ScenarioError, msg=json.dumps(body)): + uitest.load_scenarios(t) + + def test_unknown_keys_are_refused_not_ignored(self): + bad_steps = [ + {"expect": {"texts": ["Kept."]}}, # a typo would have passed silently + {"expect": {"text": "Kept."}}, # a string, not a list + {"expect": {"calls": [{"servce": "model.complete", "count": 0}]}}, + {"expect": {"calls": [{"service": "model.complete", "cont": "x"}]}}, + {"expect": {"count": [{"of": {"type": "Button"}, "n": "2"}]}}, + {"expect": {"files": [{"name": "a.json", "contain": ["x"]}]}}, + {"click": {"contain": "Keep"}}, + {"click": "Keep", "expect": {"text": ["Kept."]}}, # two actions in one step + {"wait": {"text": ["x"]}, "timeot": 3}, + {"type": {"into": "entry", "txt": "hi"}}, + {"scroll_to": "Send", "on": {"idd": "list"}}, + ] + for step in bad_steps: + with tempfile.TemporaryDirectory() as t: + (Path(t) / "x.json").write_text(json.dumps({"steps": [step]}), encoding="utf-8") + with self.assertRaises(uitest.ScenarioError, msg=json.dumps(step)): + uitest.load_scenarios(t) + with tempfile.TemporaryDirectory() as t: + (Path(t) / "x.json").write_text(json.dumps({"storge": {}, "steps": [{"sleep": 0}]}), encoding="utf-8") + with self.assertRaises(uitest.ScenarioError): + uitest.load_scenarios(t) + (Path(t) / "x.json").write_text(json.dumps({"fake_host": False, + "steps": [{"expect": {"calls": [{"service": "a.b"}]}}]}), encoding="utf-8") + with self.assertRaises(uitest.ScenarioError): + uitest.load_scenarios(t) + (Path(t) / "x.json").write_text(json.dumps({"host_profile": "rinx", "steps": [ + {"click": {"text": "Use", "after": {"contains": "B"}, "nth": 0}, "note": "why"}, + {"wait": {"calls": [{"service": "a.b", "count": 0, "excludes": "x"}]}, "timeout": 3}]}), encoding="utf-8") + self.assertEqual(uitest.load_scenarios(t)[0][1]["host_profile"], "rinx") + + def test_malformed_files_are_scenario_errors_never_crashes(self): + # Found by mutating the shipped scenarios at random: each of these used to + # raise TypeError, ValueError or UnicodeEncodeError instead of a clear + # scenario error (exit status 2), or to be accepted. + ok = '[{"sleep": 0}]' + deep_after = {"text": "x"} + for _ in range(12): + deep_after = {"text": "x", "after": deep_after} + cases = [ + # host and storage of the wrong kind, in the scenario or in a part + '{"host": "x", "steps": %s}' % ok, + '{"host": [1], "steps": %s}' % ok, + '{"host": null, "steps": %s}' % ok, + '{"storage": "0x0", "steps": %s}' % ok, + '{"use": "_bad_host.json", "steps": %s}' % ok, + '{"use": "_bad_storage.json", "steps": %s}' % ok, + # "use" that is not a part name or a list of them + '{"use": 5, "steps": %s}' % ok, + '{"use": [{"a": 1}], "steps": %s}' % ok, + '{"use": [""], "steps": %s}' % ok, + # a typo in a part, however it is used; a part does not take other parts + '{"use": "_typo.json", "steps": %s}' % ok, + '{"steps": [{"run": "_typo.json"}]}', + '{"use": "_nested.json", "steps": %s}' % ok, + '{"steps": [{"run": "_nested.json"}]}', + '{"steps": [{"run": ""}]}', + # replies + '{"host": {"a.b": {"data": 1, "delay": "soon"}}, "steps": %s}' % ok, + '{"host": {"a.b": {"data": 1, "delay": -1}}, "steps": %s}' % ok, + '{"host": {"a.b": {"data": 1, "delay": 3000}}, "steps": %s}' % ok, + '{"host": {"a.b": {"data": 1, "delay": true}}, "steps": %s}' % ok, + '{"host": {"a.b": {"never": false}}, "steps": %s}' % ok, + '{"host": {"a.b": {"data": 1, "error": "x"}}, "steps": %s}' % ok, + '{"host": {"a.b": {"error": 5}}, "steps": %s}' % ok, + '{"host": {"a.b": {"data": 1, "dealy": 2}}, "steps": %s}' % ok, + '{"host": {"": {"data": 1}}, "steps": %s}' % ok, + # text that is not UTF-8 (a lone surrogate escape), as a value or a key + '{"name": "\\ud800", "steps": %s}' % ok, + '{"host": {"a.b": {"data": {"\\udcff": 1}}}, "steps": %s}' % ok, + '{"steps": [{"click": "\\udcff"}]}', + # numbers JSON does not have, and nesting no scenario needs + '{"steps": [{"sleep": NaN}]}', + '{"host": {"a.b": {"data": Infinity}}, "steps": %s}' % ok, + "[" * 100000, + json.dumps({"steps": [{"click": deep_after}]}), + # times: seconds, and never so large that a wait overflows or never ends + '{"steps": [{"sleep": 601}]}', + '{"steps": [{"sleep": 1e300}]}', + '{"steps": [{"wait": {"text": ["x"]}, "timeout": 1e300}]}', + '{"startup_timeout": 1e12, "steps": %s}' % ok, + '{"steps": [{"scroll_to": "x", "max": 1000000000}]}', + # check lists that are not lists + '{"steps": [{"wait": {"count": 5}}]}', + '{"steps": [{"wait": {"calls": 5}}]}', + '{"steps": [{"wait": {"files": {"name": "a"}}}]}', + # storage and file names that name no file, or hold a control character + '{"storage": {".": "x"}, "steps": %s}' % ok, + '{"storage": {"a/": "x"}, "steps": %s}' % ok, + '{"storage": {"a\\u0000b": "x"}, "steps": %s}' % ok, + '{"steps": [{"wait": {"files": [{"name": "a\\nb"}]}}]}', + # names the filesystem cannot hold fail at load, not once the app runs + '{"steps": [{"wait": {"files": [{"name": "%s"}]}}]}' % ("x" * 256), + '{"storage": {"%s": "x"}, "steps": %s}' % ("/".join(["d" * 200] * 6), ok), + ] + parts = {"_bad_host.json": '{"host": [1]}', "_bad_storage.json": '{"storage": "x"}', + "_typo.json": '{"hots": {}, "steps": [{"sleep": 0}]}', + "_nested.json": '{"use": "_bad_host.json", "steps": [{"sleep": 0}]}'} + for text in cases: + with tempfile.TemporaryDirectory() as t: + for name, body in parts.items(): + (Path(t) / name).write_text(body, encoding="utf-8") + (Path(t) / "x.json").write_text(text, encoding="utf-8") + with self.assertRaises(uitest.ScenarioError, msg=text[:120]): + uitest.load_scenarios(Path(t) / "x.json") + + def test_a_jail_the_runner_cannot_read_is_an_evidence_error(self): + with tempfile.TemporaryDirectory() as t: + jail = Path(t) / "jail" + jail.mkdir() + app = type("A", (), {"jail": jail, "find_file": uitest.App.find_file})() + real = Path.is_symlink + def denied(self): + raise PermissionError(13, "Permission denied", str(self)) + Path.is_symlink = denied + try: + with self.assertRaises(uitest.EvidenceError): + app.find_file("notes/a.json") + finally: + Path.is_symlink = real + + def test_strict_loading_keeps_every_documented_form(self): + with tempfile.TemporaryDirectory() as t: + (Path(t) / "_room.json").write_text(json.dumps({ + "host_profile": "rinx", "sizes": ["412x892"], "note": "shared", + "host": {"a.b": [{"data": None}, {"error": "", "delay": 0}, {"never": True, "delay": 600}], + "*": {"error": "down", "note": "everything else"}}, + "storage": {"notes/a.json": {"x": 1}}}), encoding="utf-8") + (Path(t) / "_load.json").write_text(json.dumps({"description": "load", "steps": [{"sleep": 0}]}), encoding="utf-8") + (Path(t) / "x.json").write_text(json.dumps({ + "use": ["_room.json"], "startup_timeout": 600, + "steps": [{"run": "_load.json", "note": "load"}, {"sleep": 600, "allow_runtime_errors": True}, + {"wait": {"text": ["x"]}, "timeout": 0}, + {"scroll_to": "x", "max": 1000}]}), encoding="utf-8") + sc = uitest.load_scenarios(Path(t) / "x.json")[0][1] + self.assertEqual(sc["host_profile"], "rinx") + self.assertEqual(len(sc["steps"]), 4) + by = {e["service"]: e["replies"] for e in uitest.normalise_replies(sc["host"])} + self.assertEqual([r["never"] for r in by["a.b"]], [False, False, True]) + self.assertEqual(by["a.b"][1]["reply"], {"is_ok": False, "data": None, "error": ""}) + + def test_an_unreadable_call_log_is_not_zero_calls(self): + log = {"text": ""} + + class Logged: + nonce = "n0nce" + + def log_text(self): + return log["text"] + + calls = lambda: uitest.App.calls(Logged()) + with self.assertRaises(uitest.EvidenceError): + calls() # no ready line: the fake host never ran + log["text"] = "UITEST_READY_n0nce\n" + self.assertEqual(calls(), []) # ready and no calls: really no calls + log["text"] += 'UITEST_CALL_n0nce {"seq":1,"service":"a.b","args":{},"declared":true}\n' + log["text"] += 'UITEST_CALL_guess {"seq":9,"service":"forged"}\n' # the app cannot guess the nonce + self.assertEqual([c["service"] for c in calls()], ["a.b"]) + log["text"] += 'UITEST_CALL_n0nce {"seq":2,"service":"octos.turn.start","args":{"text":"sec' # being written + with self.assertRaises(uitest.PendingEvidence): # while the host runs: wait, do not count + calls() + with self.assertRaises(uitest.EvidenceError): # after it closed: incomplete evidence + uitest.App.calls(Logged(), final=True) + # A "count 0" or "excludes" check cannot pass on a record that is still being written. + app = FakeApp(WIDGETS) + app.calls = lambda: uitest.App.calls(Logged()) + with self.assertRaises(uitest.StepFailed): + uitest.check_expectation(app, {"calls": [{"service": "octos.turn.start", "excludes": "secret"}]}, [0, 0, 412, 892]) + log["text"] += 'ret"}}\nUITEST_CALL_n0nce {"seq":4,"service":"x"}\n' + with self.assertRaises(uitest.EvidenceError): # seq 3 is missing + calls() + log["text"] = 'UITEST_READY_n0nce\nUITEST_CALL_n0nce {broken\n' + with self.assertRaises(uitest.EvidenceError): + calls() + + class Broken(FakeApp): + def calls(self): + raise uitest.EvidenceError("unreadable") + # A "count 0" or "excludes" check must not pass on evidence it cannot read. + for exp in ({"calls": [{"service": "matrix.send_message", "count": 0}]}, + {"calls": [{"service": "octos.turn.start", "excludes": "secret"}]}): + with self.assertRaises(uitest.EvidenceError): + uitest.check_expectation(Broken(WIDGETS), exp, [0, 0, 412, 892]) + + def test_a_file_check_reads_exactly_the_named_path(self): + with tempfile.TemporaryDirectory() as t: + jail = Path(t) / "data" / "app" + (jail / "backup" / "state").mkdir(parents=True) + (jail / "backup" / "state" / "notes.json").write_text("kept", encoding="utf-8") + app = uitest.App.__new__(uitest.App) + app.jail = jail + app.snap = lambda: [] + self.assertIsNone(app.find_file("state/notes.json")) # a backup of the same name does not count + with self.assertRaises(uitest.StepFailed): + uitest.check_expectation(app, {"files": [{"name": "state/notes.json", "contains": ["kept"]}]}, + [0, 0, 412, 892]) + (jail / "state").mkdir() + (jail / "state" / "notes.json").write_text("kept", encoding="utf-8") + self.assertEqual(app.find_file("state/notes.json"), jail / "state" / "notes.json") + outside = Path(t) / "outside.txt" + outside.write_text("secret", encoding="utf-8") + link_or_skip(self, jail / "link.txt", outside) + self.assertIsNone(app.find_file("link.txt")) # a link is not the app's own file + self.assertNotIn(jail / "link.txt", uitest.app_files(jail)) + + def test_runtime_errors_include_budget_stops_and_parse_errors(self): + for line in ("splash: probe is stopped: its instruction budget is spent", + "splash: an app spent its instruction budget evaluating its body and is stopped", + "splash host: an app spent its instruction budget in a callback and is stopped", + "splash: splash:35135094784:0:0: expected expression", + "[E] splash: boom"): + self.assertTrue(uitest.is_runtime_error(line), line) + self.assertFalse(uitest.is_runtime_error("UITEST_CALL_x {}")) + + +class Housekeeping(unittest.TestCase): + def test_a_failing_evidence_step_still_closes_the_host(self): + closed = [] + + class Host: + def __init__(self, card_host, bundle, data_dir, size, log_path, env, jail=None, nonce=None): + Path(log_path).write_text("", encoding="utf-8") + self.line_offset = 0 + + def wait_ready(self, timeout): + pass + + def new_runtime_errors(self): + return [] + + def calls(self, final=False): + raise uitest.EvidenceError("the call log is unreadable") + + def peak_rss_mb(self): + raise OSError("no /proc") + + def close(self): + closed.append(True) + + with tempfile.TemporaryDirectory() as t: + bundle = Path(t) / "bundle" + bundle.mkdir() + (bundle / "main.splash").write_text("let x = 1\n", encoding="utf-8") + (bundle / "manifest.json").write_text(json.dumps({"id": "a", "capabilities": []}), encoding="utf-8") + real, uitest.App = uitest.App, Host + before = set(Path(tempfile.gettempdir()).glob("uitest-*")) + try: + r = uitest.run_one(FAKE_CARD_HOST, bundle, Path(t) / "01.json", {"steps": [], "storage": {"n.json": "{}"}}, + "412x892", Path(t) / "out", type("D", (), {"env": {}})()) + finally: + uitest.App = real + self.assertEqual(closed, [True]) + self.assertFalse(r["passed"]) # unreadable evidence is not a pass + self.assertTrue(r["environment_error"]) + self.assertIn("unreadable", r["evidence"]["calls_error"]) + self.assertTrue((Path(t) / "out" / "01@412x892" / "result.json").is_file()) + self.assertEqual(set(Path(tempfile.gettempdir()).glob("uitest-*")) - before, set()) + self.assertTrue(any("call log" in n for n in uitest.warnings_for(r))) + + def test_a_failed_rerun_never_leaves_the_last_green_summary(self): + with tempfile.TemporaryDirectory() as t: + bundle = Path(t) / "bundle" + bundle.mkdir() + (bundle / "main.splash").write_text("let x = 1\n", encoding="utf-8") + (bundle / "manifest.json").write_text(json.dumps({"id": "a", "capabilities": []}), encoding="utf-8") + scenario = Path(t) / "case.json" + scenario.write_text('{"steps": [{"sleep": 0}]}', encoding="utf-8") + out = Path(t) / "out" + + class Host: + def __init__(self, *a, **k): + Path(a[4]).write_text("", encoding="utf-8") + self.line_offset = 0 + self.proc = None + + def wait_ready(self, timeout): pass + def new_runtime_errors(self): return [] + def calls(self, final=False): return [] + def peak_rss_mb(self): return None + def close(self): pass + + class NoDisplay: + env = {} + def __enter__(self): return self + def __exit__(self, *a): pass + + real = (uitest.App, uitest.Display, uitest.find_card_host) + uitest.App, uitest.Display, uitest.find_card_host = Host, NoDisplay, lambda *a, **k: FAKE_CARD_HOST + try: + argv = ["run", str(bundle), str(scenario), "--out", str(out)] + self.assertEqual(uitest.main(argv), 0) + first = json.loads((out / "summary.json").read_text(encoding="utf-8")) + self.assertEqual((first["status"], first["passed"]), ("passed", 1)) + scenario.write_text('{"host": {"a.b": {"dta": 1}}, "steps": []}', encoding="utf-8") + self.assertEqual(uitest.main(argv), 2) + finally: + uitest.App, uitest.Display, uitest.find_card_host = real + second = json.loads((out / "summary.json").read_text(encoding="utf-8")) + self.assertEqual(second["status"], "aborted") + self.assertNotEqual(second["run_id"], first["run_id"]) + self.assertEqual(second["passed"], 0) + self.assertIn("dta", second["error"]) + self.assertEqual(len(first["bundle_sha256"]), 64) + + def test_a_host_that_fails_to_close_still_leaves_a_result(self): + class Host: + def __init__(self, *a, **k): + Path(a[4]).write_text("", encoding="utf-8") + self.line_offset = 0 + + def wait_ready(self, timeout): pass + def new_runtime_errors(self): return [] + def calls(self, final=False): return [] + def peak_rss_mb(self): return None + + def close(self): + raise OSError("the process vanished while it was being stopped") + + with tempfile.TemporaryDirectory() as t: + bundle = Path(t) / "bundle" + bundle.mkdir() + (bundle / "main.splash").write_text("let x = 1\n", encoding="utf-8") + (bundle / "manifest.json").write_text(json.dumps({"id": "a", "capabilities": []}), encoding="utf-8") + real, uitest.App = uitest.App, Host + before = set(Path(tempfile.gettempdir()).glob("uitest-*")) + try: + r = uitest.run_one(FAKE_CARD_HOST, bundle, Path(t) / "01.json", {"steps": []}, "412x892", + Path(t) / "out", type("D", (), {"env": {}})()) + finally: + uitest.App = real + self.assertTrue(r["environment_error"]) + self.assertIn("vanished", r["evidence"]["close_error"]) + self.assertTrue((Path(t) / "out" / "01@412x892" / "result.json").is_file()) + self.assertEqual(set(Path(tempfile.gettempdir()).glob("uitest-*")) - before, set()) + + def test_no_display_ends_the_summary_as_an_environment_error(self): + with tempfile.TemporaryDirectory() as t: + bundle = Path(t) / "bundle" + bundle.mkdir() + (bundle / "main.splash").write_text("let x = 1\n", encoding="utf-8") + (bundle / "manifest.json").write_text(json.dumps({"id": "a", "capabilities": []}), encoding="utf-8") + (Path(t) / "case.json").write_text('{"steps": [{"sleep": 0}]}', encoding="utf-8") + + class NoDisplay: + def __enter__(self): + raise RuntimeError("no display and no Xvfb") + def __exit__(self, *a): pass + + real = (uitest.Display, uitest.find_card_host) + uitest.Display, uitest.find_card_host = NoDisplay, lambda *a, **k: FAKE_CARD_HOST + try: + code = uitest.main(["run", str(bundle), str(Path(t) / "case.json"), "--out", str(Path(t) / "out")]) + finally: + uitest.Display, uitest.find_card_host = real + summary = json.loads((Path(t) / "out" / "summary.json").read_text(encoding="utf-8")) + self.assertEqual(code, 2) + self.assertEqual(summary["status"], "environment_error") + self.assertIn("finished", summary) + + def test_every_text_file_the_runner_touches_names_its_encoding(self): + # Windows reads and writes text as cp1252 unless told otherwise, and the + # results carry the scenarios' own (often Chinese) names. + import ast + tree = ast.parse((Path(__file__).resolve().parent / "uitest.py").read_text(encoding="utf-8")) + unnamed = [n.lineno for n in ast.walk(tree) + if isinstance(n, ast.Call) and isinstance(n.func, ast.Attribute) + and n.func.attr in ("read_text", "write_text") + and not any(k.arg == "encoding" for k in n.keywords)] + self.assertEqual(unnamed, [], "uitest.py lines that read or write text in the platform's encoding") + + def test_results_are_utf8_and_a_narrow_console_does_not_stop_the_run(self): + import io + with tempfile.TemporaryDirectory() as t: + bundle = Path(t) / "bundle" + bundle.mkdir() + (bundle / "main.splash").write_text("let x = 1\n", encoding="utf-8") + (bundle / "manifest.json").write_text(json.dumps({"id": "a", "capabilities": []}), encoding="utf-8") + scenario = Path(t) / "case.json" + scenario.write_text(json.dumps({"name": "首屏:载入群聊", "steps": [{"sleep": 0}]}), encoding="utf-8") + out = Path(t) / "out" + + class Host: + def __init__(self, *a, **k): + Path(a[4]).write_text("", encoding="utf-8") + self.line_offset = 0 + def wait_ready(self, timeout): pass + def new_runtime_errors(self): return [] + def calls(self, final=False): return [{"seq": 1, "service": "a.b", "args": {"text": "你好"}}] + def peak_rss_mb(self): return None + def close(self): pass + + class NoDisplay: + env = {} + def __enter__(self): return self + def __exit__(self, *a): pass + + console = io.TextIOWrapper(io.BytesIO(), encoding="cp1252") # a Windows console's code page + real = (uitest.App, uitest.Display, uitest.find_card_host, sys.stdout) + uitest.App, uitest.Display, uitest.find_card_host = Host, NoDisplay, lambda *a, **k: FAKE_CARD_HOST + sys.stdout = console + try: + code = uitest.main(["run", str(bundle), str(scenario), "--out", str(out)]) + finally: + uitest.App, uitest.Display, uitest.find_card_host, sys.stdout = real + self.assertEqual(code, 0) + summary = json.loads((out / "summary.json").read_bytes().decode("utf-8")) + self.assertEqual(summary["results"][0]["scenario"], "首屏:载入群聊") + console.flush() + self.assertIn(b"\\u9996", console.buffer.getvalue()) # shown escaped, not a crash + + def test_an_unreadable_card_host_still_ends_the_summary(self): + with tempfile.TemporaryDirectory() as t: + bundle = Path(t) / "bundle" + bundle.mkdir() + (bundle / "main.splash").write_text("let x = 1\n", encoding="utf-8") + (Path(t) / "case.json").write_text('{"steps": [{"sleep": 0}]}', encoding="utf-8") + real = uitest.find_card_host + uitest.find_card_host = lambda *a, **k: t # a directory: hashing it fails + try: + code = uitest.main(["run", str(bundle), str(Path(t) / "case.json"), "--out", str(Path(t) / "out")]) + finally: + uitest.find_card_host = real + summary = json.loads((Path(t) / "out" / "summary.json").read_text(encoding="utf-8")) + self.assertEqual((code, summary["status"]), (2, "aborted")) + self.assertIn("finished", summary) + + def test_card_host_is_found_by_its_windows_name(self): + with tempfile.TemporaryDirectory() as t: + exe = Path(t) / "target" / "release" / "card-host.exe" + exe.parent.mkdir(parents=True) + exe.write_text("", encoding="utf-8") + exe.chmod(0o755) + old_names, old_env = uitest.CARD_HOST_NAMES, dict(uitest.os.environ) + try: + uitest.CARD_HOST_NAMES = ("card-host.exe", "card-host") + uitest.os.environ.update(OCTOSENSE_APP_HUB=t, XDG_CONFIG_HOME=t, PATH=str(Path(t) / "none")) + uitest.os.environ.pop("OCTO_CARD_HOST", None) + uitest.os.environ.pop("CARGO_TARGET_DIR", None) + self.assertEqual(uitest.find_card_host(None), str(exe.resolve())) + finally: + uitest.CARD_HOST_NAMES = old_names + uitest.os.environ.clear() + uitest.os.environ.update(old_env) + + def test_setup_records_card_host_for_later_shells(self): + with tempfile.TemporaryDirectory() as t: + exe = Path(t) / "card-host" + exe.write_text("#!/bin/sh\n", encoding="utf-8") + exe.chmod(0o755) + old = {k: uitest.os.environ.get(k) for k in ("XDG_CONFIG_HOME", "OCTO_CARD_HOST", "OCTOSENSE_APP_HUB", + "CARGO_TARGET_DIR", "PATH")} + try: + uitest.os.environ["XDG_CONFIG_HOME"] = t + uitest.os.environ["PATH"] = "/nonexistent" + for k in ("OCTO_CARD_HOST", "OCTOSENSE_APP_HUB", "CARGO_TARGET_DIR"): + uitest.os.environ.pop(k, None) + self.assertEqual(uitest.find_card_host(None), None) + uitest.config_file().parent.mkdir(parents=True) + uitest.config_file().write_text(str(exe) + "\n", encoding="utf-8") + found, source = uitest.find_card_host(None, with_source=True) + self.assertEqual(found, str(exe.resolve())) + self.assertIn("setup.sh", source) + finally: + for k, v in old.items(): + if v is None: + uitest.os.environ.pop(k, None) + else: + uitest.os.environ[k] = v + setup = (Path(__file__).resolve().parent / "setup.sh").read_text(encoding="utf-8") + self.assertIn("splash-app-verify/card-host", setup) + + +def tiny_bundle(root): + bundle = Path(root) / "bundle" + bundle.mkdir() + (bundle / "main.splash").write_text("let x = 1\n", encoding="utf-8") + (bundle / "manifest.json").write_text(json.dumps({"id": "a", "capabilities": []}), encoding="utf-8") + return bundle + + +NO_DISPLAY = type("D", (), {"env": {}})() + + +class EnvironmentIsNotTheApp(unittest.TestCase): + """An environment failure must never reach the app's author as a repair request.""" + + def test_a_host_that_exits_at_once_writes_an_environment_report_not_feedback(self): + # The reviewer's reproduction: a valid scenario against a "card-host" that exits + # immediately. The Python interpreter refuses card-host's arguments and exits. + with tempfile.TemporaryDirectory() as t: + bundle = tiny_bundle(t) + before = set(Path(tempfile.gettempdir()).glob("uitest-*")) + r = uitest.run_one(FAKE_CARD_HOST, bundle, Path(t) / "01.json", + {"steps": [{"sleep": 0}], "startup_timeout": 30}, "412x892", Path(t) / "out", NO_DISPLAY) + run_dir = Path(t) / "out" / "01@412x892" + self.assertFalse(r["passed"]) + self.assertTrue(r["environment_error"]) + self.assertIn("card-host exited", r["actual"]) + self.assertFalse((run_dir / "FEEDBACK.md").exists()) + self.assertNotIn("feedback", r) + report = (run_dir / "ENVIRONMENT.md").read_text(encoding="utf-8") + self.assertEqual(r["environment_report"], str(run_dir / "ENVIRONMENT.md")) + self.assertIn("not a repair request", report) + self.assertIn("do not change", report) + self.assertNotIn("Repair request", report) + self.assertNotIn("Fix: change only", report) + self.assertEqual(set(Path(tempfile.gettempdir()).glob("uitest-*")) - before, set()) + + def test_an_app_failure_still_writes_the_repair_request(self): + class Refused: + def __init__(self, *a, **k): + Path(a[4]).write_text("", encoding="utf-8") + self.line_offset = 0 + + def wait_ready(self, timeout): + raise uitest.StepFailed("the bundle is admitted", "card-host refused it", {}) + + def snap(self): return [] + def get(self, *a, **k): raise OSError("no remote") + def peak_rss_mb(self): return None + def calls(self, final=False): return [] + def close(self): pass + + with tempfile.TemporaryDirectory() as t: + bundle = tiny_bundle(t) + with mock.patch.object(uitest, "App", Refused): + r = uitest.run_one(FAKE_CARD_HOST, bundle, Path(t) / "01.json", {"steps": []}, "412x892", + Path(t) / "out", NO_DISPLAY) + run_dir = Path(t) / "out" / "01@412x892" + self.assertFalse(r.get("environment_error", False)) + self.assertIn("Repair request", (run_dir / "FEEDBACK.md").read_text(encoding="utf-8")) + self.assertFalse((run_dir / "ENVIRONMENT.md").exists()) + + def test_the_console_names_the_environment_report(self): + with tempfile.TemporaryDirectory() as t: + bundle = tiny_bundle(t) + scenario = Path(t) / "case.json" + scenario.write_text('{"steps": [{"sleep": 0}], "startup_timeout": 30}', encoding="utf-8") + + class NoDisplay: + env = {} + def __enter__(self): return self + def __exit__(self, *a): pass + + out = [] + with mock.patch.object(uitest, "Display", NoDisplay), \ + mock.patch.object(uitest, "find_card_host", lambda *a, **k: FAKE_CARD_HOST), \ + mock.patch("builtins.print", lambda *a, **k: out.append(" ".join(map(str, a)))): + code = uitest.main(["run", str(bundle), str(scenario), "--out", str(Path(t) / "out")]) + self.assertEqual(code, 2) + text = "\n".join(out) + self.assertIn("ENV ", text) + self.assertIn("environment, not the app", text) + self.assertNotIn("FEEDBACK.md", text) + + +class StoppingAHungHost(unittest.TestCase): + class Proc: + """A card-host that ignores /quit; `stops_on` names the call that finally ends it.""" + pid = 4242 + + def __init__(self, stops_on): + self.stops_on, self.calls, self.stopped = stops_on, [], False + + def poll(self): + return 0 if self.stopped else None + + def wait(self, timeout=None): + if not self.stopped: + raise subprocess.TimeoutExpired("card-host", timeout) + return 0 + + def terminate(self): + self.calls.append("terminate") + self.stopped = self.stopped or self.stops_on == "terminate" + + def kill(self): + self.calls.append("kill") + self.stopped = self.stopped or self.stops_on == "kill" + + def hung_app(self, proc): + app = uitest.App.__new__(uitest.App) + app.proc = proc + app.log_file = types.SimpleNamespace(close=lambda: None) + app.get = mock.Mock(side_effect=OSError("no answer")) + return app + + def test_without_killpg_the_process_itself_is_terminated_then_killed(self): + proc = self.Proc("kill") + windows = types.SimpleNamespace(name="nt") # no killpg, as on Windows + with mock.patch.object(uitest, "os", windows): + uitest.App.close(self.hung_app(proc)) + self.assertEqual(proc.calls, ["terminate", "kill"]) + self.assertTrue(proc.stopped) + + def test_on_posix_the_process_group_gets_sigterm_then_sigkill(self): + proc = self.Proc("never") + sent = [] + + def killpg(pid, sig): + sent.append((pid, sig)) + if sig == signal.SIGKILL: + proc.stopped = True + + posix = types.SimpleNamespace(name="posix", killpg=killpg) + with mock.patch.object(uitest, "os", posix): + uitest.App.close(self.hung_app(proc)) + self.assertEqual(sent, [(4242, signal.SIGTERM), (4242, signal.SIGKILL)]) + self.assertEqual(proc.calls, []) + + def test_a_host_that_survives_everything_is_reported_not_leaked_silently(self): + proc = self.Proc("never") + with mock.patch.object(uitest, "os", types.SimpleNamespace(name="nt")): + with self.assertRaises(RuntimeError) as e: + uitest.App.close(self.hung_app(proc)) + self.assertIn("still running", str(e.exception)) + self.assertEqual(proc.calls, ["terminate", "kill"]) + + +class PrivateDisplay(unittest.TestCase): + class Xvfb: + def __init__(self, exit_code=None): + self.code, self.terminated = exit_code, False + + def poll(self): return self.code + def terminate(self): self.terminated = True + def wait(self, timeout=None): return 0 + def kill(self): pass + + def enter(self, taken, xvfb): + class FakePath(str): + def exists(self): + return taken + + env = {k: v for k, v in uitest.os.environ.items() if k not in ("DISPLAY", "WAYLAND_DISPLAY")} + display = uitest.Display() + display.SOCKET_WAIT = 0.3 + with mock.patch.object(uitest.sys, "platform", "linux"), mock.patch.dict(uitest.os.environ, env, clear=True), \ + mock.patch.object(uitest.shutil, "which", lambda name: "/usr/bin/Xvfb"), \ + mock.patch.object(uitest, "Path", FakePath), \ + mock.patch.object(uitest.subprocess, "Popen", lambda *a, **k: xvfb): + return display.__enter__() + + def test_no_free_display_number_is_an_error_about_the_display(self): + with self.assertRaises(RuntimeError) as e: + self.enter(taken=True, xvfb=self.Xvfb()) + self.assertIn("no free X display", str(e.exception)) + + def test_an_xvfb_that_never_makes_its_socket_is_stopped_and_reported(self): + xvfb = self.Xvfb() + with self.assertRaises(RuntimeError) as e: + self.enter(taken=False, xvfb=xvfb) + self.assertIn("did not create", str(e.exception)) + self.assertTrue(xvfb.terminated) + + def test_an_xvfb_that_exits_is_reported(self): + with self.assertRaises(RuntimeError) as e: + self.enter(taken=False, xvfb=self.Xvfb(exit_code=1)) + self.assertIn("exited with status 1", str(e.exception)) + + +class NothingPassesVacuously(unittest.TestCase): + view = [0, 0, 412, 892] + + def load(self, body, name="case.json", extra=None): + with tempfile.TemporaryDirectory() as t: + for n, b in (extra or {}).items(): + p = Path(t) / n + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(b, encoding="utf-8") + (Path(t) / name).write_text(json.dumps(body), encoding="utf-8") + return uitest.load_scenarios(Path(t) / name) + + def test_excludes_without_count_needs_a_call(self): + nothing = FakeApp(WIDGETS, calls=[]) + with self.assertRaises(uitest.StepFailed): + uitest.check_expectation(nothing, {"calls": [{"service": "model.complete", "excludes": "ann"}]}, self.view) + uitest.check_expectation(nothing, {"calls": [{"service": "model.complete", "count": 0, "excludes": "ann"}]}, + self.view) + + def test_empty_sizes_and_empty_texts_are_refused(self): + bad = [{"sizes": [], "steps": [{"sleep": 0}]}, + {"steps": [{"expect": {"text": [""]}}]}, + {"steps": [{"expect": {"absent": [""]}}]}, + {"steps": [{"expect": {"in_view": [{"contains": ""}]}}]}, + {"steps": [{"click": {"text": ""}}]}, + {"steps": [{"expect": {"calls": [{"service": "a.b", "contains": ""}]}}]}, + {"steps": [{"expect": {"calls": [{"service": "a.b", "excludes": ""}]}}]}, + {"steps": [{"expect": {"files": [{"name": "n.json", "contains": [""]}]}}]}] + for body in bad: + with self.subTest(body=body), self.assertRaises(uitest.ScenarioError): + self.load(body) + ((_, ok),) = self.load({"steps": [{"expect": {"text": ["Kept."]}}]}) + self.assertNotIn("sizes", ok) + + def test_part_names_stay_beside_the_scenarios(self): + part = json.dumps({"host": {"a.b": {"data": 1}}, "steps": [{"sleep": 0}]}) + with tempfile.TemporaryDirectory() as t: + # A valid part one level up: every name below reaches it, and must be refused. + (Path(t) / "_x.json").write_text(part, encoding="utf-8") + inner = Path(t) / "inner" + (inner / "parts").mkdir(parents=True) + (inner / "parts" / "_room.json").write_text(part, encoding="utf-8") + for name in ("../_x.json", str(Path(t) / "_x.json"), "parts/../../_x.json", "..\\_x.json"): + for key in ("use", "run"): + body = {"use": name, "steps": []} if key == "use" else {"steps": [{"run": name}]} + (inner / "case.json").write_text(json.dumps(body), encoding="utf-8") + with self.subTest(name=name, key=key), self.assertRaises(uitest.ScenarioError) as e: + uitest.load_scenarios(inner / "case.json") + self.assertNotIn("cannot read", str(e.exception)) + (inner / "case.json").write_text(json.dumps({"use": "parts/_room.json", + "steps": [{"run": "parts/_room.json"}]}), encoding="utf-8") + ((_, sc),) = uitest.load_scenarios(inner / "case.json") + self.assertEqual(sc["host"], {"a.b": {"data": 1}}) + + def test_a_part_that_links_outside_is_refused(self): + with tempfile.TemporaryDirectory() as outside, tempfile.TemporaryDirectory() as t: + (Path(outside) / "_x.json").write_text('{"steps": []}', encoding="utf-8") + link_or_skip(self, Path(t) / "_x.json", Path(outside) / "_x.json") + (Path(t) / "case.json").write_text('{"steps": [{"run": "_x.json"}]}', encoding="utf-8") + with self.assertRaises(uitest.ScenarioError): + uitest.load_scenarios(Path(t) / "case.json") + + def test_a_name_too_long_for_its_evidence_directory_is_a_scenario_error(self): + with tempfile.TemporaryDirectory() as t: + f = Path(t) / ("n" * 250 + ".json") # a legal file name; "@412x892" is not + try: + f.write_text('{"steps": [{"sleep": 0}]}', encoding="utf-8") + except OSError as e: + self.skipTest(f"this system cannot make a 255-byte file name here: {e}") + with self.assertRaises(uitest.ScenarioError) as e: + uitest.load_scenarios(f) + self.assertIn("too long", str(e.exception)) + + def test_storage_values_any_json(self): + class Host: + def __init__(self, *a, **k): + Path(a[4]).write_text("", encoding="utf-8") + self.line_offset = 0 + + def wait_ready(self, timeout): pass + def new_runtime_errors(self): return [] + def calls(self, final=False): return [] + def peak_rss_mb(self): return None + def close(self): pass + + with tempfile.TemporaryDirectory() as t: + bundle = tiny_bundle(t) + storage = {"a.json": [1, 2], "b.txt": "plain text", "c.json": {"v": 1}, "d.json": 3} + with mock.patch.object(uitest, "App", Host): + r = uitest.run_one(FAKE_CARD_HOST, bundle, Path(t) / "01.json", {"steps": [], "storage": storage}, + "412x892", Path(t) / "out", NO_DISPLAY) + self.assertTrue(r["passed"], r) + jail = Path(t) / "out" / "01@412x892" / "jail" + self.assertEqual((jail / "a.json").read_text(encoding="utf-8"), "[1, 2]") + self.assertEqual((jail / "b.txt").read_text(encoding="utf-8"), "plain text") + self.assertEqual(json.loads((jail / "c.json").read_text(encoding="utf-8")), {"v": 1}) + self.assertEqual((jail / "d.json").read_text(encoding="utf-8"), "3") + + +if __name__ == "__main__": + unittest.main() diff --git a/skills/splash-app-verify/scripts/uitest.py b/skills/splash-app-verify/scripts/uitest.py new file mode 100755 index 00000000..f8231884 --- /dev/null +++ b/skills/splash-app-verify/scripts/uitest.py @@ -0,0 +1,1659 @@ +#!/usr/bin/env python3 +"""uitest: run a Splash script app's scenarios in card-host, the way a backend runs unit tests. + + uitest.py run [--card-host PATH] [--out DIR] [--keep-going] + uitest.py doctor [--card-host PATH] + +A scenario is a JSON file: the host-service replies the app gets, files to put +in its storage jail before it starts, the window sizes to test, and steps that +act on the app and check what it shows. See references/SCENARIOS.md. + +For each scenario and size the runner: + +1. copies the bundle and prepends a fake `host` to the copy's `main.splash` + (the source under test is otherwise byte-for-byte the original), so every + `host.request` gets the scenario's reply instead of needing a real service; +2. starts card-host on the copy with a remote-control port, hidden, and on + Linux under a private Xvfb display when there is none; +3. drives the steps through the remote routes (`/snap`, `/click`, `/t`, `/k`, + `/m`), checks each expectation against the widget list, the calls the app + made, the files in its jail and the runtime log; +4. on the first failure, writes the evidence and FEEDBACK.md: a repair request + in the shape docs/MODEL-VALIDATION.md asks for, scoped to that one failure. + When the environment failed instead (no display, a host that died or would + not stop, evidence the runner cannot read), it writes ENVIRONMENT.md, which + says so and is never a repair request for the app. + +It never edits the bundle under test. Exit status: 0 all passed, 1 a scenario +failed, 2 the environment or a scenario file is broken. + +Python 3.9+, standard library only. +""" +import argparse +import hashlib +import json +import math +import os +import re +import shutil +import signal +import socket +import subprocess +import sys +import tempfile +import time +import urllib.parse +import urllib.request +from pathlib import Path + +HERE = Path(__file__).resolve().parent +SKILL = HERE.parent +DEFAULT_SIZE = "412x892" +RUNTIME_ERROR_MARKERS = ("[E] splash", "on_render closure failed", "splash host callback error", + "isolate panic", "card-host: refused", + # Runtimes before makepad a772b370 (Rinx still ships one) stop a heap + # whose instruction budget is spent, in these words: + "spent its instruction budget", "instruction budget is spent") +SHIM_BEGIN = "// ---- uitest fake host: test copy only ----" +SHIM_END = "// ---- end uitest fake host ----" +CALL_MARK = "UITEST_CALL_" # + run nonce: one line per host.request in the card-host log +READY_MARK = "UITEST_READY_" # + run nonce: the fake host is installed +RUNNER_VERSION = 3 +CARD_HOST_NAMES = ("card-host.exe", "card-host") if os.name == "nt" else ("card-host",) + +# How a host treats a request for a service the manifest does not name. +# octosense: capability names are disclosures, not gates (App Hub e1fdf28, +# makepad d653bc44); the request is answered and the runner +# reports the missing declaration. +# rinx: Rinx grants a mini-app only the services it was granted and +# refuses the rest with this text (Rinx crates/miniapp-core/src/lib.rs:103). +HOST_PROFILES = ("octosense", "rinx") +RINX_REFUSAL = "Mini app was not granted " +HOST_WIDGETS = ("Window", "KeyboardView", "Splash") + +# Every key a scenario may use. An unknown key is a typo, and a typo that the +# runner ignored would make a check pass without running. +SCENARIO_KEYS = {"name", "description", "note", "use", "host", "storage", "steps", "sizes", "size", "viewport", + "startup_timeout", "fake_host", "allow_runtime_errors", "host_profile"} +STEP_ACTIONS = {"click", "type", "key", "scroll", "scroll_to", "sleep", "expect", "wait"} +STEP_MODIFIERS = {"click": {"note", "allow_runtime_errors"}, + "type": {"note", "allow_runtime_errors"}, + "key": {"note", "allow_runtime_errors"}, + "scroll": {"note", "allow_runtime_errors"}, + "scroll_to": {"on", "max", "note", "allow_runtime_errors"}, + "sleep": {"note", "allow_runtime_errors"}, # errors are checked after every step + "expect": {"timeout", "note", "allow_runtime_errors"}, + "wait": {"timeout", "note", "allow_runtime_errors"}} +EXPECT_KEYS = {"text", "absent", "in_view", "enabled", "disabled", "count", "calls", "files"} +SELECTOR_KEYS = {"id", "type", "text", "contains", "after", "nth"} +CALL_KEYS = {"service", "count", "contains", "excludes"} +COUNT_KEYS = {"of", "n"} +FILE_KEYS = {"name", "missing", "contains"} +TYPE_KEYS = {"into", "text", "clear"} +SCROLL_KEYS = {"on", "dy"} +REPLY_FORMS = ("data", "error", "never") +REPLY_KEYS = set(REPLY_FORMS) | {"delay", "note"} +# A part may hold fixtures (taken with "use") and steps (taken with "run"); a +# part does not take other parts, and a typo in one is refused like anywhere else. +PART_KEYS = SCENARIO_KEYS - {"use"} +# Any time in a scenario is seconds, at most this: a larger number is almost +# always milliseconds, and an unbounded one can overflow a wait or never end. +MAX_SECONDS = 600 +MAX_SCROLLS = 1000 +MAX_DEPTH = 64 # nesting of a scenario file, fixtures included +MAX_AFTER = 8 # selectors chained with "after" + + +class ScenarioError(Exception): + """The scenario file itself is wrong (not the app).""" + + +class EvidenceError(RuntimeError): + """The runner cannot read its own evidence (the call log): neither a pass nor the app's fault.""" + + +class StepFailed(Exception): + def __init__(self, expected, actual, detail=None): + super().__init__(expected) + self.expected = expected + self.actual = actual + self.detail = detail or {} + + +class PendingEvidence(StepFailed): + """Evidence still being written: a wait retries, a final read must not accept it.""" + + +# --------------------------------------------------------------- fake host +def splash_string(text): + """A Splash double-quoted string literal for any text.""" + out = ['"'] + for ch in text: + if ch == "\\": + out.append("\\\\") + elif ch == '"': + out.append('\\"') + elif ch == "\n": + out.append("\\n") + elif ch == "\r": + out.append("\\r") + elif ch == "\t": + out.append("\\t") + else: + out.append(ch) + out.append('"') + return "".join(out) + + +def normalise_replies(host): + """{service: reply | [reply, ...]} -> [{service, replies: [reply...]}] with full reply objects. + + A reply is exactly one of {"data": …}, {"error": "…"} or {"never": true}, + with an optional "delay" in seconds. A host answers every request once + (SCRIPT-API: host services), so there is no form that calls back twice. + """ + if not isinstance(host, dict): + raise ScenarioError(f"\"host\" is an object of replies by service, got {_short(host)}") + out = [] + for service, value in host.items(): + if not isinstance(service, str) or not service.strip(): + raise ScenarioError(f"host: a service name is a non-empty string, got {service!r}") + where = f"host reply for {service!r}" + items = value if isinstance(value, list) else [value] + if not items: + raise ScenarioError(f"host replies for {service!r} are an empty list") + replies = [] + for item in items: + if not isinstance(item, dict): + raise ScenarioError(f"{where} must be an object with \"data\", \"error\" or \"never\", " + f"got {_short(item)}") + if "stream" in item: + raise ScenarioError(f"{where}: a host answers each request once; \"stream\" is not a reply form") + _unknown(item, REPLY_KEYS, where, "reply") + if len([k for k in REPLY_FORMS if k in item]) != 1: + raise ScenarioError(f"{where} has exactly one of \"data\", \"error\" or \"never\", got {_short(item)}") + _need("error" not in item or isinstance(item["error"], str), where, "\"error\" is the error text") + _need("never" not in item or item["never"] is True, where, + "\"never\" is true; leave it out for a reply that comes") + delay = item.get("delay", 0.05) + _need(_seconds(delay), where, f"\"delay\" is a number of seconds from 0 to {MAX_SECONDS}") + if "error" in item: + reply = {"is_ok": False, "data": None, "error": item["error"]} + else: + reply = {"is_ok": True, "data": item.get("data"), "error": ""} + replies.append({"never": "never" in item, "delay": float(delay), "reply": reply}) + out.append({"service": service, "replies": replies}) + return out + + +def shim_source(host, profile="octosense", declared=(), nonce="0"): + """The fake host prepended to the test copy. + + `declared` is the manifest's capability list; `profile` says what a request + for an undeclared service gets (see HOST_PROFILES). Every call is printed + to card-host's log as one line, `UITEST_CALL_ {json}`, with whether + it was declared. The log is the host's channel, not the app's storage, so + the app keeps exactly its own permissions, files and quota. The nonce is + new for every run, so the app cannot write a line the runner would take + for a call. A ready line is printed first: without it, the call evidence + is missing, never "no calls". + """ + if profile not in HOST_PROFILES: + raise ScenarioError(f"host_profile must be one of {list(HOST_PROFILES)}, got {profile!r}") + fixtures = json.dumps({"services": normalise_replies(host), "profile": profile, "declared": list(declared)}, + ensure_ascii=False) + return f"""{SHIM_BEGIN} +let uitest_real_host = host +let uitest_fx = {splash_string(fixtures)}.parse_json() +let uitest_asked = [] +print("{READY_MARK}{nonce}\\n") +fn uitest_later(callback, reply, delay){{ start_timeout(delay, || callback(reply)) }} +fn uitest_answer(service, nth){{ + let fallback = nil + for entry in uitest_fx.services {{ + if entry.service == service {{ + let k = nth + if k >= entry.replies.len() {{ k = entry.replies.len() - 1 }} + return entry.replies[k] + }} + if entry.service == "*" {{ fallback = entry.replies[0] }} + }} + if fallback != nil {{ return fallback }} + let family = service.split(".")[0] + {{never: false, delay: 0.05, reply: {{is_ok: false, data: nil, error: "no service answers \\"" + family + "\\" on this device"}}}} +}} +fn uitest_declared(service){{ + let family = service.split(".")[0] + for c in uitest_fx.declared {{ if c == family || c == service {{ return true }} }} + false +}} +let uitest_seq = 0 +let host = {{request: fn(service, args, callback){{ + let nth = 0 + for c in uitest_asked {{ if c == service {{ nth += 1 }} }} + uitest_asked.push(service) + uitest_seq += 1 + let declared = uitest_declared(service) + print("{CALL_MARK}{nonce} " + {{seq: uitest_seq, service: service, args: args, at: time_now(), declared: declared}}.to_json() + "\\n") + // Rinx answers only the services a mini-app was granted; an OctoSense + // host treats the manifest as a disclosure and answers anyway. + if !declared && uitest_fx.profile == "rinx" {{ + uitest_later(callback, {{is_ok: false, data: nil, error: "{RINX_REFUSAL}" + service}}, 0.0) + return uitest_seq + }} + let answer = uitest_answer(service, nth) + if !answer.never {{ uitest_later(callback, answer.reply, answer.delay) }} + uitest_seq +}}, has: fn(name){{ uitest_real_host.has(name) }}, capabilities: fn(){{ uitest_fx.declared }}}} +{SHIM_END} +""" + + +def check_app_id(app_id): + if not isinstance(app_id, str) or not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._-]{0,127}", app_id) or ".." in app_id: + raise ScenarioError(f"manifest id {app_id!r} cannot name a jail directory") + return app_id + + +def prepare_copy(bundle, workdir, scenario, nonce="0"): + """Copy the bundle and prepend the fake host. Returns (copy, shim_lines). + + Only main.splash changes (and the copy's signature goes, as its bytes + differ); capabilities, storage and everything else stay as written. + """ + links = bundle_links(bundle) + if links: + raise ScenarioError(f"{bundle} contains symbolic links ({', '.join(links[:3])}); a bundle holds regular " + "files only, and the runner never writes through a link to the original") + copy = workdir / "bundle" + shutil.copytree(bundle, copy, symlinks=False) + main = copy / "main.splash" + if not main.is_file() or main.is_symlink(): + raise ScenarioError(f"{bundle} has no main.splash; uitest runs script apps") + if (copy / "manifest.json").is_symlink(): + raise ScenarioError(f"{bundle}/manifest.json is a link") + manifest_path = copy / "manifest.json" + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + check_app_id(manifest.get("id")) + shim_lines = 0 + if scenario.get("fake_host", True): + shim = shim_source(scenario.get("host", {}), scenario.get("host_profile", "octosense"), + manifest.get("capabilities", []), nonce) + shim_lines = shim.count("\n") + main.write_text(shim + main.read_text(encoding="utf-8"), encoding="utf-8") + integrity = manifest.setdefault("integrity", {}) + if integrity.pop("signature", None) is not None or integrity.pop("github", None) is not None: + manifest_path.write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + return copy, shim_lines + + +def bundle_links(bundle): + """Every symbolic link under the bundle, as paths relative to it.""" + out = [] + for root, dirs, files in os.walk(bundle, followlinks=False): + for name in dirs + files: + p = Path(root) / name + if p.is_symlink(): + out.append(str(p.relative_to(bundle))) + return out + + +def jail_path(jail, name, where="storage"): + """The file `name` inside `jail`, refusing any name that would leave it. + + Scenario files come from the repository under test; a name such as + "../../.bashrc" or "/etc/x" must never become a write outside the jail. + """ + check_jail_name(name, where) + root = Path(jail).resolve() + target = (root / name).resolve() + if target != root and root not in target.parents: + raise ScenarioError(f"{where}: {name!r} resolves outside the app's jail") + return target + + +def check_jail_name(name, where): + if not isinstance(name, str) or not name.strip(): + raise ScenarioError(f"{where}: a file name is a non-empty string, got {name!r}") + if any(ord(ch) < 32 or ch == "\x7f" for ch in name): + raise ScenarioError(f"{where}: {name!r} contains a control character") + p = Path(name) + if p.is_absolute() or name.startswith(("/", "\\")) or re.match(r"^[A-Za-z]:", name): + raise ScenarioError(f"{where}: {name!r} is absolute; name a file inside the app's jail") + if "\\" in name or ".." in p.parts or any(ch in name for ch in "*?[]"): + raise ScenarioError(f"{where}: {name!r} may not contain '..', a backslash or a glob character") + if not p.parts or name.rstrip().endswith("/"): + raise ScenarioError(f"{where}: {name!r} names a directory; name a file") + # Common filesystem limits (NAME_MAX 255 bytes; well under PATH_MAX with the jail + # in front): past them the name can only fail once the app is already running. + if len(name.encode("utf-8")) > 1024 or any(len(part.encode("utf-8")) > 255 for part in p.parts): + raise ScenarioError(f"{where}: {_short(name, 60)} is longer than a file name may be " + "(255 bytes a part, 1024 in all)") + + +# --------------------------------------------------------------- card-host +def config_file(): + """Where scripts/setup.sh records the card-host it found or built, for later shells.""" + base = os.environ.get("XDG_CONFIG_HOME") or str(Path.home() / ".config") + return Path(base) / "splash-app-verify" / "card-host" + + +def card_host_candidates(explicit): + out = [("--card-host", explicit), ("OCTO_CARD_HOST", os.environ.get("OCTO_CARD_HOST"))] + try: + saved = config_file().read_text(encoding="utf-8").strip() + except OSError: + saved = None + out.append((f"{config_file()} (scripts/setup.sh)", saved)) + for var in ("OCTOSENSE_APP_HUB", "CARGO_TARGET_DIR"): + root = os.environ.get(var) + if root: + for sub in ("target/release", "release"): + out += [(var, str(Path(root) / sub / name)) for name in CARD_HOST_NAMES] + out.append(("PATH", shutil.which("card-host"))) + return out + + +def find_card_host(explicit, with_source=False): + for source, c in card_host_candidates(explicit): + if c and Path(c).is_file() and os.access(c, os.X_OK): + found = str(Path(c).resolve()) + return (found, source) if with_source else found + return (None, None) if with_source else None + + +def free_port(): + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + return s.getsockname()[1] + + +class Display: + """Use $DISPLAY or a native window system; on headless Linux start one private Xvfb.""" + + def __init__(self): + self.proc = None + self.env = {} + + DISPLAYS = range(90, 140) + SOCKET_WAIT = 5.0 # seconds for a new Xvfb to create its socket + + def __enter__(self): + if sys.platform.startswith("linux") and not os.environ.get("DISPLAY") and not os.environ.get("WAYLAND_DISPLAY"): + xvfb = shutil.which("Xvfb") + if not xvfb: + raise RuntimeError("no display and no Xvfb: install xvfb (see scripts/setup.sh)") + n = next((n for n in self.DISPLAYS if not Path(f"/tmp/.X11-unix/X{n}").exists() + and not Path(f"/tmp/.X{n}-lock").exists()), None) + if n is None: + raise RuntimeError(f"no free X display among :{self.DISPLAYS[0]}-:{self.DISPLAYS[-1]} for a private " + "Xvfb; stop stale Xvfb servers or set DISPLAY") + self.proc = subprocess.Popen([xvfb, f":{n}", "-screen", "0", "1600x1200x24", "-nolisten", "tcp"], + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + # The runner must not go on to blame card-host for a display that never came up. + deadline = time.time() + self.SOCKET_WAIT + while not Path(f"/tmp/.X11-unix/X{n}").exists(): + code = self.proc.poll() + if code is not None or time.time() >= deadline: + self.__exit__(None, None, None) + self.proc = None + raise RuntimeError(f"Xvfb :{n} " + (f"exited with status {code}" if code is not None else + f"did not create /tmp/.X11-unix/X{n} in {self.SOCKET_WAIT:g}s") + + "; no scenario was run") + time.sleep(0.1) + self.env = {"DISPLAY": f":{n}", "LIBGL_ALWAYS_SOFTWARE": "1"} + self.env["MAKEPAD_HIDE_WINDOWS"] = "1" + return self + + def __exit__(self, *exc): + if self.proc: + self.proc.terminate() + try: + self.proc.wait(5) + except subprocess.TimeoutExpired: + self.proc.kill() + + +def is_runtime_error(line): + """A log line that means the app's script failed: an error, a panic, a + stop on its instruction budget, or a parse error (`splash: splash:::: …`).""" + if any(m in line for m in RUNTIME_ERROR_MARKERS): + return True + return bool(re.search(r"splash: .* is stopped: ", line) or re.search(r"splash: splash:\d+:\d+:\d+:", line)) + + +class App: + def __init__(self, card_host, bundle, data_dir, size, log_path, env, jail=None, nonce=None): + self.port = free_port() + self.base = f"http://127.0.0.1:{self.port}" + self.log_path = log_path + self.data_dir = data_dir + self.jail = Path(jail) if jail else None + self.nonce = nonce + self.log_file = open(log_path, "w", encoding="utf-8", errors="replace") + full_env = dict(os.environ, **env, MAKEPAD_REMOTE=str(self.port)) + self.proc = subprocess.Popen( + [card_host, "--bundle", str(bundle), "--app-data", str(data_dir), "--allow-unsigned", "--stamp", + "--size", size], + stdout=self.log_file, stderr=subprocess.STDOUT, env=full_env, cwd=str(bundle.parent), + start_new_session=True) + self.log_seen = 0 + self.line_offset = 0 + + def get(self, route, timeout=10.0, **params): + query = urllib.parse.urlencode({k: v for k, v in params.items() if v is not None}) + url = f"{self.base}{route}" + (f"?{query}" if query else "") + with urllib.request.urlopen(url, timeout=timeout) as r: + body = r.read() + if route in ("/g", "/grab") and params.get("raw"): + return body + try: + return json.loads(body.decode("utf-8")) + except ValueError: + return body.decode("utf-8", "replace") + + def log_text(self): + try: + return Path(self.log_path).read_text(encoding="utf-8", errors="replace") + except OSError: + return "" + + def wait_ready(self, timeout): + deadline = time.time() + timeout + last = None + while time.time() < deadline: + if self.proc.poll() is not None: + raise RuntimeError(f"card-host exited ({self.proc.returncode}); see {self.log_path}") + log = self.log_text() + if "card-host: refused" in log: + raise StepFailed("the bundle is admitted", "card-host refused it", + {"log": [l for l in log.splitlines() if "refused" in l][-3:]}) + try: + snap = self.get("/snap", timeout=3) + if isinstance(snap, dict) and snap.get("s"): + time.sleep(0.3) # let start_timeout(0.05, …) loaders run + return + last = snap + except (OSError, ValueError) as e: + last = str(e) + time.sleep(0.25) + raise RuntimeError(f"card-host did not answer /snap with widgets in {timeout}s (last: {last!r}); see {self.log_path}") + + def snap(self): + data = self.get("/snap") + widgets = data.get("s", []) if isinstance(data, dict) else [] + # The `Splash` host widget reports the whole script source as its text; + # it is not something the app shows. + return [w for w in widgets if w.get("ty") != "Splash"] + + def new_runtime_errors(self): + lines = self.log_text().splitlines() + fresh = lines[self.log_seen:] + self.log_seen = len(lines) + errors = [l for l in fresh if is_runtime_error(l)] + # Error positions count the test shim's lines; report them against main.splash. + def shift(m): + return f"splash:{m.group(1)}:{max(1, int(m.group(2)) - self.line_offset)}:{m.group(3)}" + return [re.sub(r"splash:(\d+):(\d+):(\d+)", shift, l) for l in errors] + + def exited(self): + proc = getattr(self, "proc", None) + return proc is not None and proc.poll() is not None + + def find_file(self, name): + """The file at exactly this path in the app's jail, or None. No search: + a file of the same name elsewhere (a backup, say) does not count.""" + if self.jail is None: + raise EvidenceError("the runner does not know the app's jail") + check_jail_name(name, "files") + path = self.jail + try: + for part in Path(name).parts: + path = path / part + if path.is_symlink(): # a link may point anywhere: not the app's file + return None + return path if path.is_file() else None + except OSError as e: + raise EvidenceError(f"cannot look at {name!r} in the app's jail: {e}") from None + + def calls(self, final=False): + """Every host.request the app made, from the fake host's lines in card-host's log. + + The ready line comes first; without it the evidence is missing, which + is an error, never "no calls". A line that does not parse, a gap in the + sequence numbers, or a record cut off at the end of a finished log is an + error too. While the host runs, a record still being written makes the + check wait (PendingEvidence) rather than count what is there so far. + """ + if not self.nonce: + raise EvidenceError("this run has no fake host, so there is no call evidence") + text = self.log_text() + cut = text.rfind("\n") + 1 + complete, tail = text[:cut], text[cut:] + if f"{CALL_MARK}{self.nonce}" in tail: + if final: + raise EvidenceError("the card-host log ends inside a call record; " + "the call evidence is incomplete") + raise PendingEvidence("the call log has no record still being written", + "the last call record is still being written", {"tail": tail[:200]}) + lines = complete.splitlines() + if f"{READY_MARK}{self.nonce}" not in (l.strip() for l in lines): + raise EvidenceError("the fake host never reported itself ready in the card-host log, " + "so no call check can be trusted") + mark = f"{CALL_MARK}{self.nonce} " + out = [] + for l in lines: + if l.startswith(mark): + try: + call = json.loads(l[len(mark):]) + except ValueError as e: + raise EvidenceError(f"a call line in the card-host log does not parse ({e}): {l[:200]}") + if not isinstance(call, dict) or not isinstance(call.get("service"), str): + raise EvidenceError(f"a call line in the card-host log is not a call: {l[:200]}") + if call.get("seq") != len(out) + 1: + raise EvidenceError(f"call record {len(out) + 1} is missing from the card-host log " + f"(the next one says seq {call.get('seq')!r})") + out.append(call) + return out + + def peak_rss_mb(self): + """card-host's peak resident memory (Linux), for the resource record.""" + try: + for line in Path(f"/proc/{self.proc.pid}/status").read_text(encoding="utf-8").splitlines(): + if line.startswith("VmHWM:"): + return round(int(line.split()[1]) / 1024, 1) + except OSError: + pass + return None + + def close(self): + try: + if self.proc.poll() is None: + try: + self.get("/quit", timeout=3) + except Exception: + pass + try: + self.proc.wait(5) + except subprocess.TimeoutExpired: + for force in (False, True): + try: + stop_process(self.proc, force) + except ProcessLookupError: + break # it exited meanwhile + try: + self.proc.wait(5) + break + except subprocess.TimeoutExpired: + continue + if self.proc.poll() is None: + raise RuntimeError(f"card-host (pid {self.proc.pid}) ignored /quit, terminate and kill, " + "and is still running") + finally: + self.log_file.close() + + +def stop_process(proc, force): + """Stop a host that ignored /quit. + + On POSIX the host was started in its own session, so the whole process + group goes (SIGTERM, then SIGKILL with force). Windows has no os.killpg and + ignores start_new_session, so there the process itself is terminated; with + force, killed. + """ + killpg = getattr(os, "killpg", None) + if killpg is not None and os.name != "nt": + killpg(proc.pid, signal.SIGKILL if force else signal.SIGTERM) + elif force: + proc.kill() + else: + proc.terminate() + + +# --------------------------------------------------------------- selectors and checks +def describe(sel): + return json.dumps(sel, ensure_ascii=False) if not isinstance(sel, str) else f'"{sel}"' + + +def matches(w, sel): + if isinstance(sel, str): + sel = {"text": sel} + text = (w.get("t") or "") + ("\n" + str(w["value"]) if w.get("value") not in (None, "") else "") + if "id" in sel and w.get("i") != sel["id"]: + return False + if "type" in sel and w.get("ty") != sel["type"]: + return False + if "text" in sel and (w.get("t") or "").strip() != sel["text"]: + return False + if "contains" in sel and sel["contains"] not in text: + return False + return True + + +def find(widgets, sel): + """All widgets matching sel, and the one it picks. + + `after` keeps only widgets listed after the first match of another + selector: the button in the same card as a text, in a list without ids. + `nth` counts what is on screen now, so it shifts when the view scrolls. + """ + pool = widgets + if isinstance(sel, dict) and "after" in sel: + _, anchor = find(widgets, sel["after"]) + # By identity: two cards with the same heading report equal widgets. + at = next((i for i, w in enumerate(widgets) if w is anchor), None) + pool = widgets[at + 1:] if at is not None else [] + found = [w for w in pool if matches(w, sel)] + nth = sel.get("nth", 0) if isinstance(sel, dict) else 0 + return found, (found[nth] if len(found) > nth else None) + + +def nearest(widgets, sel, limit=5): + """What the screen has that looks like the selector: for the feedback, not for passing.""" + key = sel if isinstance(sel, str) else (sel.get("text") or sel.get("contains") or sel.get("id") or "") + key = str(key) + scored = [] + for w in widgets: + hay = f"{w.get('i','')} {w.get('t','')}" + common = sum(1 for ch in set(key) if ch in hay) + if common: + scored.append((common, w)) + scored.sort(key=lambda x: -x[0]) + return [{"id": w.get("i"), "type": w.get("ty"), "text": w.get("t"), "rect": w.get("r")} for _, w in scored[:limit]] + + +def widget_text(w): + """What one widget shows: its text, and an input's value.""" + text = w.get("t") or "" + if w.get("value") not in (None, ""): + text += "\n" + str(w["value"]) + return text + + +def shows(widgets, t): + """Whether one widget shows t. Text is never matched across two widgets.""" + return any(t in widget_text(w) for w in widgets) + + +def inside(rect, view): + x, y, w, h = rect + vx, vy, vw, vh = view + return x >= vx - 0.5 and y >= vy - 0.5 and x + w <= vx + vw + 0.5 and y + h <= vy + vh + 0.5 + + +def clipped_by(widgets, w): + """The container that may cut w off, if any. + + /snap reports the visible part of a widget inside a scroll view, so a + half-hidden button looks like a short one. Its cut edge then lies exactly + on the scroll view's edge. A whole widget can also sit flush against a + container's edge, so this is only a suspicion: `is_clipped` settles it. + """ + x, y, ww, hh = w["r"] + for c in widgets: + if c is w or c.get("ty") in ("Window", "KeyboardView"): + continue + cx, cy, cw, ch = c["r"] + if ch < 2 * hh + 20 or not inside(w["r"], c["r"]): + continue + if abs(y - cy) < 0.5 or abs((y + hh) - (cy + ch)) < 0.5: + return c + return None + + +def _positions(widgets): + return json.dumps([x.get("r") for x in widgets]) + + +def is_clipped(app, widgets, w, sel): + """Whether w is cut off, settled by looking, as a person would. + + Only when w touches a container's edge (`clipped_by`): scroll that + container a little toward the touching edge and snap again. If the widget + grows, part of it was hidden. If it moves without growing, or nothing + moves (the content ends there), it was whole. The scroll is then undone. + """ + cut = clipped_by(widgets, w) + if cut is None: + return None + x, y, ww, hh = w["r"] + at_bottom = abs((y + hh) - (cut["r"][1] + cut["r"][3])) < 0.5 + dy = 60.0 if at_bottom else -60.0 + px, py = center(cut["r"]) + before = _positions(widgets) + app.get("/m", k="scroll", x=px, y=py, dy=dy, wait=1) + after_widgets = app.snap() + _, w2 = find(after_widgets, sel) + if _positions(after_widgets) == before: + return None # nothing scrolls there: the widget is whole, flush against the edge + grown = bool(w2) and w2["r"][3] > hh + 0.5 + # Undo the probe, then correct once if the content did not come back exactly. + app.get("/m", k="scroll", x=px, y=py, dy=-dy, wait=1) + if w2: + _, w3 = find(app.snap(), sel) + moved = y - w2["r"][1] + if w3 and moved and abs(w3["r"][1] - y) > 0.5: + app.get("/m", k="scroll", x=px, y=py, dy=round((w3["r"][1] - y) * dy / moved, 1), wait=1) + return cut if grown else None + + +def fully_visible(app, widgets, w, sel, view): + return inside(w["r"], view) and is_clipped(app, widgets, w, sel) is None + + +def check_expectation(app, exp, view): + """Raise StepFailed unless every clause of exp holds now (ScenarioError for a malformed exp).""" + validate_expect(exp, "expectation") + widgets = app.snap() + for t in exp.get("text", []): + if not shows(widgets, t): + raise StepFailed(f'text "{t}" is shown', "not on screen", + {"similar": nearest(widgets, {"contains": t})}) + for t in exp.get("absent", []): + if isinstance(t, str): + hits = [w for w in widgets if t in widget_text(w)] + what = f'text "{t}"' + else: + hits, _ = find(widgets, t) + what = f"a widget {describe(t)}" + if hits: + raise StepFailed(f"{what} is not shown", "it is on screen", + {"widgets": [{"id": w.get("i"), "type": w.get("ty"), "text": w.get("t"), "rect": w.get("r")} + for w in hits[:5]]}) + for sel in exp.get("in_view", []): + found, w = find(widgets, sel) + if not w: + raise StepFailed(f"{describe(sel)} is fully inside the viewport {view}", "no such widget", + {"similar": nearest(widgets, sel)}) + if not inside(w["r"], view): + x, y, ww, hh = w["r"] + over = {"right": round(x + ww - (view[0] + view[2]), 1), "bottom": round(y + hh - (view[1] + view[3]), 1)} + raise StepFailed(f"{describe(sel)} is fully inside the viewport {view}", + f"its rect is {w['r']}: past the right edge by {over['right']} and the bottom by {over['bottom']} points", + {"widget": {"id": w.get("i"), "type": w.get("ty"), "text": w.get("t"), "rect": w.get("r")}}) + cut = is_clipped(app, widgets, w, sel) + if cut is not None: + raise StepFailed(f"{describe(sel)} is fully inside the viewport {view}", + f"only {w['r']} of it shows: the scroll view {cut.get('i') or cut.get('ty')} {cut['r']} cuts it off " + "(scrolling it toward that edge shows more of the widget)", + {"widget": {"id": w.get("i"), "type": w.get("ty"), "text": w.get("t"), "rect": w.get("r")}}) + for sel in exp.get("enabled", []): + _, w = find(widgets, sel) + if not w or not w.get("enabled", True): + raise StepFailed(f"{describe(sel)} is enabled", "missing" if not w else "disabled", {}) + for sel in exp.get("disabled", []): + _, w = find(widgets, sel) + if not w or w.get("enabled", True): + raise StepFailed(f"{describe(sel)} is disabled", "missing" if not w else "enabled", {}) + for item in exp.get("count", []): + found, _ = find(widgets, item["of"]) + if len(found) != item["n"]: + raise StepFailed(f"{item['n']} widgets match {describe(item['of'])}", f"{len(found)} match", {}) + for item in exp.get("calls", []): + calls = [c for c in app.calls() if c.get("service") == item["service"]] + if "contains" in item: + calls = [c for c in calls if item["contains"] in json.dumps(c.get("args"), ensure_ascii=False)] + want = item.get("count") + # Without "count", at least one call, with or without "excludes": an + # "excludes" over no calls at all would pass without testing anything. + # To require that no call was made, write "count": 0. + if (want is None and not calls) or (want is not None and len(calls) != want): + raise StepFailed(f"{'≥1' if want is None else want} call(s) to {item['service']}" + + (f" with args containing \"{item['contains']}\"" if "contains" in item else ""), + f"{len(calls)} such call(s)", {"calls": app.calls()[-5:]}) + if "excludes" in item: + leaked = [c for c in calls if item["excludes"] in json.dumps(c.get("args"), ensure_ascii=False)] + if leaked: + raise StepFailed(f"no call to {item['service']} sends \"{item['excludes']}\"", + f"{len(leaked)} call(s) do", {"calls": leaked[-2:]}) + for item in exp.get("files", []): + p = app.find_file(item["name"]) + if item.get("missing"): + if p: + raise StepFailed(f"no file {item['name']} in the jail", f"{p} exists", {}) + continue + if not p: + raise StepFailed(f"file {item['name']} in the jail", "missing", {}) + try: + body = p.read_text(encoding="utf-8", errors="replace") + except OSError as e: + raise EvidenceError(f"cannot read {item['name']} in the app's jail: {e}") from None + for c in item.get("contains", []): + if c not in body: + raise StepFailed(f"{item['name']} contains \"{c}\"", "it does not", {"head": body[:400]}) + + +def wait_for(app, exp, view, timeout): + deadline = time.time() + timeout + while True: + try: + check_expectation(app, exp, view) + return + except StepFailed: + if time.time() >= deadline: + raise + time.sleep(0.2) + + +def center(rect): + x, y, w, h = rect + return round(x + w / 2, 1), round(y + h / 2, 1) + + +def scroll_into_view(app, sel, on, view, attempts): + """Scroll, as a person would, until the widget lies wholly inside the viewport. + + Works at every size without per-size distances. Fails when the widget is + taller than the viewport or scrolling stops moving anything. + """ + vx, vy, vw, vh = view + ratio = 1.0 # measured content movement per unit of scroll + before = None # (target y, dy) of the previous attempt + last_sig = None + direction = 1 # while the widget is not listed: search down, then up + for _ in range(attempts): + widgets = app.snap() + _, w = find(widgets, sel) + cut = is_clipped(app, widgets, w, sel) if w else None + if w and cut is None and inside(w["r"], view): + return + if w and w["r"][3] > vh + 0.5: + raise StepFailed(f"{describe(sel)} fits inside the viewport {view}", + f"it is {w['r'][3]} points tall", {"rect": w["r"]}) + sig = json.dumps([x.get("r") for x in widgets]) + if sig == last_sig: + if not w and direction == 1: + direction = -1 # reached the end without seeing it: look above + else: + raise StepFailed(f"{describe(sel)} can be scrolled into the viewport {view}", + "scrolling no longer moves anything" + ("" if w else "; the widget never appeared"), + {"similar": nearest(widgets, sel)} if not w else {"rect": w["r"]}) + last_sig = sig + if w: + y, h = w["r"][1], w["r"][3] + if before is not None and before[1]: + moved = before[0] - y + if moved and (moved / before[1]) > 0: + ratio = moved / before[1] + if cut is not None: + # Only its visible part is listed: move a third of a screen toward the cut edge. + at_bottom = abs((y + h) - (cut["r"][1] + cut["r"][3])) < 0.5 + dy = (1 if at_bottom else -1) * vh * 0.33 / ratio + before = None + else: + need = (y + h) - (vy + vh) + 12 if y + h > vy + vh else y - vy - 12 + dy = need / ratio + before = (y, dy) + else: + dy = direction * vh * 0.6 + before = None + if on: + _, host_w = find(widgets, on) + if not host_w: + raise StepFailed(f"a widget {describe(on)} to scroll", "no such widget", {}) + px, py = center(host_w["r"]) + else: + px, py = vx + vw / 2, vy + vh / 2 + app.get("/m", k="scroll", x=round(px, 1), y=round(py, 1), dy=round(dy, 1), wait=1) + raise StepFailed(f"{describe(sel)} can be scrolled into the viewport {view}", + f"still outside after {attempts} scrolls", {}) + + +def do_step(app, step, view): + validate_step(step, "step") + if "click" in step: + sel = step["click"] + widgets = app.snap() + _, w = find(widgets, sel) + if not w: + raise StepFailed(f"a widget {describe(sel)} to click", "no such widget on screen", + {"similar": nearest(widgets, sel)}) + if not inside(w["r"], view): + raise StepFailed(f"{describe(sel)} is inside the viewport so a person can click it", + f"its rect is {w['r']}, viewport {view}", {}) + x, y = center(w["r"]) + app.get("/click", x=x, y=y, wait=1) + elif "type" in step: + spec = step["type"] + if "into" in spec: + do_step(app, {"click": spec["into"]}, view) + if spec.get("clear"): + # The click leaves the caret wherever it landed: delete on both sides of it. + _, w = find(app.snap(), spec["into"]) + n = len(str((w or {}).get("value") or (w or {}).get("t") or "")) + 2 + for key in ("Delete", "Backspace"): + for _ in range(n): + app.get("/k", k="down", c=key) + app.get("/t", t=spec["text"], wait=1) + elif "key" in step: + app.get("/k", k="down", c=step["key"], wait=1) + app.get("/k", k="up", c=step["key"], wait=1) + elif "scroll" in step: + spec = step["scroll"] + _, w = find(app.snap(), spec["on"]) + if not w: + raise StepFailed(f"a widget {describe(spec['on'])} to scroll", "no such widget", {}) + x, y = center(w["r"]) + app.get("/m", k="scroll", x=x, y=y, dy=spec.get("dy", 300), wait=1) + elif "scroll_to" in step: + targets = step["scroll_to"] if isinstance(step["scroll_to"], list) else [step["scroll_to"]] + for sel in targets: + scroll_into_view(app, sel, step.get("on"), view, int(step.get("max", 25))) + elif "sleep" in step: + time.sleep(float(step["sleep"])) + elif "expect" in step: + wait_for(app, step["expect"], view, float(step.get("timeout", 2.0))) + elif "wait" in step: + wait_for(app, step["wait"], view, float(step.get("timeout", 10.0))) + else: + raise ScenarioError(f"unknown step {step!r}") + errors = app.new_runtime_errors() + if errors and not step.get("allow_runtime_errors"): + raise StepFailed("no runtime error in the log", errors[-1], {"log": errors[-5:]}) + if hasattr(app, "exited") and app.exited(): + raise StepFailed("the app keeps running", f"card-host exited with status {app.proc.returncode}", {}) + + +# --------------------------------------------------------------- run and report +def sha256(path): + return hashlib.sha256(Path(path).read_bytes()).hexdigest() + + +def environment_report(result, bundle, card_host): + """What an environment error leaves instead of FEEDBACK.md: not a repair request. + + A dead display, a card-host that exited or would not stop, or call evidence + the runner cannot read says nothing about the app, and a model that reads + a repair request would go and change the app's source for it. + """ + s = result + steps = "\n".join(f" {i + 1}. {json.dumps(st, ensure_ascii=False)}" for i, st in enumerate(s["steps_run"])) or " (none)" + problems = [f"{k}: {v}" for k, v in s.get("evidence", {}).items() if k.endswith("_error")] + return f"""# Environment error, not a repair request: {s['scenario']} @ {s['size']} + +The runner could not run this scenario to the end, or could not read its own +evidence. This says nothing about the app: do not change {bundle}/main.splash +for it. Exit status 2. + +While: {s.get('expected')} +What happened: {s.get('actual')} +{chr(10).join(problems) if problems else ''} +Runtime: card-host {card_host}, window {s['size']}. +Steps reached: +{steps} +Evidence: {s['evidence'].get('log', '(no log)')} + +Fix the machine, then run every scenario again: `uitest.py doctor`, the display +(Xvfb), the card-host binary, the disk, and the card-host log above. +""" + + +def feedback(result, bundle, card_host): + s = result + steps = "\n".join(f" {i + 1}. {json.dumps(st, ensure_ascii=False)}" for i, st in enumerate(s["steps_run"])) + detail = json.dumps(s["detail"], ensure_ascii=False, indent=2) if s["detail"] else "(none)" + return f"""# Repair request: {s['scenario']} @ {s['size']} + +Artifact: {bundle}/main.splash, sha256 {s['source_sha256']}. +Runtime: card-host {card_host}, window {s['size']}, viewport {s['viewport']}; host services are the scenario's fixtures. +Start state: a fresh launch of scenario "{s['scenario']}" ({s['file']}); storage seeded as the scenario says. + +Steps (the last one failed): +{steps} + +Expected: {s['expected']} +Actual: {s['actual']} +Details: +```json +{detail} +``` +Evidence: {s['evidence'].get('snap')}, {s['evidence'].get('log')}{', ' + s['evidence']['png'] if s['evidence'].get('png') else ''} + +Fix: change only what makes this step pass. Keep widget ids, button texts and the behavior other scenarios check. +Retest: run every scenario again (`uitest.py run `); report what still fails. Do not score yourself. +""" + + +def app_files(jail): + """The app's own files in its jail: regular files only (no links), not the host's.""" + jail = Path(jail) + if not jail.is_dir() or jail.is_symlink(): + return [] + out = [] + for p in jail.rglob("*"): + rel = p.relative_to(jail) + if ".host" in rel.parts or p.is_symlink() or not p.is_file(): + continue + if any((jail / Path(*rel.parts[:i])).is_symlink() for i in range(1, len(rel.parts))): + continue + out.append(p) + return out + + +def copy_jail(jail, dest, limit=4 * 1024 * 1024): + """Copy the app's own files if they are small enough.""" + files = app_files(jail) + if not files or sum(p.stat().st_size for p in files) > limit: + return False + for p in files: + target = dest / p.relative_to(jail) + target.parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(p, target) + return True + + +def run_one(card_host, bundle, scenario_path, scenario, size, out_dir, display, run_id=None): + view_cfg = scenario.get("viewport") + w, h = (float(v) for v in size.lower().split("x")) + view = view_cfg if view_cfg else [0, 0, w, h] + tag = f"{scenario_path.stem}@{size}" # its length is checked when the scenarios load + run_dir = out_dir / tag + if run_dir.exists(): + shutil.rmtree(run_dir) + run_dir.mkdir(parents=True) + nonce = os.urandom(8).hex() if scenario.get("fake_host", True) else None + result = {"scenario": scenario.get("name", scenario_path.stem), "file": str(scenario_path), "size": size, + "viewport": view, "run_id": run_id, "scenario_sha256": scenario_digest(scenario), + "host_profile": scenario.get("host_profile", "octosense") if scenario.get("fake_host", True) else None, + "source_sha256": sha256(bundle / "main.splash"), "passed": False, "steps_run": [], "evidence": {}} + app = None + # Created last, right before the try whose finally removes it: nothing above can leak it. + work = Path(tempfile.mkdtemp(prefix="uitest-")) + jail = work / "data" + started = time.time() + try: + copy, shim_lines = prepare_copy(bundle, work, scenario, nonce or "0") + data = work / "data" + manifest = json.loads((copy / "manifest.json").read_text(encoding="utf-8")) + jail = data / check_app_id(manifest.get("id")) + jail.mkdir(parents=True, exist_ok=True) + for name, body in (scenario.get("storage") or {}).items(): + target = jail_path(jail, name) + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(body if isinstance(body, str) else json.dumps(body, ensure_ascii=False), encoding="utf-8") + app = App(card_host, copy, data, size, run_dir / "card-host.log", display.env, jail=jail, nonce=nonce) + app.line_offset = shim_lines + app.wait_ready(float(scenario.get("startup_timeout", 60))) + errors = app.new_runtime_errors() + if errors and not scenario.get("allow_runtime_errors"): + raise StepFailed("the app starts without a runtime error", errors[-1], {"log": errors[-5:]}) + for step in scenario.get("steps", []): + result["steps_run"].append(step) + do_step(app, step, view) + result["passed"] = True + except StepFailed as f: + result.update(expected=f.expected, actual=f.actual, detail=f.detail) + except ScenarioError: + raise + except Exception as e: # environment or evidence trouble: report, do not blame the app + result.update(expected="the runner reaches the step and reads its evidence", + actual=f"{type(e).__name__}: {e}", detail={}, environment_error=True) + finally: + result["seconds"] = round(time.time() - started, 2) + # Each piece of evidence is collected on its own: one that fails is + # recorded, and the host is still closed and the work directory removed. + if app: + try: + if not result["passed"]: + try: + if not result.get("environment_error") and not [w for w in app.snap() if w.get("ty") not in HOST_WIDGETS]: + result.setdefault("detail", {})["hint"] = ( + "the app shows no widget at all: its script stopped before drawing anything. " + "A parse error does this, and some runtimes write no log line for it; " + "check the source near your last edit") + except Exception: + pass + try: + (run_dir / "snap.json").write_text(json.dumps(app.get("/snap"), ensure_ascii=False, indent=1), + encoding="utf-8") + result["evidence"]["snap"] = str(run_dir / "snap.json") + png = app.get("/g", raw=1, timeout=20) + if isinstance(png, (bytes, bytearray)) and png[:4] == b"\x89PNG": + (run_dir / "screen.png").write_bytes(png) + result["evidence"]["png"] = str(run_dir / "screen.png") + except Exception as e: + result["evidence"]["snap_error"] = f"{type(e).__name__}: {e}" + try: + result["peak_rss_mb"] = app.peak_rss_mb() + except Exception: + pass + finally: + try: + app.close() + except Exception as e: # the host must not take the cleanup and the result with it + result["evidence"]["close_error"] = f"{type(e).__name__}: {e}" + result["environment_error"] = True + result["evidence"]["log"] = str(run_dir / "card-host.log") + if nonce: + try: + calls = app.calls(final=True) + (run_dir / "calls.json").write_text(json.dumps(calls, ensure_ascii=False, indent=1), encoding="utf-8") + result["evidence"]["calls"] = str(run_dir / "calls.json") + undeclared = sorted({c["service"] for c in calls if c.get("declared") is False}) + if undeclared: + result["undeclared_services"] = undeclared + except Exception as e: + result["evidence"]["calls_error"] = f"{type(e).__name__}: {e}" + if result["passed"]: + # Without readable call evidence the run did not show what the scenario + # describes (the fake host's replies): not a pass, not the app's fault. + result["passed"] = False + result.update(expected="the fake host's call evidence is complete", + actual=result["evidence"]["calls_error"], detail={}, environment_error=True) + # What the app left in its jail: evidence, and a seed for a restart scenario. + try: + if copy_jail(jail, run_dir / "jail"): + result["evidence"]["jail"] = str(run_dir / "jail") + except Exception as e: + result["evidence"]["jail_error"] = f"{type(e).__name__}: {e}" + shutil.rmtree(work, ignore_errors=True) + if not result["passed"]: + # The app's failure gets a repair request; the environment's never does. + name, write = (("ENVIRONMENT.md", environment_report) if result.get("environment_error") + else ("FEEDBACK.md", feedback)) + key = "environment_report" if result.get("environment_error") else "feedback" + try: + (run_dir / name).write_text(write(result, bundle, card_host), encoding="utf-8") + result[key] = str(run_dir / name) + except Exception as e: + result[key] = f"(not written: {type(e).__name__}: {e})" + (run_dir / "result.json").write_text(json.dumps(result, ensure_ascii=False, indent=2), encoding="utf-8") + return result + + +def scenario_digest(scenario): + """The scenario as it ran: after parts were merged and steps expanded.""" + return hashlib.sha256(json.dumps(scenario, ensure_ascii=False, sort_keys=True).encode("utf-8")).hexdigest() + + +def tree_digest(root): + """One digest over every file under root, with its relative path.""" + h = hashlib.sha256() + root = Path(root) + for p in sorted(x for x in root.rglob("*") if x.is_file()): + h.update(str(p.relative_to(root)).encode("utf-8") + b"\0") + h.update(p.read_bytes() + b"\0") + return h.hexdigest() + + +def warnings_for(result): + """Lines the summary prints under a run: true of the run, not failures of the app.""" + out = [] + if result.get("undeclared_services"): + names = ", ".join(result["undeclared_services"]) + if result.get("host_profile") == "rinx": + out.append(f"called without declaring: {names} (refused, as Rinx does)") + else: + out.append(f"called without declaring: {names} (answered, as an OctoSense host does, which lists " + "it as undisclosed; Rinx would refuse it: set \"host_profile\": \"rinx\" to test that)") + if result.get("evidence", {}).get("calls_error"): + out.append("call log: " + result["evidence"]["calls_error"]) + return out + + +SHARED_KEYS = ("sizes", "size", "viewport", "startup_timeout", "fake_host", "allow_runtime_errors", "host_profile") + + +def _unknown(keys, allowed, where, what): + bad = sorted(set(keys) - set(allowed)) + if bad: + raise ScenarioError(f"{where}: unknown {what} key(s) {bad}; allowed: {sorted(allowed)}") + + +def _num(v): + """A finite number that is not a bool (JSON true would otherwise count as 1).""" + return isinstance(v, (int, float)) and not isinstance(v, bool) and math.isfinite(v) + + +def _whole(v): + return isinstance(v, int) and not isinstance(v, bool) and v >= 0 + + +def _seconds(v, positive=False): + """A wait the runner can make: a finite number from 0 (above 0 if positive) to MAX_SECONDS.""" + return _num(v) and (v > 0 if positive else v >= 0) and v <= MAX_SECONDS + + +def _short(v, limit=160): + text = repr(v) + return text if len(text) <= limit else text[:limit] + "…" + + +def _need(cond, where, message): + if not cond: + raise ScenarioError(f"{where}: {message}") + + +def _list(value, where, key, what): + if not isinstance(value, list): + raise ScenarioError(f"{where}: \"{key}\" is a list of {what}, got {_short(value)}") + return value + + +def _list_of(value, kind, where, key): + if not isinstance(value, list) or not all(isinstance(v, kind) for v in value): + raise ScenarioError(f"{where}: \"{key}\" is a list of {'strings' if kind is str else 'selectors'}") + if kind is str and "" in value: + raise ScenarioError(f"{where}: \"{key}\" holds an empty text, which every widget and file contains: " + "the check would pass without testing anything") + + +def validate_selector(sel, where, depth=0): + if isinstance(sel, str): + if not sel: + raise ScenarioError(f"{where}: an empty selector") + return + if not isinstance(sel, dict) or not sel: + raise ScenarioError(f"{where}: a selector is a text or an object, got {_short(sel)}") + _unknown(sel, SELECTOR_KEYS, where, "selector") + if not set(sel) - {"nth", "after"}: + raise ScenarioError(f"{where}: a selector needs one of id, type, text or contains: {_short(sel)}") + for key in ("id", "type", "text", "contains"): + _need(key not in sel or (isinstance(sel[key], str) and sel[key] != ""), where, + f"selector \"{key}\" is a non-empty string") + _need("nth" not in sel or _whole(sel["nth"]), where, "\"nth\" is a whole number from 0") + if "after" in sel: + _need(depth < MAX_AFTER, where, f"\"after\" chains more than {MAX_AFTER} selectors") + validate_selector(sel["after"], where + " after", depth + 1) + + +def validate_expect(exp, where, fake_host=True): + if not isinstance(exp, dict) or not exp: + raise ScenarioError(f"{where}: an expectation is a non-empty object") + _unknown(exp, EXPECT_KEYS, where, "expectation") + if "text" in exp: + _list_of(exp["text"], str, where, "text") + if "absent" in exp: + if not isinstance(exp["absent"], list): + raise ScenarioError(f"{where}: \"absent\" is a list of texts or selectors") + for t in exp["absent"]: + if not isinstance(t, str): + validate_selector(t, where + " absent") + elif not t: + raise ScenarioError(f"{where}: \"absent\" holds an empty text, which every widget contains") + for key in ("in_view", "enabled", "disabled"): + if key in exp: + if not isinstance(exp[key], list): + raise ScenarioError(f"{where}: \"{key}\" is a list of selectors") + for sel in exp[key]: + validate_selector(sel, f"{where} {key}") + for item in _list(exp.get("count", []), where, "count", "{\"of\", \"n\"} objects"): + if not isinstance(item, dict): + raise ScenarioError(f"{where}: a count item is an object") + _unknown(item, COUNT_KEYS, where, "count") + if "of" not in item or not _whole(item.get("n")): + raise ScenarioError(f"{where}: a count item has \"of\" and a whole number \"n\"") + validate_selector(item["of"], where + " count") + if exp.get("calls") and not fake_host: + raise ScenarioError(f"{where}: \"calls\" needs the fake host, and this scenario sets \"fake_host\": false") + for item in _list(exp.get("calls", []), where, "calls", "call checks"): + if not isinstance(item, dict): + raise ScenarioError(f"{where}: a calls item is an object") + _unknown(item, CALL_KEYS, where, "calls") + if not isinstance(item.get("service"), str): + raise ScenarioError(f"{where}: a calls item names its \"service\"") + if "count" in item and not _whole(item["count"]): + raise ScenarioError(f"{where}: calls \"count\" is a whole number") + for key in ("contains", "excludes"): + if key in item and (not isinstance(item[key], str) or not item[key]): + raise ScenarioError(f"{where}: calls \"{key}\" is a non-empty string") + for item in _list(exp.get("files", []), where, "files", "file checks"): + if not isinstance(item, dict): + raise ScenarioError(f"{where}: a files item is an object") + _unknown(item, FILE_KEYS, where, "files") + check_jail_name(item.get("name"), where + " files") + _need("missing" not in item or isinstance(item["missing"], bool), where, "files \"missing\" is true or false") + if "contains" in item: + _list_of(item["contains"], str, where, "contains") + _need(not (item.get("missing") is True and item.get("contains")), where, + "a files item cannot require a file to be missing and to contain text") + + +def validate_step(step, where, fake_host=True): + actions = set(step) & STEP_ACTIONS + if len(actions) != 1: + raise ScenarioError(f"{where}: a step has exactly one of {sorted(STEP_ACTIONS)} or \"run\", got {step!r}") + action = actions.pop() + _unknown(set(step) - {action}, STEP_MODIFIERS[action], where, f"\"{action}\" step") + _need("allow_runtime_errors" not in step or isinstance(step["allow_runtime_errors"], bool), where, + "\"allow_runtime_errors\" is true or false") + _need("note" not in step or isinstance(step["note"], str), where, "\"note\" is a string") + value = step[action] + if action == "click": + validate_selector(value, where) + elif action == "type": + if not isinstance(value, dict) or not isinstance(value.get("text"), str): + raise ScenarioError(f"{where}: \"type\" is an object with \"text\"") + _unknown(value, TYPE_KEYS, where, "type") + _need("clear" not in value or isinstance(value["clear"], bool), where, "\"clear\" is true or false") + if "into" in value: + validate_selector(value["into"], where + " into") + elif action == "key": + if not isinstance(value, str) or not value: + raise ScenarioError(f"{where}: \"key\" names a key") + elif action == "scroll": + if not isinstance(value, dict) or "on" not in value: + raise ScenarioError(f"{where}: \"scroll\" is an object with \"on\"") + _unknown(value, SCROLL_KEYS, where, "scroll") + _need("dy" not in value or _num(value["dy"]), where, "\"dy\" is a number") + validate_selector(value["on"], where + " scroll") + elif action == "scroll_to": + for sel in (value if isinstance(value, list) else [value]): + validate_selector(sel, where + " scroll_to") + if "on" in step: + validate_selector(step["on"], where + " on") + _need("max" not in step or (_whole(step["max"]) and 0 < step["max"] <= MAX_SCROLLS), where, + f"\"max\" is a whole number of scrolls from 1 to {MAX_SCROLLS}") + elif action == "sleep": + _need(_seconds(value), where, f"\"sleep\" is a number of seconds from 0 to {MAX_SECONDS}") + else: + validate_expect(value, where, fake_host) + _need("timeout" not in step or _seconds(step["timeout"]), where, + f"\"timeout\" is a number of seconds from 0 to {MAX_SECONDS}") + + +SIZE_RE = re.compile(r"[1-9][0-9]{1,4}x[1-9][0-9]{1,4}") + + +def validate_scenario(sc, where): + """Types of the scenario's own fields, after its parts are merged.""" + for key in ("fake_host", "allow_runtime_errors"): + _need(key not in sc or isinstance(sc[key], bool), where, f"\"{key}\" is true or false") + for key in ("name", "description", "note"): + _need(key not in sc or isinstance(sc[key], str), where, f"\"{key}\" is a string") + _need(sc.get("host_profile", "octosense") in HOST_PROFILES, where, + f"host_profile must be one of {list(HOST_PROFILES)}") + _need("startup_timeout" not in sc or _seconds(sc["startup_timeout"], positive=True), where, + f"\"startup_timeout\" is a number of seconds above 0, at most {MAX_SECONDS}") + sizes = sc.get("sizes") if "sizes" in sc else ([sc["size"]] if "size" in sc else [DEFAULT_SIZE]) + _need(isinstance(sizes, list) and all(isinstance(z, str) and SIZE_RE.fullmatch(z) for z in sizes), where, + "\"sizes\" is a list of \"WxH\" strings such as \"412x892\"") + # An empty list would quietly run the default size instead of the sizes the author meant. + _need(sizes, where, f"\"sizes\" lists at least one size; leave it out to run {DEFAULT_SIZE}") + if "viewport" in sc: + v = sc["viewport"] + _need(isinstance(v, list) and len(v) == 4 and all(_num(x) for x in v) and v[2] > 0 and v[3] > 0, where, + "\"viewport\" is [x, y, w, h] in points") + _need(isinstance(sc.get("host"), dict), where, "\"host\" is an object of replies by service") + _need(isinstance(sc.get("storage"), dict), where, "\"storage\" is an object of files by name") + + +def _not_json_number(name): + raise ValueError(f"{name} is not a JSON number") + + +def _check_text(s, where): + try: + s.encode("utf-8") + except UnicodeEncodeError: + raise ScenarioError(f"{where}: a text holds a lone surrogate escape such as \\ud800, which is not " + f"UTF-8: {_short(s, 60)}") from None + + +def check_document(doc, where): + """Refuse what Python's JSON reader lets through but nothing after it can use: + text that is not UTF-8 (a lone surrogate escape) and nesting deeper than + MAX_DEPTH. Walked without recursion, so a deep file cannot crash the check.""" + stack = [(doc, 0)] + while stack: + o, depth = stack.pop() + if depth > MAX_DEPTH: + raise ScenarioError(f"{where}: nested more than {MAX_DEPTH} deep") + if isinstance(o, dict): + for k, v in o.items(): + _check_text(k, where) + stack.append((v, depth + 1)) + elif isinstance(o, list): + stack.extend((v, depth + 1) for v in o) + elif isinstance(o, str): + _check_text(o, where) + + +def read_json(f): + """A scenario or part file as standard JSON: no NaN or Infinity, UTF-8 text, bounded nesting.""" + try: + text = Path(f).read_text(encoding="utf-8") + except OSError as e: + raise ScenarioError(f"{f}: cannot read ({e})") + except ValueError as e: + raise ScenarioError(f"{f}: not UTF-8 ({e})") + try: + data = json.loads(text, parse_constant=_not_json_number) + except RecursionError: + raise ScenarioError(f"{f}: nested too deep to read") from None + except ValueError as e: + raise ScenarioError(f"{f}: not JSON ({e})") + check_document(data, f) + return data + + +def _part_file(base, name, where, key): + """A part named by "use" or "run": a file inside the directory of the file that names it. + + The same rules as a storage name (no absolute path, "..", backslash, glob + character or control character), and the resolved file, links followed, + must stay inside that directory. + """ + if not isinstance(name, str) or not name.strip(): + raise ScenarioError(f"{where}: \"{key}\" names a part file, got {_short(name)}") + check_jail_name(name, f"{where}: \"{key}\"") + root = Path(base).resolve() + target = (root / name).resolve() + if root not in target.parents: + raise ScenarioError(f"{where}: \"{key}\" {name!r} resolves outside {root}; a part lives beside " + "the scenarios that use it") + return target + + +def _object(value, where, key, what): + if not isinstance(value, dict): + raise ScenarioError(f"{where}: \"{key}\" is an object of {what}, got {_short(value)}") + return value + + +def expand_steps(steps, base, f, depth=0): + """Inline {"run": "_part.json"} steps: shared step sequences such as "load the room".""" + if depth > 5: + raise ScenarioError(f"{f}: \"run\" nests more than 5 deep") + out = [] + for step in steps: + if not isinstance(step, dict): + raise ScenarioError(f"{f}: a step is an object, got {_short(step)}") + if "run" in step: + _unknown(step, {"run", "note"}, f, "\"run\" step") + part = _part_file(base, step["run"], f, "run") + data = read_json(part) + if not isinstance(data, dict) or not isinstance(data.get("steps"), list): + raise ScenarioError(f"{part}: a part used by \"run\" is an object with a \"steps\" list") + _unknown(data, PART_KEYS, part, "part") + out += expand_steps(data["steps"], part.parent, part, depth + 1) + continue + out.append(step) + return out + + +def resolve_scenario(f, data): + """Apply "use" (shared host replies, storage and sizes from _parts) and expand "run" steps.""" + base = Path(f).parent + uses = data.get("use", []) + uses = [uses] if isinstance(uses, str) else uses + if not isinstance(uses, list): + raise ScenarioError(f"{f}: \"use\" names a part file or a list of them, got {_short(uses)}") + host, storage, shared = {}, {}, {} + for name in uses: + path = _part_file(base, name, f, "use") + part = read_json(path) + if not isinstance(part, dict): + raise ScenarioError(f"{path}: a part is an object") + _unknown(part, PART_KEYS, path, "part") + host.update(_object(part.get("host", {}), path, "host", "replies by service")) + storage.update(_object(part.get("storage", {}), path, "storage", "files by name")) + shared.update({k: part[k] for k in SHARED_KEYS if k in part}) + out = dict(shared) + out.update({k: v for k, v in data.items() if k != "use"}) + out["host"] = {**host, **_object(data.get("host", {}), f, "host", "replies by service")} + out["storage"] = {**storage, **_object(data.get("storage", {}), f, "storage", "files by name")} + out["steps"] = expand_steps(data["steps"], base, f) + return out + + +def load_scenarios(target): + """Every *.json in name order; files whose names start with "_" are parts, not scenarios.""" + target = Path(target) + files = sorted(p for p in target.glob("*.json") if not p.name.startswith("_")) if target.is_dir() else [target] + out = [] + for f in files: + data = read_json(f) + if not isinstance(data, dict) or not isinstance(data.get("steps"), list): + raise ScenarioError(f"{f}: a scenario is an object with a \"steps\" list") + _unknown(data, SCENARIO_KEYS, f, "scenario") + scenario = resolve_scenario(f, data) + try: + normalise_replies(scenario["host"]) # report a bad fixture before anything starts + except ScenarioError as e: + raise ScenarioError(f"{f}: {e}") from None + validate_scenario(scenario, str(f)) + # Each run writes its evidence to /@/: a name the filesystem + # cannot hold is the scenario file's problem, reported now. + for size in scenario.get("sizes") or [scenario.get("size") or DEFAULT_SIZE]: + if len(f"{Path(f).stem}@{size}".encode("utf-8")) > 255: + raise ScenarioError(f"{f}: the file name is too long for its evidence directory " + f"\"{_short(Path(f).stem, 40)}@{size}\" (255 bytes at most); shorten it") + for name in scenario["storage"]: + check_jail_name(name, f"{f}: storage") + for i, step in enumerate(scenario["steps"]): + validate_step(step, f"{f}: step {i + 1}", scenario.get("fake_host", True)) + out.append((f, scenario)) + if not out: + raise ScenarioError(f"no scenario files in {target}") + return out + + +def write_summary(out_dir, summary): + tmp = out_dir / "summary.json.tmp" + tmp.write_text(json.dumps(summary, ensure_ascii=False, indent=2), encoding="utf-8") + tmp.replace(out_dir / "summary.json") + + +def cmd_run(args): + """Every invocation writes a new summary.json first, marked "running", so a + summary from an earlier run can never stand for this one.""" + bundle = Path(args.bundle).resolve() + out_dir = Path(args.out or (bundle.parent / "build" / "uitest")).resolve() + out_dir.mkdir(parents=True, exist_ok=True) + run_id = time.strftime("%Y%m%dT%H%M%S") + "-" + os.urandom(3).hex() + summary = {"run_id": run_id, "status": "running", "runner": RUNNER_VERSION, "bundle": str(bundle), + "scenarios": str(Path(args.scenarios).resolve()), "started": time.strftime("%Y-%m-%dT%H:%M:%S%z"), + "passed": 0, "failed": 0, "results": []} + write_summary(out_dir, summary) + + def abort(message): + summary.update(status="aborted", error=message, finished=time.strftime("%Y-%m-%dT%H:%M:%S%z")) + write_summary(out_dir, summary) + print(f"uitest: {message}", file=sys.stderr) + return 2 + + card_host = find_card_host(args.card_host) + if not card_host: + return abort("card-host not found; pass --card-host or set OCTO_CARD_HOST (scripts/setup.sh records it)") + if not (bundle / "main.splash").is_file(): + return abort(f"{bundle} has no main.splash; uitest runs script apps") + try: + summary.update(card_host=card_host, card_host_sha256=sha256(Path(card_host)), + bundle_sha256=tree_digest(bundle)) + except OSError as e: + return abort(f"cannot read {e.filename or 'an input'} to record it: {e.strerror or e}") + try: + scenarios = load_scenarios(args.scenarios) + except ScenarioError as e: + return abort(str(e)) + results = summary["results"] + try: + with Display() as display: + for path, sc in scenarios: + sizes = sc.get("sizes") or [sc.get("size") or DEFAULT_SIZE] + for size in sizes: + try: + r = run_one(card_host, bundle, path, sc, size, out_dir, display, run_id) + except ScenarioError as e: + return abort(f"{path}: {e}") + results.append(r) + summary["passed"] = sum(x["passed"] for x in results) + summary["failed"] = sum(not x["passed"] for x in results) + write_summary(out_dir, summary) + mark = "PASS" if r["passed"] else ("ENV " if r.get("environment_error") else "FAIL") + mem = f", {r['peak_rss_mb']} MB" if r.get("peak_rss_mb") else "" + print(f"{mark} {r['scenario']} @ {r['size']} ({r['seconds']}s{mem})") + for line in warnings_for(r): + print(f" note: {line}") + if not r["passed"]: + report = (f" environment, not the app: {r.get('environment_report')}" + if r.get("environment_error") else f" feedback: {r.get('feedback')}") + print(f" expected: {r['expected']}\n actual: {r['actual']}\n{report}") + if not args.keep_going: + break + if results and not results[-1]["passed"] and not args.keep_going: + break + except Exception as e: # the display, the host or the disk: the environment, never a pass + summary.update(status="environment_error", error=f"{type(e).__name__}: {e}", + finished=time.strftime("%Y-%m-%dT%H:%M:%S%z")) + write_summary(out_dir, summary) + print(f"uitest: environment error: {type(e).__name__}: {e}", file=sys.stderr) + return 2 + except BaseException: + summary.update(status="aborted", error="interrupted", finished=time.strftime("%Y-%m-%dT%H:%M:%S%z")) + write_summary(out_dir, summary) + raise + env_error = any(r.get("environment_error") for r in results) + summary["status"] = "environment_error" if env_error else ("passed" if summary["failed"] == 0 else "failed") + summary["finished"] = time.strftime("%Y-%m-%dT%H:%M:%S%z") + write_summary(out_dir, summary) + print(f"{summary['passed']} passed, {summary['failed']} failed; run {run_id}; details in {out_dir}") + if env_error: + return 2 + return 0 if summary["failed"] == 0 else 1 + + +def cmd_doctor(args): + ok = True + card_host, source = find_card_host(args.card_host, with_source=True) + if card_host: + print(f"[ok] card-host: {card_host} (from {source})") + else: + print("[fail] card-host: not found; looked at " + ", ".join(s for s, _ in card_host_candidates(args.card_host)) + + ". Run scripts/setup.sh, or pass --card-host") + ok &= bool(card_host) + if sys.platform.startswith("linux") and not os.environ.get("DISPLAY") and not os.environ.get("WAYLAND_DISPLAY"): + x = shutil.which("Xvfb") + print(f"[{'ok' if x else 'fail'}] display: none; Xvfb {'found at ' + x if x else 'missing (apt install xvfb)'}") + ok &= bool(x) + else: + print("[ok] display: available") + print(f"[ok] python {sys.version.split()[0]}") + return 0 if ok else 2 + + +def main(argv=None): + # Scenario names are often not in the console's code page (cp1252 on Windows): + # show such characters escaped rather than stop the run. Files are always UTF-8. + for stream in (sys.stdout, sys.stderr): + try: + stream.reconfigure(errors="backslashreplace") + except (AttributeError, ValueError): + pass # a stream someone replaced keeps its own settings + p = argparse.ArgumentParser(prog="uitest.py", description=__doc__.splitlines()[0]) + sub = p.add_subparsers(dest="cmd", required=True) + r = sub.add_parser("run", help="run scenarios against a bundle") + r.add_argument("bundle") + r.add_argument("scenarios", help="a scenario .json file or a directory of them") + r.add_argument("--card-host") + r.add_argument("--out", help="evidence directory (default /build/uitest)") + r.add_argument("--keep-going", action="store_true", help="run every scenario even after a failure") + d = sub.add_parser("doctor", help="check that card-host and a display are available") + d.add_argument("--card-host") + args = p.parse_args(argv) + return cmd_run(args) if args.cmd == "run" else cmd_doctor(args) + + +if __name__ == "__main__": + sys.exit(main())