Skip to content

feat: make browser control reusable on dynamic pages - #530

Merged
iuyo5678 merged 12 commits into
mainfrom
feature/targeted-browser-control
Nov 1, 2026
Merged

iuyo5678 merged 12 commits into
mainfrom
feature/targeted-browser-control

Conversation

@iuyo5678

@iuyo5678 iuyo5678 commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Dynamic page updates and supplementary observations can invalidate the context for the next action. This change lets callers inspect a page, refine a decision, and perform multiple DOM-targeted actions with one control receipt while preserving human takeover and input dispatch checks.

Behavior

  • Preserve DOM references across full and partial observations. References identify physical nodes; document replacement invalidates them, and visual references retain their capture freshness rules.
  • Negotiate auto / target / strict policies. Publish the initial policy and refs only after a successful sync. Failed or overlapping initialization cannot overwrite an accepted policy or rebind issued refs.
  • Check a DOM action's recipient and explicit expect conditions without comparing the whole page. Scripts, untargeted keys, visual actions and displacement-only drags require an explicit, single-use observation. Protect relevant prices, record IDs and action text explicitly; a stable node does not imply unchanged business facts.
  • During readiness waits, recheck the target without collecting every frame on each retry. Initial collection, human-input notifications and the final collection before effects remain enforced. Hidden blur does not wait for visibility. No fixed delays or automatic action replay were added.
  • Keep pointer hit tests within their target document. Preserve confirmed handler completion across ordinary frame navigation or failed post-action collection while revoking stale control. Human intervention, stop, cancellation and unsuccessful script results remain enforced.
  • Preserve original selectors in successful receipts. Errors expose reason, phase, effect state and recovery; diagnostics explain iframe selector scope. Shared action schemas expose control metadata, and all expectation validators count Unicode code points consistently.
  • Keep browser evaluation readiness explicit. Configure mobile device setup before navigation, require the site's exact viewport event, and sample OOPIF scrollbars after layout readiness. Real browser tests separately cover emulation of an already loaded page under both policies.

Review fixes

  • Merged current main, retaining the completed-action handling and avoiding a duplicate unguarded post-action collection.
  • Reject non-existence checks on detached nodes, including attribute equals null; exists:false remains valid. A failed action check remains latched through handler fallbacks, so a price changing back cannot cause input after a refusal. Initial readiness retries retain their existing behavior.
  • Report revoked control instead of a stale readiness error. Target resolution failures, including drag destinations, explicitly report effect_state:none, allowing callers to select a new target without reconciling an unsent action.
  • Allow supplementary observations during target-policy drags while retaining document-replacement checks and strict-policy observation checks. Document endpoint and displacement drag receipts separately.
  • Account for recent human input in newly inserted or navigated frames and preserve the user's page. Reuse existing activity sequence tracking to ignore duplicates; uncollected documents cannot replace the current frame state.
  • Reject an explicit target-policy request during discovery against incompatible daemons, before any action or negotiation is sent. Reuse the status already collected. Bootstrap setup rejects explicit conditions that it cannot honor instead of silently discarding them.

Compatible protocol 1.x peers at version 1.8 or later are required for target control and conditions. Older extensions require explicit strict policy; conditions are never silently dropped. Auto keeps its existing acknowledgement requirement. Browser preconditions are not server transactions, and a completed input receipt does not establish application-level business success. Reconcile uncertain outcomes before another action.

Validation

Validation was repeated on the merged tree:

  • Extension: 3,521 passed, 358 opt-in browser tests skipped by the default run.
  • Plugin: 706 passed in the default run. The subsequently migrated real-browser suite passed 10/10 (strict and target policies across click, Enter, drag, fill and select), preserving thumbnail cancellation, single navigation and no-replay assertions.
  • Rust workspace: 990 passed, 1 ignored. Formatting and all-target Clippy with warnings denied passed.
  • Real Chrome: 29 controller/target-hit tests and 59 drag tests passed. New drag tests verify exact movement, strict refusal, document invalidation and button cleanup.
  • Browser harness: 46 passed; 12 case manifests and 21 routes valid. The original strict-policy assertions in the tests merged from main are retained explicitly; the target cases additionally verify context reuse and observation renewal.
  • TypeScript, Biome, Stylelint, extension/plugin builds and skill packaging checks passed. All 122 schemas parse and regenerate byte-for-byte.
  • New controller regressions were also run against the pre-fix implementation: 9 condition/error/bootstrap cases and 10 new-frame input cases failed there and pass with the fix. No production guard was relaxed to satisfy a test.

The full installed cooperation eval is not fully passing. On Chrome 156 it timed out in the foreground responsive fixture while the renderer reported hidden; the merged pre-fix revision reproduced the same failure. The isolated CLI/DSH compatibility suite passed. On Chrome for Testing 145, the full run failed an unrelated-tab visibility assertion (hidden instead of visible), while its expected untouched value and click count matched. These failures remain recorded; the full suite is not reported as green.

Validation limits

Browser validation uses isolated Chrome 156 and Chrome for Testing 145 with local fixtures; it does not establish compatibility with the reported production business website or provide semantic toast history.

The first Rust workspace run failed one existing auto_spawn daemon-stop assertion. The isolated test and final complete workspace run passed without changing that path or its assertions. The initial failure is retained in the validation record; its cause was not established.

The initial full cooperation eval timed out waiting for responsive text on a hidden page, although the actual viewport was 390 pixels and the media query had switched. The merged pre-fix revision reproduced the same failure; an extension-free Chrome reproduction also delayed the resize listener until the page was shown. The full Chrome 156 run remains a validation limitation. No emulation production logic, event delivery, assertions or timeouts were changed to make this test pass.

Publish initial control policy and refs only after a successful sync, preserving concurrent observations. Reduce readiness polling to target checks while keeping final dispatch fences, and preserve selector provenance in successful receipts.

Cover hidden blur, negotiation and observation overlap, Unicode limits, compatible-peer guidance, and shared action schemas with regression tests.
Apply device setup on the initial blank tab before opening the responsive fixture. Keep exact viewport, execution and cleanup assertions, and cover failed setup without navigation.

Retain loaded-page emulation coverage with real Chrome tests through strict and target control, including responsive text, viewport, touch and user-agent checks.
@iuyo5678
iuyo5678 merged commit 8cb85ed into main Nov 1, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants