Skip to content

feat: add native drag gestures with VOM targets - #513

Open
iuyo5678 wants to merge 5 commits into
mainfrom
feat/drag-interactions
Open

iuyo5678 wants to merge 5 commits into
mainfrom
feat/drag-interactions

Conversation

@iuyo5678

@iuyo5678 iuyo5678 commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Web applications often require a continuous press, optional hold, movement and release. Add bsk drag / tool.drag so these interactions run as one bounded native-input operation, with VOM refs identifying elements even when names repeat.

Implementation

  • Accept a source and destination ref/selector, or a relative delta, with hold, duration, offsets, button and modifiers. Selectors must identify exactly one element; optional observation IDs reject expired refs.
  • Reuse target resolution, frame geometry, input readiness, overlay bypass and shared-control checks. Protect both source and destination, and validate targets before pressing.
  • Extract the existing file-transfer pointer transaction for reuse. Cancellation and timeouts clean up only input owned by the operation, including native drag cancellation; uncertain delivery is surfaced instead of retried.
  • Expose draggable, drop-target and drag-handle cues in VOM and document the CLI and interaction limits.

Examples:

bsk drag @e3 --to @e8 --session "$SESSION" --context "$CONTEXT"
bsk drag @e3 --dx 180 --hold 400ms --session "$SESSION" --context "$CONTEXT"

Validation

  • Rust workspace: 911 tests passed; formatting and Clippy with warnings denied passed.
  • Extension: 3,253 tests passed (311 browser/optional tests skipped in the default suite); type checking and Biome passed.
  • Explicit Chrome drag regression: 35 browser tests passed, including real SortableJS native/fallback modes and dnd-kit; 14 drag unit tests and 2 transaction-boundary tests passed.
  • Full CLI → daemon → installed extension passed on Chrome 156: mouse drag, long press, native DataTransfer drop, range, resize, sort, timeout recovery and cross-origin iframe refs.
  • VOM: 125 tests passed. Browser eval framework: 29 tests passed. Skill checks and frozen-lockfile installation passed.

Fixtures live in evals/browser/cases/regression/drag-interactions. Assertions check DOM order, values, pixels, trusted events and native DataTransfer contents, including duplicate-name refs, stale observations, covered/detached elements, cancellation and background tabs. CI runs the browser suite and installed-extension runner and retains evidence.

Self-review covered input ownership, cancellation races, attachment replacement, selector ambiguity, observation expiry and shared-control destination checks. Test dependencies are pinned and reuse the existing build-tool version.

Scope

Desktop mouse gestures within one tab, with both endpoints visible after initial scrolling. No scrolling while holding or touch gestures. Cross-origin frame refs support default action points and relative movement; custom offsets within those frames are rejected. Successful input delivery still requires observing the application result; cancellation does not roll back changes already made by the page.

CI follow-up

Keep the skill entry below its existing 7,100-byte budget under both LF and CRLF. Fix an existing remote-server integration-test race: receiving a WebSocket close frame precedes release of the browser capacity slot, so the final connection waits at most three seconds and retries only the exact capacity response. Add operation context to authorization filesystem errors to diagnose a separate intermittent Windows access-denied failure without changing storage behavior. Local script tests (18), authorization tests (6), remote-server integration tests (6), formatting and Clippy pass.

Review fixes

  • Check both drag endpoints through all cross-process frame ancestors at the actual input point. Shared hit testing inverse-projects the point through frame geometry and rejects parent/intermediate overlays before pressing.
  • Carry intentional source offsets through the production pointer validator while retaining movement and hit checks. Browser drag tests now install this validator by default.
  • Recover existing textbox/button roles, names, values and states before adding drag hints. Draggable cards retain fill/click semantics.

Added regression coverage for nested frame occlusion, projection and stale frame identity, positive/negative source offsets, covered/moved/outside offset points, and combined editable/clickable drag semantics. The installed-extension runner verifies exact Canvas input coordinates/pixels, source and destination offsets, normal fill/click actions on draggable cards, blocked nested-frame drops with zero presses/drops, and successful unobstructed native drops.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants