Skip to content

feat: add continuous multi-touch and long-press drag gestures - #163

Draft
yuan0818 wants to merge 3 commits into
google:mainfrom
yuan0818:feat/continuous-gestures
Draft

yuan0818 wants to merge 3 commits into
google:mainfrom
yuan0818:feat/continuous-gestures

Conversation

@yuan0818

Copy link
Copy Markdown

Summary

This PR extends Artemis' existing Android input path with continuous multi-touch
and long-press drag gestures that cannot be represented reliably by a normal
swipe. perform_gesture is added alongside the existing input primitives and
uses the existing Operator, Validator, MCP, controller, and driver path.

  • Support simultaneous pointers and continuous hold/move phases without lifting
    contacts between phases.
  • Provide a generic long_press_drag mode with explicit start and end points,
    an initial hold, and an optional delay before release at the endpoint.
  • Support cubic Bezier paths with caller-selected endpoints anywhere on screen.
    Edge dwell uses the same drag mode with an endpoint chosen near an edge.
  • Update the bundled helper to v1.3.3 with matching capability negotiation and
    APK metadata.

Existing click, swipe, text input, transcript/history representation, and action
execution semantics remain unchanged. Results use the existing ActionResult;
this change introduces no global execution receipt or gesture agent.

Compatibility and execution

Helpers that do not advertise the required gesture capabilities return
UNSUPPORTED. Continuous phases and long-press drag require Android API 26+.
Multi-touch and continuous input are never emulated as sequential swipes.

perform_gesture is not automatically retried after transport timeout because
Android may already have dispatched the input. This rule is scoped to the new
action; existing action retry behavior is unchanged.

Cancellation releases contacts after the current bounded phase and prevents
subsequent phases from starting. Helper-side busy/release protection applies to
helper gesture injection; it does not arbitrate raw ADB input.

Validation

  • Python gesture contract, compatibility, and existing action regression tests.
  • Android JVM/Robolectric gesture construction, continuation, cancellation,
    release, and busy-state tests.
  • Formatting, lint, and type checks passed locally.
  • Local Python suite: 2,556 passed, 80 failed, 7 skipped, 8 deselected. All 80
    failing cases also failed against baseline 351ca84 production code in the
    same local environment. The local suite is not fully green.

Real-device validation on A065 / Android 16 (1080 x 2412), using an installed APK
whose SHA-256 matches the bundled artifact:

Scenario Result
Two-finger pinch Pass
Two-finger zoom Pass
Three-finger swipe Pass
Explicit continuous hold and drag Pass
Generic long-press drag Pass
Bezier drag to an interior endpoint Pass
Caller-selected edge endpoint with delayed release Pass
Cancellation during a hold Pass
Ordinary click after cancellation Pass

The native test app recorded actual Android MotionEvents. The Bezier test ended
at normalized [650, 550] with 0.35 px release-position error and a measured
505 ms stationary interval before release for a requested 500 ms delay. The
edge test measured 1,006 ms for a requested 1,000 ms delay. Cancellation requested
about 500 ms into a 1,800 ms hold released at 1,804 ms without starting movement.
These checks cover live input integration; they do not cover LLM task planning.

@google-cla

google-cla Bot commented Sep 30, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant