Skip to content

fix: pan VNC with two fingers and show a soft keyboard on mobile - #53

Merged
LukeGus merged 10 commits into
Termix-SSH:dev-1.4.2from
Rhiz3K:fix/vnc-touch-pan-keyboard
Nov 16, 2026
Merged

LukeGus merged 10 commits into
Termix-SSH:dev-1.4.2from
Rhiz3K:fix/vnc-touch-pan-keyboard

Conversation

@Rhiz3K

@Rhiz3K Rhiz3K commented Nov 11, 2026

Copy link
Copy Markdown

Summary

On a phone, a VNC/ARD session could only be pinch-zoomed. Two fingers could not pan the zoomed display, and there was no way to bring up the on-screen keyboard. This PR adds:

  • Two-finger pan and pinch for VNC/ARD. Pinch and pan are told apart with an 8 px threshold, and zoom stays centred under the fingers.
  • A "Show keyboard" toolbar button on touch devices (VNC, ARD and RDP). It focuses a hidden textarea and forwards characters and special keys to the session. It is not shown without a touch mode.
  • No stray clicks or drags around two-finger gestures from Guacamole's touch emulators.

Root causes

  • Pan: the VNC/ARD capture-phase touch handler always called preventDefault and only zoomed from the finger distance, so the scroll container could not move.
  • Keyboard: input was bound to the display div through Guacamole.Keyboard. Mobile browsers only show the on-screen keyboard for a focused text field.
  • Clicks and drags around gestures: Guacamole.Mouse.Touchscreen judges a gesture from the first finger only.
    • A pinch whose first finger stays still ends in a left press with a 250 ms delayed release. A touch inside that window cancels the release, so the next one-finger move became a drag.
    • Its long-press timer can right-click during a held pinch.
    • Guacamole.Mouse.Touchpad stays mid-gesture after a quick pinch and measures the next touch from the old finger, so the cursor jumps.
    • Touchscreen ignores touchcancel, so the long-press timer of a cancelled pinch still fires.
  • Soft keyboard closed right after a tap: sendMouseState() focused the display for every forwarded mouse state. That included the tap release the touchscreen emulator fires from a 250 ms timer, so opening the keyboard within 250 ms of a tap moved focus back to the display and closed the phone keyboard.
  • Fingers outside the display: the two-finger handler counted event.touches, which also lists a finger resting on the toolbar. That finger's final touchend never reaches the container, so a claimed gesture could stay owned and swallow the next tap.
  • Held special keys: keys typed into a textarea bypass Guacamole.Keyboard and its client-side key repeat.

Changes

  • guacamole-touch-gestures.ts (new, pure): classify pinch vs pan, zoom around the fingers, and pan by adjusting scrollLeft/scrollTop. GuacamoleDisplay.tsx owns a two-finger sequence until the last finger lifts and calls preventDefault only for claimed gestures. RDP, telnet and RustDesk behaviour is unchanged.
  • guacamole-pointer.ts, a TouchGestureGate between the emulators and the client:
    • It drops all touch-emulator output from a claimed two-finger gesture until a fresh single-finger touch, and releases a button held when the gesture starts.
    • It drops compatibility mouse presses for 800 ms afterwards.
    • It never forwards a button the remote did not see pressed.
  • The touch emulator is retired and recreated, keeping the cursor position, when touches are re-enabled after a gesture and when all touches are cancelled. It runs on a small proxy host, so only the current instance receives touch events and stale timers of a retired one are ignored.
  • guacamole-soft-keyboard.ts (new):
    • keysym mapping, IME/composition and beforeinput handling, and paste line breaks;
    • a pressed-key tracker that releases held keys on blur, pagehide, hidden visibility and disconnect;
    • autorepeat of a held key sent as key up + key down.
  • GuacamoleDisplay.tsx focus: forwarded mouse states no longer take focus while the soft-keyboard textarea is focused, so hover-to-type on desktop is unchanged. A real mousedown/touchstart on the display always focuses it (capture phase).
  • GuacamoleDisplay.tsx gestures: only touches that started inside the display container count. A claimed gesture is also finalized from a document-level touchend/touchcancel when no display touches remain.
  • GuacamoleToolbar.tsx / toolbar-items.ts: a touch-only "Show keyboard" item (new en strings).

No new dependencies and no CI changes.

Verification

On 677927a, Node 24 (.nvmrc), the same steps as the plugin CI:

  • npm test: 72 files passed, 1 skipped; 612 tests passed, 1 skipped
  • npm run typecheck ✅
  • npm run format:check ✅
  • npm run build ✅
  • npx termix-plugin validate ✅ (the KUBERNETES_SERVICE_HOST and LoadingState notes are pre-existing)
  • npx termix-plugin pack ✅

New tests:

  • guacamole-touch-gestures.test.ts: thresholds, zoom clamp, the point under the fingers stays put, pan.
  • guacamole-pointer.test.ts:
    • The gate, plus the real Touchscreen/Touchpad emulators in jsdom.
    • A pinch and then a move within 250 ms never sends left: true.
    • There is no touchpad jump after a quick pinch, and no right click after a cancelled pinch.
    • A plain tap still clicks.
  • guacamole-soft-keyboard.test.ts: keysyms, IME, held-key release, and autorepeat as up/down pairs.
  • GuacamoleDisplay-focus.test.tsx: GuacamoleDisplay mounted with the real touchscreen emulator. The soft keyboard stays focused through a tap's delayed release, a real press or mouse move focuses the display, and a finger resting outside the display does not swallow the next tap.
  • GuacamoleToolbar.test.tsx: the keyboard button is shown only with a touch mode.

The new pointer, keyboard, focus and touch-ownership tests fail against the earlier versions of these files.

Browser check (Chrome, 390×760 mobile viewport, DPR 3, touch via CDP)

A small harness bundles guacamole-pointer.ts and guacamole-soft-keyboard.ts with the real Guacamole emulators, plus a two-finger handler and focus handling that mirror GuacamoleDisplay.tsx. The log lists what would be sent to the remote. "Before" is an earlier revision of this branch; main has no two-finger handling at all.

Scenario Before After
Two-finger pan, remote button presses 0 0
Pinch (first finger still), lift, new one-finger move within 250 ms 7 drag moves 0
Plain tap 1 left click 1 left click
Held Backspace (1 press + 5 repeats) 1 key press 6 key presses
Touchpad: quick pinch, then a 1 px move cursor jumped 1228 px 1 px
Cancelled pinch, then a fresh touch held 350 ms 1 right click 0
Tap, then open the keyboard 100 ms later; focus after the delayed release (+250 ms) display (keyboard closes) keyboard textarea
Before After
Pinch then move
Tap then keyboard
Held Backspace

Not tested yet on a physical phone in a live VNC/ARD session.

Two-finger moves on VNC/ARD no longer only pinch-zoom: a stable spread
pans the overflow container, and a pinch zooms around the fingers.
preventDefault runs only after the gesture is classified.

Mobile browsers never showed an on-screen keyboard because input was
bound to a display div. A toolbar button on touch devices now focuses a
hidden textarea and forwards keys to the session.
Pinch scroll math treated the scroll origin as the display origin,
so a flex-centered framebuffer jumped when zooming around the fingers.
Measure the inner Guacamole element's offset after applyZoom, the same
way as the pan-only branch.
Promote a pan to a pinch once the spread crosses PINCH_PX, and do not
lock as pan on a sub-pixel midpoint shift. Take ownership of claimed
two-finger gestures so Guacamole's touchpad does not emit a right-click,
while still letting the initial start through so a touchscreen long-press
is cancelled. Handle software-keyboard text via native beforeinput (React
does not expose inputType), including deletion and composition.
…ste breaks

A 2-to-1 lift no longer drops a claimed gesture before the last finger
is up, so the emulator cannot treat a short pan as a two-finger tap.
Cancel the first-finger long-press with a one-finger spoil move instead
of forwarding a two-finger move that scrolled the remote wheel. Keep the
software-keyboard button on the visible toolbar row so overflow-menu
focus does not dismiss the OSK. Map pasted newlines and tabs to Return
and Tab keysyms.
… keys

Guacamole's touchscreen emulator judges a tap from the final finger
position, so a pinch whose first finger stayed still could end in a left
click, and its long-press timer could right-click during a held pinch.
Replace the synthetic spoil move with a pointer gate that drops all
touch-emulator output from the moment a two-finger gesture is claimed
until the next fresh single-finger touch, releases any button held when
the gesture starts, and drops compatibility mouse presses for 800 ms
after the last finger lifts.

Special keys typed into the soft-keyboard textarea bypass
Guacamole.Keyboard's pressed-key tracking. Track them and release every
held key on textarea blur, window blur, page hide, visibility loss,
client change, disconnect and teardown.
Reset the touch emulator before a fresh touch reopens the gesture gate, and drop emulator states that hold a button the remote never saw pressed until a released state arrives. Replay each autorepeat of a held soft-keyboard key as a key up/down pair so held Backspace, Delete and arrows repeat on the remote.
reset() only releases buttons; the Guacamole emulators keep private timers and gesture state. After a quick pinch the Touchpad measured the next touch from the old finger and jumped the cursor, and the Touchscreen kept the long-press timer of a cancelled pinch. Each emulator now runs on a proxy host that only the current instance receives touch events through; it is retired and recreated when touches are re-enabled after a claimed gesture and when every touch is cancelled, and states from a retired instance are dropped.
The touchscreen emulator releases a tap from a 250 ms timer. sendMouseState focused the display for every forwarded state, so opening the soft keyboard right after a tap lost focus and closed the phone keyboard. Forwarded states no longer take focus while the soft-keyboard textarea is focused, and real mousedown/touchstart presses on the display focus it in the capture phase.
event.touches also lists fingers outside the display, such as one resting on the toolbar, whose touchend never reaches the container. A claimed gesture could then stay owned after every finger lifted and swallow the next tap. Count only touches that started inside the container, and finalize a claimed gesture from a document-level touchend/touchcancel when no display touches remain.
@termix-triage-bot

termix-triage-bot Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

AI review | Looks safe to merge

Adds two-finger pan and pinch support for VNC/ARD sessions and a hidden soft keyboard textarea for mobile input. The implementation safely manages touch event ownership, emulator lifecycle, and focus state without introducing security risks or bugs.

Reviewed faf346d. AI can be wrong, a maintainer still decides. Comment /termix review to run it again.

@LukeGus
LukeGus changed the base branch from main to dev-1.4.2 November 16, 2026 23:18

@LukeGus LukeGus left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@LukeGus
LukeGus merged commit 71a6522 into Termix-SSH:dev-1.4.2 Nov 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants