Repository navigation
fix: pan VNC with two fingers and show a soft keyboard on mobile - #53
Merged
LukeGus merged 10 commits intoNov 16, 2026
Merged
Conversation
Two-finger moves on VNC/ARD no longer only pinch-zoom: a stable spread pans the overflow container, and a pinch zooms around the fingers. preventDefault runs only after the gesture is classified. Mobile browsers never showed an on-screen keyboard because input was bound to a display div. A toolbar button on touch devices now focuses a hidden textarea and forwards keys to the session.
Pinch scroll math treated the scroll origin as the display origin, so a flex-centered framebuffer jumped when zooming around the fingers. Measure the inner Guacamole element's offset after applyZoom, the same way as the pan-only branch.
Promote a pan to a pinch once the spread crosses PINCH_PX, and do not lock as pan on a sub-pixel midpoint shift. Take ownership of claimed two-finger gestures so Guacamole's touchpad does not emit a right-click, while still letting the initial start through so a touchscreen long-press is cancelled. Handle software-keyboard text via native beforeinput (React does not expose inputType), including deletion and composition.
…ste breaks A 2-to-1 lift no longer drops a claimed gesture before the last finger is up, so the emulator cannot treat a short pan as a two-finger tap. Cancel the first-finger long-press with a one-finger spoil move instead of forwarding a two-finger move that scrolled the remote wheel. Keep the software-keyboard button on the visible toolbar row so overflow-menu focus does not dismiss the OSK. Map pasted newlines and tabs to Return and Tab keysyms.
… keys Guacamole's touchscreen emulator judges a tap from the final finger position, so a pinch whose first finger stayed still could end in a left click, and its long-press timer could right-click during a held pinch. Replace the synthetic spoil move with a pointer gate that drops all touch-emulator output from the moment a two-finger gesture is claimed until the next fresh single-finger touch, releases any button held when the gesture starts, and drops compatibility mouse presses for 800 ms after the last finger lifts. Special keys typed into the soft-keyboard textarea bypass Guacamole.Keyboard's pressed-key tracking. Track them and release every held key on textarea blur, window blur, page hide, visibility loss, client change, disconnect and teardown.
Reset the touch emulator before a fresh touch reopens the gesture gate, and drop emulator states that hold a button the remote never saw pressed until a released state arrives. Replay each autorepeat of a held soft-keyboard key as a key up/down pair so held Backspace, Delete and arrows repeat on the remote.
reset() only releases buttons; the Guacamole emulators keep private timers and gesture state. After a quick pinch the Touchpad measured the next touch from the old finger and jumped the cursor, and the Touchscreen kept the long-press timer of a cancelled pinch. Each emulator now runs on a proxy host that only the current instance receives touch events through; it is retired and recreated when touches are re-enabled after a claimed gesture and when every touch is cancelled, and states from a retired instance are dropped.
The touchscreen emulator releases a tap from a 250 ms timer. sendMouseState focused the display for every forwarded state, so opening the soft keyboard right after a tap lost focus and closed the phone keyboard. Forwarded states no longer take focus while the soft-keyboard textarea is focused, and real mousedown/touchstart presses on the display focus it in the capture phase.
event.touches also lists fingers outside the display, such as one resting on the toolbar, whose touchend never reaches the container. A claimed gesture could then stay owned after every finger lifted and swallow the next tap. Count only touches that started inside the container, and finalize a claimed gesture from a document-level touchend/touchcancel when no display touches remain.
|
AI review | Looks safe to merge Adds two-finger pan and pinch support for VNC/ARD sessions and a hidden soft keyboard textarea for mobile input. The implementation safely manages touch event ownership, emulator lifecycle, and focus state without introducing security risks or bugs. Reviewed faf346d. AI can be wrong, a maintainer still decides. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On a phone, a VNC/ARD session could only be pinch-zoomed. Two fingers could not pan the zoomed display, and there was no way to bring up the on-screen keyboard. This PR adds:
Root causes
preventDefaultand only zoomed from the finger distance, so the scroll container could not move.divthroughGuacamole.Keyboard. Mobile browsers only show the on-screen keyboard for a focused text field.Guacamole.Mouse.Touchscreenjudges a gesture from the first finger only.Guacamole.Mouse.Touchpadstays mid-gesture after a quick pinch and measures the next touch from the old finger, so the cursor jumps.Touchscreenignorestouchcancel, so the long-press timer of a cancelled pinch still fires.sendMouseState()focused the display for every forwarded mouse state. That included the tap release the touchscreen emulator fires from a 250 ms timer, so opening the keyboard within 250 ms of a tap moved focus back to the display and closed the phone keyboard.event.touches, which also lists a finger resting on the toolbar. That finger's finaltouchendnever reaches the container, so a claimed gesture could stay owned and swallow the next tap.Guacamole.Keyboardand its client-side key repeat.Changes
guacamole-touch-gestures.ts(new, pure): classify pinch vs pan, zoom around the fingers, and pan by adjustingscrollLeft/scrollTop.GuacamoleDisplay.tsxowns a two-finger sequence until the last finger lifts and callspreventDefaultonly for claimed gestures. RDP, telnet and RustDesk behaviour is unchanged.guacamole-pointer.ts, aTouchGestureGatebetween the emulators and the client:guacamole-soft-keyboard.ts(new):beforeinputhandling, and paste line breaks;pagehide, hidden visibility and disconnect;GuacamoleDisplay.tsxfocus: forwarded mouse states no longer take focus while the soft-keyboard textarea is focused, so hover-to-type on desktop is unchanged. A realmousedown/touchstarton the display always focuses it (capture phase).GuacamoleDisplay.tsxgestures: only touches that started inside the display container count. A claimed gesture is also finalized from a document-leveltouchend/touchcancelwhen no display touches remain.GuacamoleToolbar.tsx/toolbar-items.ts: a touch-only "Show keyboard" item (newenstrings).No new dependencies and no CI changes.
Verification
On
677927a, Node 24 (.nvmrc), the same steps as the plugin CI:npm test: 72 files passed, 1 skipped; 612 tests passed, 1 skippednpm run typecheck✅npm run format:check✅npm run build✅npx termix-plugin validate✅ (the KUBERNETES_SERVICE_HOST and LoadingState notes are pre-existing)npx termix-plugin pack✅New tests:
guacamole-touch-gestures.test.ts: thresholds, zoom clamp, the point under the fingers stays put, pan.guacamole-pointer.test.ts:Touchscreen/Touchpademulators in jsdom.left: true.guacamole-soft-keyboard.test.ts: keysyms, IME, held-key release, and autorepeat as up/down pairs.GuacamoleDisplay-focus.test.tsx:GuacamoleDisplaymounted with the real touchscreen emulator. The soft keyboard stays focused through a tap's delayed release, a real press or mouse move focuses the display, and a finger resting outside the display does not swallow the next tap.GuacamoleToolbar.test.tsx: the keyboard button is shown only with a touch mode.The new pointer, keyboard, focus and touch-ownership tests fail against the earlier versions of these files.
Browser check (Chrome, 390×760 mobile viewport, DPR 3, touch via CDP)
A small harness bundles
guacamole-pointer.tsandguacamole-soft-keyboard.tswith the real Guacamole emulators, plus a two-finger handler and focus handling that mirrorGuacamoleDisplay.tsx. The log lists what would be sent to the remote. "Before" is an earlier revision of this branch;mainhas no two-finger handling at all.Not tested yet on a physical phone in a live VNC/ARD session.