Skip to content

Add ControllersToGamepadRetargeter - #1255

Draft
rwiltz wants to merge 5 commits into
mainfrom
rwiltz/gamepad-controllers
Draft

rwiltz wants to merge 5 commits into
mainfrom
rwiltz/gamepad-controllers

Conversation

@rwiltz

@rwiltz rwiltz commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Description

Maps a pair of XR controllers (ControllersSource) to gamepad_buttons and gamepad_axes in the standard gamepad layout, so the gamepad retargeters (SE3, SE2, gripper) drive a robot from controllers
unchanged.

Mapping

  • Thumbsticks → LEFT_X/Y, RIGHT_X/Y (OpenXR's up-positive Y passes straight through)
  • Triggers → LEFT/RIGHT_TRIGGER; grips → LEFT/RIGHT_SHOULDER (above grip_threshold)
  • A / B (right) → SOUTH / EAST; X / Y (left) → WEST / NORTH; thumbstick clicks → LEFT/RIGHT_STICK; left menu → START
  • Controllers have no D-pad: dpad_stick="left"|"right" makes that thumbstick the D-pad while its grip is held (enables SE3 roll/pitch). Off by default.
  • A missing controller reads as released and centered; outputs are absent only while both are.

The gamepad printer example gains --source controllers (and --dpad-stick) to try it from a headset.

Fixes #(issue)

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update

Testing

Testing

  • 8 unit tests with synthetic controller input, including an end-to-end gripper toggle on left X.
  • Tested on a Quest 3 through CloudXR with the printer example.

Checklist

  • I have read and understood the contribution guidelines
  • I have run the linter and formatter with SKIP=check-copyright-year pre-commit run --all-files
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix/feature works (or explained why not)
  • I have signed off all my commits (git commit -s) per the DCO

@coderabbitai

coderabbitai Bot commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@rwiltz
rwiltz force-pushed the rwiltz/gamepad-plugin branch from 354c7c7 to bd0f410 Compare October 13, 2026 16:52
@rwiltz
rwiltz force-pushed the rwiltz/gamepad-controllers branch from 7c85af4 to 4fc2a61 Compare October 13, 2026 16:54
@rwiltz
rwiltz force-pushed the rwiltz/gamepad-plugin branch from bd0f410 to 508e4bc Compare October 13, 2026 20:54
@rwiltz
rwiltz force-pushed the rwiltz/gamepad-controllers branch from 4fc2a61 to 75ca8a2 Compare October 13, 2026 20:55
rwiltz and others added 4 commits October 14, 2026 13:18
GamepadInputState holds the inputs every standard gamepad has, the same
set as OpenXR's xbox_controller profile, as named fields in the style of
ControllerInputState: a generic controller split in two. Face buttons are
named by position (south_click is A on an Xbox pad and Cross on a
PlayStation pad), the thumbsticks are in [-1, 1] with right and up
positive, and the triggers are in [0, 1]. GamepadSnapshot carries it,
absent while the plugin has no pad open. GamepadTracker is generated from its
trackers.toml entry, so it records and replays like the other
schema-based trackers.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Rafael Wiltz <rwiltz@nvidia.com>
Reads a gamepad through SDL3 and pushes GamepadSnapshot at 90 Hz via
SchemaPusher. SDL's drivers and SDL_GameControllerDB mappings resolve
each pad's own numbering into the standard layout, so Xbox, PlayStation,
Switch Pro, Steam and any mapped pad read the same, and the mapping is
resolved once per device in the plugin rather than by every reader.
--player=N reads the Nth connected pad and keeps it until it
disconnects, so several pads run as one plugin each on their own
collection. While no pad is open the plugin keeps pushing snapshots
without inputs, so a reader never holds the last stick deflection
of a pad that went away. The pad's name, type and USB IDs are logged when
it is opened.

SDL3 (zlib) is fetched at configure time and linked statically with only
its joystick and HIDAPI subsystems, so the bundled binary still needs
nothing beyond libc/libstdc++. core/ holds the reader, unit-tested
against SDL virtual gamepads; app/ is the OpenXR push loop. With
--plugin-root-id the plugin reports its device as CONNECTED, DISCONNECTED
or FAILED through PluginDeviceStatusPublisher.

The plugin is bundled into the isaaccapture package
(isaaccapture/plugins/gamepad), so a wheel install can start it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Rafael Wiltz <rwiltz@nvidia.com>
A TeleopSession that starts the gamepad plugin now tracks its
gamepad/device_status collection, so the pad's CONNECTED, DISCONNECTED
and FAILED states reach get_device_status() like the SpaceMouse's.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Rafael Wiltz <rwiltz@nvidia.com>
GamepadSource reads the generated GamepadTracker and outputs a gamepad
tensor group (GamepadInput, indexed by GamepadInputIndex) with the
fields of GamepadInputState, absent while the plugin is not pushing or
has no pad open.

GamepadToSe3RelRetargeter and GamepadToSe2Retargeter bind the sticks and
D-pad by position, so they drive the same way from any supported pad, to
an end-effector delta and a base velocity. Their dead zone defaults to
0.15, since an idle pad's sticks rest up to 0.14 off center.
GamepadGripperRetargeter toggles on the west face button (X on Xbox,
Square on PlayStation) and lives with the other gripper retargeters, so
it needs no scipy.

Adds a printer example, which starts the bundled plugin and shows the raw
input, the retargeters' outputs and the poses a downstream app would
integrate from them, and the device page.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Rafael Wiltz <rwiltz@nvidia.com>
@rwiltz
rwiltz force-pushed the rwiltz/gamepad-plugin branch from 508e4bc to af35e20 Compare October 14, 2026 17:20
Joins a pair of XR controllers (ControllersSource) into one GamepadInput,
so the gamepad retargeters drive a robot from controllers unchanged: a
gamepad is a generic controller split in two. Thumbsticks, triggers and
grips become the sticks, triggers and shoulders; A/B and X/Y keep the
positions an Xbox pad gives them (south/east on the right, west/north on
the left), and the left menu button is menu. Controllers have no D-pad,
so dpad_stick optionally makes one thumbstick the D-pad while its grip is
held. The output is absent only while both controllers are.

The gamepad printer example gains --source controllers to try it from a
headset.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Rafael Wiltz <rwiltz@nvidia.com>
@rwiltz
rwiltz force-pushed the rwiltz/gamepad-controllers branch from 75ca8a2 to c5e67f5 Compare October 14, 2026 17:21
Base automatically changed from rwiltz/gamepad-plugin to main October 14, 2026 17:43

This branch was successfully deployed

1 active deployment
dev — c5e67f57 Deployed Oct 14, 2026 by rwiltz via publish-wheel #5623
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants