diff --git a/docs/source/device/sharpa_avatar.rst b/docs/source/device/sharpa_avatar.rst index 1e3f7ca37b..68a89dcf0d 100644 --- a/docs/source/device/sharpa_avatar.rst +++ b/docs/source/device/sharpa_avatar.rst @@ -5,9 +5,9 @@ Sharpa AvatarExo Glove ====================== A Linux-only plugin for integrating Sharpa AvatarExo (AE) Gloves into the -Isaac Capture framework. The gloves stream IMU and encoder data through the -official Avatar SDK over USB. The plugin maps that stream into Isaac Capture's -usual surfaces: OpenXR hand tracking for the HUMAN skeleton, +Isaac Capture framework. The gloves stream IMU and encoder data wirelessly +through a USB dongle and the official Avatar SDK. The plugin maps that stream +into Isaac Capture's usual surfaces: OpenXR hand tracking for the HUMAN skeleton, ``JointStateOutput`` tensors for the RAW and ROBOT 22-DoF joint sets, and inbound per-finger vibration on a haptic collection. @@ -33,7 +33,7 @@ named joint-state collections: .. code-block:: text - Sharpa AvatarExo Gloves (USB) + Sharpa AvatarExo Gloves (wireless via USB dongle) │ ▼ avatar_hand_plugin (Avatar SDK, production package) @@ -56,9 +56,7 @@ Prerequisites - **Pinned production Avatar SDK** (``avatar-sdk`` 1.7.3-17) from Sharpa's signed APT repository. ``install_avatar_sdk.sh`` retrieves it; do not substitute ``avatar-sdk-dev`` or ``avatar-sdk-beta``. -- **Sharpa AvatarExo Gloves** powered and connected over USB. Transport settings - are selected in ``/share/sdk_config.json`` (default root - ``/opt/avatar-sdk``). +- **Sharpa AvatarExo Gloves** powered and paired with the USB wireless dongle. - **CloudXR runtime** on the plugin host, unless a ``TeleopSession`` example launches it for you. - A built Isaac Capture checkout. CMake 3.24 or newer is required. diff --git a/examples/sharpa_avatar/README.md b/examples/sharpa_avatar/README.md index 1806a04b04..37d426a599 100644 --- a/examples/sharpa_avatar/README.md +++ b/examples/sharpa_avatar/README.md @@ -48,8 +48,14 @@ above first. `uv pip install -e` pulls `viser`; CloudXR comes from the --no-launch-cloudxr-runtime connect to an already running CloudXR runtime --world-frame display the unmodified OpenXR poses --plugin-search-path DIR directory that contains sharpa_avatar/ +--sdk-config FILE Avatar SDK configuration file +--datasets LIST comma-separated human,raw,robot,haptic selection ``` +All four datasets and haptics are enabled by default, so they do not need to be +listed for a normal run. The supported workflow uses the USB dongle for a +wireless glove connection. This plugin does not accept a `--transport` option. + ## Behavior `live_view.py` launches the plugin through `TeleopSession`, reads `/hand/left` diff --git a/examples/sharpa_avatar/python/isaaccapture_examples/sharpa_avatar/live_view.py b/examples/sharpa_avatar/python/isaaccapture_examples/sharpa_avatar/live_view.py index ff154b364e..1e2a108c63 100644 --- a/examples/sharpa_avatar/python/isaaccapture_examples/sharpa_avatar/live_view.py +++ b/examples/sharpa_avatar/python/isaaccapture_examples/sharpa_avatar/live_view.py @@ -157,8 +157,8 @@ def _finger_chain(root: int, *joints: int) -> list[tuple[int, int]]: HandJointIndex.WRIST, HandJointIndex.LITTLE_METACARPAL, HandJointIndex.LITTLE_PROXIMAL, - HandJointIndex.LITTLE_INTERMEDIATE, - HandJointIndex.LITTLE_DISTAL, + # Avatar HUMAN has no little intermediate/distal landmarks; connect + # the real proximal and tip poses directly for display. HandJointIndex.LITTLE_TIP, ) ) diff --git a/src/plugins/sharpa_avatar/AGENTS.md b/src/plugins/sharpa_avatar/AGENTS.md index 560e94301d..913d996559 100644 --- a/src/plugins/sharpa_avatar/AGENTS.md +++ b/src/plugins/sharpa_avatar/AGENTS.md @@ -3,7 +3,7 @@ SPDX-FileCopyrightText: Copyright (c) 2025-2026 Avatar SDK contributors. All rig SPDX-License-Identifier: Apache-2.0 --> -# Sharpa Avatar plugin — agent notes +# Sharpa AvatarExo plugin — agent notes **CRITICAL:** Before editing this tree, read this file and the repo root [`AGENTS.md`](../../../AGENTS.md). @@ -34,6 +34,9 @@ SPDX-License-Identifier: Apache-2.0 - Installers must not delete files derived from a configured install prefix. - Do not mutate loader environment variables or SDK configuration at runtime. Runtime paths come from the selected SDK root and its `sdk_config.json`. +- Document the supported user workflow as wireless through the USB dongle. Do + not advertise wired, Ethernet, USB Type-C, or SDK link names outside focused + diagnostics. - Carry Avatar SDK enums such as `DeviceSide` and `DeviceDataCategory` through plugin internals; convert only at an existing external API boundary. - Adapt this plugin to shared APIs; do not refactor other plugins or shared @@ -48,11 +51,18 @@ SPDX-License-Identifier: Apache-2.0 - OpenXR setup is required for publication; initialization errors must abort construction instead of leaving an Avatar-only idle plugin. - Keep initialization non-blocking; `update()` owns retries for absent gloves. +- Reconcile glove handles from the SDK manager's discovery inventory, not the + handle's cached `online` snapshot. Gate publication and haptics on a fresh + ingress callback; do not add transport-specific parameter probes for liveness. +- Keep each side's HUMAN injector and RAW/ROBOT collections alive only while + that glove is streaming; destroy them on disconnect and recreate them after + fresh ingress so consumers cannot retain stale online data. - Keep the static HUMAN-to-OpenXR index table synchronized with the pinned production SDK's landmark order. Do not read joint names from `sdk_config.json` at runtime or fill unsupported OpenXR slots with neighbouring - poses. Refresh that table and the example's RAW/ROBOT name snapshots together - with the package pin. + poses. Publish RAW/ROBOT with their pinned semantic name tables, never the + SDK frame's positional `joint_` names. Refresh those tables and the + example's RAW/ROBOT name snapshots together with the package pin. - When reconciling the maintained feature branch, use it as the baseline and retain local divergence only for an explicit API or correctness requirement. - Keep this plugin tree C++-only (`core/` + `app/`). The TeleopSession sample diff --git a/src/plugins/sharpa_avatar/README.md b/src/plugins/sharpa_avatar/README.md index 1b26bb4dbd..8bad4f42e3 100644 --- a/src/plugins/sharpa_avatar/README.md +++ b/src/plugins/sharpa_avatar/README.md @@ -22,7 +22,7 @@ The visualization example is separate: - Linux x86_64 (Ubuntu 22.04) - A built Isaac Capture checkout -- Sharpa AvatarExo Gloves connected through a USB dongle or USB cable +- Sharpa AvatarExo Gloves connected wirelessly through the USB dongle `install_avatar_sdk.sh` retrieves a pinned production `avatar-sdk` version from Sharpa's signed production APT repository, without selecting the `avatar-sdk-dev` or @@ -72,8 +72,10 @@ or dongle: udev does not run inside containers, so this command must run on the host. The plugin follows the Sharpa host application layout and uses the SDK, configuration, and runtime assets directly from the selected SDK root -(`AVATAR_SDK_ROOT`, default `/opt/avatar-sdk`). Transport selection, including -wired Ethernet, is controlled by `/share/sdk_config.json`. +(`AVATAR_SDK_ROOT`, default `/opt/avatar-sdk`). The supported workflow uses the +USB dongle for a wireless glove connection. This plugin does not accept a +`--transport` option and must not run concurrently with `avatar-backend` or +Avatar Desktop. ## Run diff --git a/src/plugins/sharpa_avatar/core/avatar_hand_tracking_plugin.cpp b/src/plugins/sharpa_avatar/core/avatar_hand_tracking_plugin.cpp index 9a7df1454d..73f13b421f 100644 --- a/src/plugins/sharpa_avatar/core/avatar_hand_tracking_plugin.cpp +++ b/src/plugins/sharpa_avatar/core/avatar_hand_tracking_plugin.cpp @@ -23,7 +23,6 @@ #include #include #include -#include #include #include @@ -36,7 +35,6 @@ namespace { constexpr size_t kJointFlatbufferSize = 4096; -constexpr auto kAvatarDataTimeout = std::chrono::seconds(3); constexpr auto kGloveRetryInterval = std::chrono::seconds(2); constexpr auto kHapticCommandTimeout = std::chrono::milliseconds(200); @@ -46,6 +44,22 @@ const std::shared_ptr& logger() return instance; } +// Copied from avatar-sdk 1.7.3-17 share/sdk_config.json. RAW and ROBOT use +// different orders, while the SDK frame currently carries positional names. +constexpr std::array kRawJointNames = { + "thumb_CMC_FE", "thumb_CMC_AA", "thumb_MCP_FE", "thumb_MCP_AA", "thumb_IP", "index_MCP_AA", + "index_MCP_FE", "index_PIP", "index_DIP", "middle_MCP_AA", "middle_MCP_FE", "middle_PIP", + "middle_DIP", "ring_MCP_AA", "ring_MCP_FE", "ring_PIP", "ring_DIP", "pinky_MCP_AA", + "pinky_MCP_FE", "pinky_DIP1", "pinky_DIP2", "pinky_DIP3", +}; + +constexpr std::array kRobotJointNames = { + "thumb_CMC_FE", "thumb_CMC_AA", "thumb_MCP_FE", "thumb_MCP_AA", "thumb_IP", "index_MCP_FE", + "index_MCP_AA", "index_PIP", "index_DIP", "middle_MCP_FE", "middle_MCP_AA", "middle_PIP", + "middle_DIP", "ring_MCP_FE", "ring_MCP_AA", "ring_PIP", "ring_DIP", "pinky_CMC", + "pinky_MCP_FE", "pinky_MCP_AA", "pinky_PIP", "pinky_DIP", +}; + size_t side_index(::avatar::DeviceSide side) { switch (side) @@ -149,6 +163,20 @@ const ::avatar::Hand& hand_payload(const ::avatar::AvatarDataFrame& frame, ::ava throw std::invalid_argument("HUMAN data has no joint payload"); } +const std::array& joint_names(::avatar::DeviceDataCategory category) +{ + switch (category) + { + case ::avatar::DeviceDataCategory::RAW: + return kRawJointNames; + case ::avatar::DeviceDataCategory::ROBOT: + return kRobotJointNames; + case ::avatar::DeviceDataCategory::HUMAN: + break; + } + throw std::invalid_argument("HUMAN data has no joint names"); +} + constexpr std::string_view error_name(::avatar::ErrorCode error) { using ErrorCode = ::avatar::ErrorCode; @@ -249,6 +277,7 @@ void GloveState::reset() noexcept { if (device) { + device->set_raw_human_frame_callback({}); device->stop(); } } @@ -261,9 +290,9 @@ void GloveState::reset() noexcept raw_frame = {}; robot_frame = {}; device.reset(); - last_successful_fetch.reset(); - last_sample_stamp.reset(); - streaming = false; + fresh_sample_seen.store(false, std::memory_order_release); + started = false; + ready = false; } AvatarTracker::AvatarTracker(AvatarPluginConfig config) : m_config(std::move(config)), m_sdk(m_config.sdk_config_path) @@ -321,37 +350,6 @@ void AvatarTracker::initialize_openxr() m_time_converter.emplace(m_handles); - if (m_config.human) - { - for (const ::avatar::DeviceSide side : kDeviceSides) - { - m_injectors[side_index(side)] = std::make_unique( - m_handles.instance, m_handles.session, xr_hand(side), m_handles.space); - } - } - - auto make_joint_pusher = [this](::avatar::DeviceSide side, ::avatar::DeviceDataCategory category) - { - const std::string localized_name = - "Avatar " + std::string(to_string(category)) + " " + std::string(to_string(side)); - return std::make_unique( - m_handles, core::SchemaPusherConfig{ .collection_id = collection_id(side, category), - .max_flatbuffer_size = kJointFlatbufferSize, - .tensor_identifier = "joint_state", - .localized_name = localized_name }); - }; - for (const ::avatar::DeviceDataCategory category : kJointDataCategories) - { - if (!dataset_enabled(category)) - { - continue; - } - for (const ::avatar::DeviceSide side : kDeviceSides) - { - m_joint_pushers[side_index(side)][joint_category_index(category)] = make_joint_pusher(side, category); - } - } - m_deviceio_session = core::DeviceIOSession::run(trackers, m_handles); m_wrist_source = std::make_unique( wrist_config, m_handles, m_deviceio_session.get(), std::move(wrist_requirements.controller_tracker)); @@ -379,6 +377,10 @@ void AvatarTracker::start_glove_if_present(GloveState& glove, ::avatar::DeviceSi return; } glove.device->set_human_frame_build_enabled(m_config.human); + glove.fresh_sample_seen.store(false, std::memory_order_release); + GloveState* const state = &glove; + glove.device->set_raw_human_frame_callback([state](const ::avatar::AvatarDataFrame&, const ::avatar::AvatarDataFrame&) + { state->fresh_sample_seen.store(true, std::memory_order_release); }); const auto start_error = glove.device->start(); if (start_error != ::avatar::ErrorCode::SUCCESS) @@ -388,17 +390,11 @@ void AvatarTracker::start_glove_if_present(GloveState& glove, ::avatar::DeviceSi glove.reset(); return; } - glove.last_successful_fetch = std::chrono::steady_clock::now(); + glove.started = true; } void AvatarTracker::try_connect_missing_gloves() { - const auto has_device = [](const GloveState& state) { return state.device != nullptr; }; - if (std::all_of(m_gloves.begin(), m_gloves.end(), has_device)) - { - return; - } - const auto now = std::chrono::steady_clock::now(); if (m_last_glove_retry && now - *m_last_glove_retry < kGloveRetryInterval) { @@ -406,11 +402,28 @@ void AvatarTracker::try_connect_missing_gloves() } m_last_glove_retry = now; + const auto infos = m_sdk.get().list_device_info(::avatar::DeviceType::GLOVE); + const auto is_online = [&infos](::avatar::DeviceSide side) + { + return std::any_of(infos.begin(), infos.end(), + [side](const ::avatar::DeviceInfo& info) { return info.device_side == side && info.online; }); + }; for (const ::avatar::DeviceSide side : kDeviceSides) { - start_glove_if_present(glove(side), side); + GloveState& state = glove(side); + if (state.device && !is_online(side)) + { + logger()->info("{} glove disconnected.", to_string(side)); + destroy_output_resources(side); + state.reset(); + } + if (!state.device && is_online(side)) + { + start_glove_if_present(state, side); + } } // Log once per stretch with no glove; a glove connecting re-arms the message. + const auto has_device = [](const GloveState& state) { return state.device != nullptr; }; const bool waiting = std::none_of(m_gloves.begin(), m_gloves.end(), has_device); if (waiting && !m_glove_wait_logged) { @@ -419,36 +432,71 @@ void AvatarTracker::try_connect_missing_gloves() m_glove_wait_logged = waiting; } -void AvatarTracker::refresh_data() +void AvatarTracker::refresh_glove_readiness() { for (const ::avatar::DeviceSide side : kDeviceSides) { GloveState& state = glove(side); - if (!state.device) + if (!state.started || state.ready || !state.fresh_sample_seen.load(std::memory_order_acquire)) { continue; } - // Liveness comes from fetched data, not get_device_info().online: the SDK - // leaves that flag false on a glove first discovered offline and powered on later. - // A haptic-only glove still fetches RAW so the data timeout below can drop a dead handle. - const bool probe_raw = !m_config.human && !m_config.raw && !m_config.robot; - - // fetch_data() keeps returning a silent glove's last sample, so only a newer - // sample stamp counts as data for the timeout below. - bool fetched_any = false; - const auto note_stamp = [&state, &fetched_any](const ::avatar::Stamp& stamp) + create_output_resources(side); + state.ready = true; + state.device->set_raw_human_frame_callback({}); + logger()->info("{} glove connected and streaming.", to_string(side)); + } +} + +void AvatarTracker::create_output_resources(::avatar::DeviceSide side) +{ + const size_t index = side_index(side); + std::unique_ptr injector; + std::array, 2> joint_pushers; + + if (m_config.human) + { + injector = std::make_unique( + m_handles.instance, m_handles.session, xr_hand(side), m_handles.space); + } + for (const ::avatar::DeviceDataCategory category : kJointDataCategories) + { + if (!dataset_enabled(category)) { - if (!state.last_sample_stamp) - { - state.last_sample_stamp = stamp; - } - else if (std::tie(stamp.sec, stamp.nanosec) > - std::tie(state.last_sample_stamp->sec, state.last_sample_stamp->nanosec)) - { - state.last_sample_stamp = stamp; - fetched_any = true; - } - }; + continue; + } + const std::string localized_name = + "Avatar " + std::string(to_string(category)) + " " + std::string(to_string(side)); + joint_pushers[joint_category_index(category)] = std::make_unique( + m_handles, core::SchemaPusherConfig{ .collection_id = collection_id(side, category), + .max_flatbuffer_size = kJointFlatbufferSize, + .tensor_identifier = "joint_state", + .localized_name = localized_name }); + } + + m_injectors[index] = std::move(injector); + m_joint_pushers[index] = std::move(joint_pushers); +} + +void AvatarTracker::destroy_output_resources(::avatar::DeviceSide side) noexcept +{ + const size_t index = side_index(side); + m_injectors[index].reset(); + for (auto& pusher : m_joint_pushers[index]) + { + pusher.reset(); + } +} + +void AvatarTracker::refresh_data() +{ + for (const ::avatar::DeviceSide side : kDeviceSides) + { + GloveState& state = glove(side); + if (!state.ready) + { + continue; + } if (m_config.human) { ::avatar::AvatarDataFrame frame; @@ -456,7 +504,6 @@ void AvatarTracker::refresh_data() state.device->fetch_data(frame, ::avatar::DeviceDataCategory::HUMAN) == ::avatar::ErrorCode::SUCCESS; if (fetched) { - note_stamp(frame.skeleton.header.stamp); state.landmarks = std::move(frame.skeleton.landmark); } else @@ -466,7 +513,7 @@ void AvatarTracker::refresh_data() } for (const ::avatar::DeviceDataCategory category : kJointDataCategories) { - if (!dataset_enabled(category) && !(probe_raw && category == ::avatar::DeviceDataCategory::RAW)) + if (!dataset_enabled(category)) { continue; } @@ -475,7 +522,6 @@ void AvatarTracker::refresh_data() ::avatar::AvatarDataFrame& cached = cached_joint_frame(state, category); if (fetched) { - note_stamp(hand_payload(frame, category).header.stamp); cached = std::move(frame); } else @@ -483,22 +529,6 @@ void AvatarTracker::refresh_data() cached = {}; } } - const auto now = std::chrono::steady_clock::now(); - if (fetched_any) - { - if (!state.streaming) - { - state.streaming = true; - isaaccapture::Logger::get("isaaccapture.plugins.sharpa_avatar.AvatarTracker") - ->info("{} glove connected and streaming.", to_string(side)); - } - state.last_successful_fetch = now; - } - else if (!state.last_successful_fetch || now - *state.last_successful_fetch >= kAvatarDataTimeout) - { - logger()->warn("{} glove disconnected: no data for {} s.", to_string(side), kAvatarDataTimeout.count()); - state.reset(); - } } } @@ -515,6 +545,7 @@ GloveState& AvatarTracker::glove(::avatar::DeviceSide side) void AvatarTracker::update() { try_connect_missing_gloves(); + refresh_glove_readiness(); m_deviceio_session->update(); refresh_data(); if (m_haptic_reader) @@ -610,11 +641,12 @@ void AvatarTracker::push_joint_frame(const GloveState& glove, output.has_effort = false; output.ee_pose_valid = false; output.joints.reserve(hand.joint.position.size()); + const auto& names = joint_names(category); for (size_t i = 0; i < hand.joint.position.size(); ++i) { auto joint = std::make_shared(); - joint->name = i < hand.joint.name.size() ? hand.joint.name[i] : std::string{}; + joint->name = i < names.size() ? std::string(names[i]) : std::string{}; joint->position = hand.joint.position[i]; joint->valid = std::isfinite(joint->position); output.joints.push_back(std::move(joint)); @@ -630,7 +662,7 @@ void AvatarTracker::push_joint_frame(const GloveState& glove, void AvatarTracker::apply_haptic_command(::avatar::DeviceSide side, const std::array& powers) { GloveState& state = glove(side); - if (!state.device) + if (!state.ready) { return; } diff --git a/src/plugins/sharpa_avatar/core/inc/avatar/avatar_hand_tracking_plugin.hpp b/src/plugins/sharpa_avatar/core/inc/avatar/avatar_hand_tracking_plugin.hpp index 0fa977ab95..58a2de8016 100644 --- a/src/plugins/sharpa_avatar/core/inc/avatar/avatar_hand_tracking_plugin.hpp +++ b/src/plugins/sharpa_avatar/core/inc/avatar/avatar_hand_tracking_plugin.hpp @@ -14,6 +14,7 @@ #include #include +#include #include #include #include @@ -98,11 +99,9 @@ struct GloveState std::vector<::avatar::Pose> landmarks; ::avatar::AvatarDataFrame raw_frame; ::avatar::AvatarDataFrame robot_frame; - // Empty until start() or a successful fetch; drives the data-timeout window. - std::optional last_successful_fetch; - // The first stamp is a baseline; only a later stamp proves the glove is streaming. - std::optional<::avatar::Stamp> last_sample_stamp; - bool streaming = false; + std::atomic fresh_sample_seen{ false }; + bool started = false; + bool ready = false; }; class __attribute__((visibility("default"))) AvatarTracker @@ -118,6 +117,9 @@ class __attribute__((visibility("default"))) AvatarTracker void initialize_openxr(); void try_connect_missing_gloves(); void start_glove_if_present(GloveState& glove, ::avatar::DeviceSide side); + void refresh_glove_readiness(); + void create_output_resources(::avatar::DeviceSide side); + void destroy_output_resources(::avatar::DeviceSide side) noexcept; void refresh_data(); void inject_hand_data(); void push_joint_frames();