Tactile Schema: New Feature and Feedbacks Collection

[lotusl-code]

Summary

We are adding first-class support for tactile sensors, such as taxel arrays on gloves and robot hands, as an upcoming feature. The design adds two FlatBuffers tables:

• TactileData: per-frame force readings, one entry per taxel.
• TactileShape: the device's static layout, meaning where each taxel sits on a hand skeleton and how it moves with the hand.

A generated TactileTracker ships both, and they work with live sessions, MCAP recording and replay, and the retargeting graph.

Status: in development, not yet released. Implementation: #1242.

This thread has three purposes:

1. Announce the feature before it is released.
2. Collect feedback on any part of it.
3. Track schema updates, the release, and integrations as they land.

Why

Tactile devices differ widely in taxel count, placement and mounting. A consumer needs two things to use a reading: the forces, and where each force was measured on the bodypart (like hand). Consumers include retargeting, visualization, recording and policy learning. The design separates the two because they change at very different rates.

Design

Data flow

device plugin -> "<collection_id>"          TactileData, every frame
              -> "<collection_id>/metadata" TactileShape, republished periodically
                   -> TactileTracker -> get_data() / get_metadata()
                                     -> MCAP recording and replay
                                     -> TactileSource (retargeting graph)

TactileShape: static layout

struct TactileJoint {
  bind_pose: Pose;    // hand-root space; meters, unit quaternion
  parent: uint32;     // parent joint ordinal; 0x7FFFFFFF (XR_HAND_JOINT_MAX_ENUM_EXT) marks the root
}

struct TactileTaxel {
  position: Point;              // bind position, hand-root space, meters
  normal: Point;                // unit normal
  tangent: Point;               // unit tangent, perpendicular to the normal
  joint_indices: [uint32:4];    // skinning joints (XrHandJointEXT ordinals)
  joint_weights: [float:4];     // skinning weights
}

table TactileShape {
  joints: [TactileJoint] (id: 0);  // indexed by XrHandJointEXT ordinal
  taxels: [TactileTaxel] (id: 1);  
}

Conventions

• The skeleton follows OpenXR hand joints: joints[i] is XrHandJointEXT ordinal i, and the hierarchy is acyclic.

• Taxels follow the hand by Linear Blend Skinning over four slots:

  \mathbf{p}^{\text{current}} = \sum_{k=0}^{3} w_k \, \mathbf{T}^{\text{current}}_{j_k} \left(\mathbf{T}^{\text{bind}}_{j_k}\right)^{-1} \mathbf{p}^{\text{bind}}
  

  where $j_k$ is joint_indices[k], $w_k$ is joint_weights[k], $\mathbf{p}^{\text{bind}}$ is the taxel's position, $\mathbf{T}^{\text{bind}}{j}$ is joint $j$'s bind_pose, and $\mathbf{T}^{\text{current}}{j}$ is joint $j$'s current joint-to-hand-root transform. Both transforms are rigid; the sum blends transformed points, not poses.

  • Weights are finite and nonnegative, with 1 to 4 of them nonzero, and they sum to 1.
  • Every slot names a valid joint, including zero-weight slots.

• Size: 32 bytes per joint plus 68 bytes per taxel. A 26-joint hand with 200 taxels is about 14 KB. There's no size limit for TactileShape.

TactileData: per frame

table TactileData {
  // Force vector per taxel, in TactileShape taxel order.
  taxel_force: [Point] (id: 0); // in TactileShape.taxels order. in Newton unit.
}

• Each taxel takes 12 bytes. The default sample size is 16 KiB, about 1,300 taxels. Larger devices pass max_flatbuffer_size to the tracker.
• Each frame carries the standard DeviceDataTimestamp in its MCAP record.

Why the layout is tracker metadata, not a second stream

TactileShape is fixed for a device, so it is published as the tracker's metadata rather than as timestamped samples.

• The tracker adopts the first shape it receives and records it once. Frames are withheld until a verified shape is available, and also if the producer later publishes a different shape. A frame is therefore never interpreted against the wrong layout.
• On replay, the shape is available as soon as the session opens, before the first frame. Data recorded without its shape is withheld.

Using it

• Python schema objects (TactileData, TactileShape) are built from NumPy arrays, which are validated for lengths and joint indices. Every field reads back as a zero-copy NumPy view.

Producers

• Push TactileData on the data collection every frame.
• Push the encoded TactileShape on "<collection_id>/metadata" periodically:
  • in C++, with SchemaPusher and SchemaMetadataPusher;
  • in Python, with TensorPushTracker.

shape = TactileShape(...).to_bytes()  # the device's layout
data_tracker = TensorPushTracker(COLLECTION_ID, "tactile", 16384)
shape_tracker = TensorPushTracker(
    tracker_metadata_collection_id(COLLECTION_ID), TRACKER_METADATA_TENSOR_IDENTIFIER, len(shape)
)

# In the DeviceIOSession loop:
shape_tracker.push(session, shape)  # periodically, e.g. once per second
data_tracker.push(session, TactileData(forces).to_bytes())  # every frame; forces: (N, 3) float32

Consumers

Major approach of reading tactile data is device plugins, which is a separate process that the session starts, reads the device and publishes TactileData and TactileShape. The application declares the plugin and reads the data through TactileSource; both sides use the same collection id.

tactile = TactileSource("tactile", COLLECTION_ID, num_joints=26, num_taxels=200)
config = TeleopSessionConfig(
    app_name="<app>",
    pipeline=tactile,
    plugins=[PluginConfig(plugin_name="<tactile_plugin>", plugin_root_id="<root_id>", search_paths=[Path("<plugin_dir>")])],
)

with TeleopSession(config) as session:
    result = session.step()
    forces = result[TactileSource.FORCES]  # (N, 3) taxel forces; none while the device is inactive
    shape = result[TactileSource.SHAPE]    # TactileShape arrays; none until the shape arrives

• Retargeting graphs use TactileSource. It outputs the forces and the shape as fixed-size arrays.

Example: examples/tactile has a mock feeder and a printer that run end to end without hardware, including MCAP recording and replay.

Feedback we are looking for

Once released, these schemas follow the repository's schema evolution rules: tables can gain new optional fields, but structs can never change, because recordings embed their schema. Feedbacks are therefore most valuable before release.

We are open to any feedback on any part of the design. Some directions for example:

1. Force semantics. The schema use Newton as the unit for taxel_force. Would it be better unified?
2. Other per-taxel quantities. Do you need contact flags, pressure, temperature, vibration or contact area alongside force?
3. Skeleton scope. The layout is tied to OpenXR hand joints with convensions. Do you have better options? Or remove this restriction and make it fully general?
4. Skinning. Is four-influence linear blend skinning sufficient for your devices?
5. Static layout. Does any device change its layout or calibration while running?
6. Identification. Left and right devices are told apart only by collection id. Is there better way to do this?
7. Timing and rate. There is one timestamp per frame. Do your devices sample taxels asynchronously, or at rates high enough that batching several frames per sample matters?
8. Scale. How many taxels do your devices have, and at what frame rate?

When you reply, it helps to include your device or use case: taxel count, rate, measured quantities, and where the taxels are mounted.

What's next

We will update this thread with:

• Schema revisions that come out of this discussion.
• The release that ships the feature: late October 2026.
• Integrations: device plugins, visualization, and retargeting and learning pipelines that consume tactile data.