XRT

We present XRT: an extended reality teleoperation framework that returns a robot's touch to its operator

Drive a robot with your body from a Meta Quest 3, see through its stereo cameras, and feel its fingertip contact as finger-localized vibration or open-ear sound. In our study, we ran two XRT stations together so two operators could share one object through two humanoids.

Paper Video Code BibTeX
XRT system

XRT: one operator, one robot

Quest 3 + glove→laptop→robot

The framework. Body tracking drives the robot through closed-form SEW-MIMIC retargeting; stereo video and tactile-skin contact cues come back to the operator. Robot-agnostic; client and server code are on GitHub.

Explore the system → Dual-humanoid study (paper)

Two XRT stations, one shared crate

XRT station 1↔lock-step link↔XRT station 2

Our paper. Two synchronized XRT stations drive two Unitree G1 humanoids; 16 dyads (32 operators) compared vision-only, auditory, vibrotactile and combined contact feedback.

Read the study →
XRT system

Your body is the controller. The Quest 3's inside-out body and hand tracking drives the robot's arms, hands and neck through closed-form SEW-MIMIC retargeting, with no exoskeleton or hand-held controllers.

Operators drive the robots from Quest 3 body tracking (staged demonstration, curtain drawn back)

Signal paths of one XRT station: tracked pose → retargeting → arm, hand and neck commands

The XRT client driving a simulated robot in MuJoCo / robosuite (from the original project page)

XRT system

The robot's touch comes back. One quantizer turns the robot's tactile-skin activation into four contact levels, rendered on the matching finger of a vibrotactile glove, as an open-ear note, or both. No force-feedback hardware, no extra control loop.

Test sweep across the six contact regions: glove pulses and notes share one schedule (turn sound on)

Cues reconstructed from recorded tactile data during a real flip (turn sound on)

What the operator sees: the robot's ZED Mini stereo view, streamed to the headset

Dual-humanoid study

Two XRT stations running in lock-step let two operators share one object through two humanoids, coordinating only through vision, speech and the contact cues.

Two operators, two Unitree G1s, one crate: a full flip (staged demonstration; in the study a curtain separated operators from robots)

Attempt 1, flip over: overhead recording beside the digital twin's tactile contact

Attempt 2, flip back: the same dyad returns the crate upright

Dual-humanoid study

Sixteen dyads lifted and flipped a crate under four conditions, a 2×2 of tactile and auditory cues added to stereo video, in a counterbalanced Williams square.

One dyad, all four conditions (block attempts, overhead view)

V: stereo video only

A+V: open-ear sonification + video

T+V: vibrotactile glove + video

T+A+V: both cues + video

Dual-humanoid study

Every take is logged on a shared clock (median offset 2.1 ms) and replays as a synchronized two-robot digital twin of both robots and the crate.

Shared replay clock, clock offset and end-to-end latency

MuJoCo twin replaying both robots and the crate

Dual-humanoid study

Operators rated our vibrotactile cues more effective and far easier to interpret than our auditory cues, yet we detected no condition difference in task performance, which changed substantially with practice. See the results →

All 32 operators ranked tactile + vision above audio + vision

Practice: one dyad's flip-over in block 1 vs block 4 (12× speed)

Practice: a second dyad, block 1 vs block 4 (12× speed)

The full video

System, study design, recorded feedback, synchronized replay and results in 2 min 40 s. Transcript excerpts use synthetic voices.