What hand-label error does to robot policies.

Part 1 scored three fixed cameras and published head-camera methods against OakInk2’s marker motion capture: our cameras place hands 3.0 cm from where they are in the room with no alignment, and head-camera methods given the true camera pose 7.0 to 11.7 cm off. Replayed with a robot hand in a physics replay of 694 episodes, our labels reach a task success of 0.25 against 0.19 from WiLoR’s. This part trains policies on those labels.

What this does to robot policies

A replay follows the labels exactly; a policy learns from them and might average some of the error away. For each label source we built demos by replaying its labels with an Inspire hand on a Franka arm, trained policies on them by imitation and by tracking the labelled motion with a correction learned by reinforcement learning, and tested every policy from the same held-out start poses. Success means the same as in the physics replay in Part 1. OakInk2, scored against marker motion capture, is the independent check and comes first, with three tasks; HO-Cap follows and is partly circular.

OakInk2: pick-and-place, against motion capture

On 123 OakInk2 pick-and-place episodes, imitation policies trained on our labels succeed in 0.21 of held-out tests, against 0.03 to 0.12 for head-camera labels (MediaPipe to HaWoR; ours − WiLoR +0.13, 95% CI [+0.08, +0.18]); the motion-capture labels reach 0.29 and the combination of our placement with the head camera’s hand shape 0.23. With tracking and reinforcement learning, ours reaches 0.41 against 0.24 to 0.28 for head-camera labels and 0.46 for motion capture.

OakInk2: sliding an object

On 150 OakInk2 slide episodes, imitation policies trained on our labels succeed in 0.31 of held-out tests, against 0.12 to 0.20 for head-camera labels (MediaPipe to our head-camera lifter); ours − WiLoR is +0.15, 95% CI [+0.09, +0.20]. Our labels are level with the motion-capture labels (0.33; ours − motion capture −0.02, 95% CI [−0.07, +0.02]). With tracking and reinforcement learning, ours reaches 0.42 against 0.24 to 0.33 for head-camera labels and 0.45 for motion capture.

OakInk2: lift-and-return

On 36 OakInk2 lift-and-return episodes there is no clear difference between our labels and WiLoR’s: imitation policies succeed in 0.20 of held-out tests with ours and 0.18 with WiLoR’s (ours − WiLoR +0.02, 95% CI [−0.08, +0.12]). Ours sits well below the motion-capture labels (0.36; ours − motion capture −0.16, 95% CI [−0.27, −0.07]), and so does every head-camera source. With tracking and reinforcement learning the picture is the same: 0.32 for ours, 0.29 for WiLoR’s and 0.50 for motion capture. With so few episodes the intervals are wide.

HO-Cap: reorienting an object

On 41 reorientation episodes, imitation policies trained on our labels succeed in 0.29 of held-out tests, level with the reference labels (0.27), against 0.03 to 0.10 for head-camera labels (MediaPipe to our head-camera lifter); ours leads each head-camera source by 0.19 to 0.26, every interval above zero.

HO-Cap: lift-and-return

On 14 lift-and-return episodes, our labels come in below the reference (0.44 against 0.57; −0.13, 95% CI [−0.31, +0.01], an interval that includes zero) and lead WiLoR’s by +0.38 [+0.18, +0.58].

HO-Cap: pouring

Pouring (15 episodes) is too hard for imitation to separate the sources, with the reference labels at only 0.14; with reinforcement learning, ours reaches 0.53 against 0.20 for WiLoR (+0.33, 95% CI [+0.07, +0.60]).

With fewer demos

On OakInk2 pick-and-place, our labels lead WiLoR’s and HaWoR’s at every share of the demos: 0.08 against at most 0.03 with a quarter, 0.21 against at most 0.12 with all of them. With a quarter of the HO-Cap reorientation demos our labels reach 0.19, and no head-camera source gets above 0.10 with all of them. On the lift-and-return tasks and on pouring, where every source is noisier, the curves sit closer together.

More reinforcement learning

On OakInk2 pick-and-place, with the same compute for every source, success rises from 0.34 at 24 rollouts per episode to 0.47 at 384 for our labels, from 0.17 to 0.29 for WiLoR’s and from 0.21 to 0.31 for HaWoR’s; the motion-capture labels reach 0.51. The gap does not close with more compute: ours minus WiLoR goes from +0.17 to +0.18, and ours minus HaWoR from +0.13 to +0.16.

On HO-Cap reorientation, longer reinforcement learning levels off at different heights: at 384 rollouts per episode, 0.61 for the reference labels, 0.53 for ours, 0.30 for WiLoR’s and 0.16 for HaWoR’s (ours [0.39, 0.67] against WiLoR’s [0.19, 0.43]).

Synthetic demos from a few real ones

We expanded a tenth of the reorientation demos a hundredfold into synthetic demos, re-posed or with varied objects, table and clutter. With our labels these reach 0.11 and 0.09, against 0.29 from all the real demos: they succeed on the seed episodes (0.55) and not on the rest (0.06). Nor do they lift our labels above WiLoR’s full set of real demos (+0.03 [−0.03, +0.09] re-posed, +0.01 [−0.04, +0.07] with varied scenes).

Generation cost did separate our labels from the head-camera ones. A synthetic demo is kept only if it succeeds in simulation, and keeping 1,000 from a quarter of the demos took 2,160 attempts with our labels, against 7,008 with WiLoR’s and 9,504 with HaWoR’s, 3 to 4 times fewer (3,800, 11,720 and 16,360 with varied scenes).

Credits and licences

• OakInk2: Zhan et al., “OakInk2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion”, CVPR 2024 (arXiv 2403.19417). CC BY-SA 4.0.
• HO-Cap: Wang et al., “HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction,” NeurIPS 2025 Datasets and Benchmarks Track (paper, project page). CC BY 4.0.
• HaWoR: Zhang, Deng, Ma and Potamias, “HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos”, CVPR 2025 (arXiv 2501.02973). Code and weights used for evaluation only, under their licence.
• WiLoR: Potamias et al., “WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild”, CVPR 2025 (arXiv 2409.12259). Evaluation only, under its licence.
• MANO: Romero, Tzionas and Black, “Embodied Hands: Modeling and Capturing Hands and Bodies Together”, SIGGRAPH Asia 2017. Used by the baselines under its research licence.
• DPVO: Teed, Lipson and Deng, “Deep Patch Visual Odometry”, NeurIPS 2023 (arXiv 2208.04726). MediaPipe Hands: Zhang et al., “MediaPipe Hands: On-device Real-time Hand Tracking” (arXiv 2006.10214), Apache-2.0. Physics: MuJoCo; robot hand: Inspire.

Benchmark split, metrics, replay and baseline runs by Autelica, 2026. Numbers on this page are generated from the evaluation files and checked against them at build time.