Hand labels: fixed cameras vs head cameras.

A robot that learns from human hand labels follows them in the room, so the first question about a label is where it puts the hand. We scored three fixed cameras and published head-camera methods against OakInk2’s marker motion capture, a reference independent of every camera, and then replayed the labels with a robot hand in a physics replay.

Against marker motion capture: OakInk2

OakInk2’s hand reference is a body-and-hand fit to markers tracked by 12 motion-capture cameras, with no image keypoints in the fit, so it is independent of every camera on both sides of the comparison. Its rig is three fixed cameras and a head camera, the 3 + 1 layout we use on site. We scored 355 sequences, 18,141 hand-segments of about 5 s each. None of the methods here was trained on OakInk2.

Two numbers per method matter, and they answer different questions. Placement with no alignment is where the hand ends up in the room, which is what a robot following the labels gets: 30.2 mm for our three cameras, 69.5 mm for WiLoR and 116.7 mm for HaWoR. The benchmark’s W-MPJPE first aligns each clip once on its opening frames, which removes a global offset, most of what is wrong with a head-camera estimate; on that metric ours is 22.1 mm and WiLoR 29.8 mm. A robot replaying the labels gets no such correction, which is why we lead with placement.

Placement comes from geometry, not a trained model: with no learned part at all, the same three cameras are at 3.3 cm. Where our cameras see the hand, the median placement error is 15.3 mm, against 61.3 mm for WiLoR and 76.5 mm for HaWoR on the same frames (138-sequence subset). Our hands also jitter 3 to 6 times less (acceleration error 0.2 m/s² against 0.7 to 1.2).

The head-camera methods win on the rest. Their hand shape is better (PA-MPJPE 7.0 mm for WiLoR against our 9.6 mm), and with three cameras at 848 × 480 we have an estimate on 74% of the reference hand-frames, against 95% for WiLoR, so its success@40 mm is higher (0.79 vs 0.66).

Errors in mm, world frame. “Placement, no alignment” is the mean joint error with no fit of any kind; W-MPJPE, median and PA-MPJPE are the benchmark metrics. The head-camera rows use the motion-capture camera pose. The lifter row is our own head-camera-only baseline, trained on HO-Cap. An earlier run on 25 sequences gave 31.7 mm for ours and 71.9 mm for WiLoR.

The two together

Our three cameras lose the hand on about a quarter of the frames; the head camera sees it on 95%. Combining our placement in the room with the head camera’s hand shape, a test we fixed before running it, covers 97% of the reference hand-frames at 40.5 mm with no alignment, against 68.5 mm for WiLoR, and has the best success@40 mm of any label source (0.83). In the physics replay below it has the highest task success short of the motion-capture hands (0.28) and the highest grasp rate (0.75, against 0.61 for our cameras alone and 0.62 for WiLoR).

Label error in a physics replay

We replayed 694 manipulation episodes from OakInk2 (247 pick-and-places, 150 slides, 79 reorientations, 77 pours, 75 two-handed tasks, 36 lift-and-returns, 30 handovers) in a physics replay, once per label source: 1,431 replays of the 694 episodes for each source, in the same scenes for every source. Each source’s world-frame hand trajectory is retargeted by the same retargeter to a floating Inspire robot hand in MuJoCo, with the real object mesh at its reference start pose and the same controller and grasp schedule for every source. Task success means the hand acquires the grasp and the object ends within 50 mm of where the person left it. Ranges are 95% bootstrap confidence intervals over episodes, and differences are paired episode by episode.

Task success follows the labels: 0.25 from ours, against 0.19 for WiLoR (ours − WiLoR: +0.06, 95% CI [+0.02, +0.10]) and 0.14 for HaWoR (ours − HaWoR: +0.10, 95% CI [+0.07, +0.13]); the motion-capture hands reach 0.41. The lead is largest where orientation matters (pouring 0.37 vs 0.23, reorienting 0.34 vs 0.20) and reversed on handovers (0.04 vs 0.10), where our cameras lose the hand.

Giving the head-camera methods SLAM in place of the true camera pose makes no measurable difference in the replay: on the 326 episodes where we ran it, WiLoR with SLAM minus WiLoR with the true pose is 0.00 [−0.03, +0.04], the same for HaWoR (0.00 [−0.03, +0.03]), and ours minus WiLoR with SLAM is +0.12 [+0.07, +0.17].

The drift line holds for any label source: task success falls from 0.41 to 0.25 at 2.5 cm of drift and to 0.11 at 5 cm. The head-camera labels sit above that line because most of their error is a constant offset along the camera’s line of sight, and once the replay holds the object, a hand that is several centimetres off but still makes contact moves the object by the right amount. Drift is what the replay punishes.

Object-anchored check. If every trajectory is shifted so the hand meets the object where the reference hand did at grasp onset, removing most of the placement error, task success is 0.28 for ours, 0.27 for WiLoR and 0.26 for HaWoR. The sources come out level (ours − WiLoR +0.01, 95% CI [−0.03, +0.04]), so the replay gap is placement.

Hand error: world-frame joint error over the replayed episode frames, without the benchmark’s alignment. The lifter row is our own head-camera-only baseline; the MediaPipe row is smoothed.

The same test on HO-Cap

HO-Cap’s reference is not independent of our cameras. It is a multi-view fit from the fixed cameras around the table, and our four cameras are four of the eight that built it, so the numbers in this section measure agreement with a reference built partly from the same images. The head-camera methods get no such help. With that said: on HO-Cap’s test people, whom our label models never saw, our four fixed cameras are on average 1.8 cm from the reference (18.0 mm W-MPJPE, median 14.9 mm), and the best published head-camera method is 7.5 cm off.

How our runs of the baselines differ from their papers: HaWoR’s image backbone runs in half precision (the same to within a fraction of a millimetre); its SLAM stage uses DPVO in place of DROID-SLAM, with HaWoR’s own metric-depth model for scale, and the “given head poses” rows skip SLAM entirely; WiLoR runs per frame with its own detector and the camera’s true focal length. Our HaWoR run lands in its published range on its own benchmark, so these rows are not a broken port. Under HaWoR’s own, more lenient evaluation (right hand only, inside each method’s detection runs) the numbers are 12.0 mm for ours and 52.1 mm for the best head-camera method.

150-frame segments

Errors in mm. Means are over hand-segments; a few bad segments pull HaWoR’s mean up, so read it with the median. The two lifter rows are our own head-camera-only baselines, trained on HO-Cap’s other people.

100-frame segments

What the error looks like

From the head camera, a hand that is several centimetres too near or too far still lines up with the image. A second camera shows the difference.

Ours 14.7 mm · WiLoR 52.2 mm

Ours 13.8 mm · WiLoR 52.5 mm

HO-Cap referenceours, four fixed camerasWiLoR + given head poses

In the same physics replay on 107 HO-Cap test episodes, task success is 0.35 for ours and 0.41 for the reference hands, against 0.14 for WiLoR (ours − WiLoR: +0.22, 95% CI [+0.13, +0.30]) and 0.04 for HaWoR (ours − HaWoR: +0.31, 95% CI [+0.23, +0.39]). The shared cameras flatter our row here too.

What this does not show

• Coverage during the grasp. On average our cameras see the hand on 69% of a replayed episode’s frames (WiLoR 93%), and in 5% of replays they never see it at all; when the hand is missing during the grasp, the replay fails regardless of accuracy.
• The reference is not contact-accurate. OakInk2’s motion-capture fingertips sit 10–27 mm from the object surface at grasp onset, so the reference ceiling of 0.41 and finger-level comparisons carry that uncertainty.
• OakInk2’s images come at 7.5 Hz, a quarter of the usual video rate, so HaWoR’s motion model runs at a quarter of its usual rate, which is part of its result. Images are 848 × 480, and OakInk2’s own camera-to-motion-capture calibration is off by 10–16 px in its left camera in 8 of the 25 sequences we checked; we use it as given.
• The head-camera methods got the true camera pose from the motion-capture system, the best case for them.
• The replay holds the object once the hand touches it. Without that hold, nothing works for any source: in pure physics the motion-capture hands reach a task success of 0.00. One retargeter, one floating robot hand, no arm, and no real robot.
• Ours is a label source, not a head-camera model. It needs the fixed cameras at capture time.

Prior work

• Rim et al., “Ego-Exo 3D Hand Tracking in the Wild with a Mobile Multi-Camera Rig”, 2025 (arXiv 2510.02601) measure an eight-camera markerless hand rig against a 30-camera dome and report median joint errors of 6.3 to 7.9 mm. They do not score head-camera methods against the same reference in the world frame.
• EgoAllo, Yi et al., CVPR 2025 (arXiv 2410.03665) scores head-camera hands in the world frame against Ego-Exo4D’s labels from outside cameras.
• HaWoR, Zhang, Deng, Ma and Potamias, CVPR 2025 (arXiv 2501.02973) reports world-frame hand error for head-camera video; we follow its evaluation code.
• Ego-Exo4D, Grauman et al., CVPR 2024 (arXiv 2311.18259) scores head-camera methods against labels triangulated from outside cameras, in the camera frame.

Credits and licences

• OakInk2: Zhan et al., “OakInk2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion”, CVPR 2024 (arXiv 2403.19417). CC BY-SA 4.0.
• HO-Cap: Wang et al., “HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction,” NeurIPS 2025 Datasets and Benchmarks Track (paper, project page). CC BY 4.0. The images on this page are HO-Cap frames, cropped, with our drawings.
• HaWoR: Zhang, Deng, Ma and Potamias, “HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos”, CVPR 2025 (arXiv 2501.02973). Code and weights used for evaluation only, under their licence.
• WiLoR: Potamias et al., “WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild”, CVPR 2025 (arXiv 2409.12259). Evaluation only, under its licence.
• MANO: Romero, Tzionas and Black, “Embodied Hands: Modeling and Capturing Hands and Bodies Together”, SIGGRAPH Asia 2017. Used by the baselines under its research licence.
• DPVO: Teed, Lipson and Deng, “Deep Patch Visual Odometry”, NeurIPS 2023 (arXiv 2208.04726). MediaPipe Hands: Zhang et al., “MediaPipe Hands: On-device Real-time Hand Tracking” (arXiv 2006.10214), Apache-2.0. Physics: MuJoCo; robot hand: Inspire.

Benchmark split, metrics, replay and baseline runs by Autelica, 2026. Numbers on this page are generated from the evaluation files and checked against them at build time.