3D hand capture from a few cameras: 5.4 mm on people we never trained on.

01What we measured

HO-Cap records people picking up and using everyday objects in a lab, filmed by 8 fixed colour cameras at 640×480 and 30 fps, plus a head-mounted camera. It ships 3D hand joints for every frame and is released under CC BY 4.0 (Wang et al., “HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction,” NeurIPS 2025 Datasets and Benchmarks Track (paper)). We grouped its 64 sequences by person, which gives 8 people, and ran leave-one-person-out: for each person, every learned part of our pipeline was trained on the other seven and then scored on that person alone. The reference labels are used only for training and scoring; at run time the pipeline sees only the camera images and calibration.

The score is the distance between our 3D joints and HO-Cap’s, over all 21 joints of each hand. Errors are absolute positions in the camera rig’s world frame, with no alignment to the reference, so wrist position counts too. Many published hand-pose numbers are reported after aligning each hand to the ground truth, which removes that part of the error, so compare with care. We report the median per person, then the median over people. We ran everything twice: with all 8 cameras, and with 4 of them. The 4 were picked per sequence as the cameras where the detector found hands most often, without looking at the reference labels.

02Results

Each step helps at both camera counts. The refiner closes most of the gap between them: 1.0 mm before it, 0.2 mm after. With it, 4 cameras (5.6 mm) beat 8 cameras without it (6.2 mm). Fingertips, which matter most for grasping, go from 11.6 mm to 7.0 mm with 8 cameras, and reach 7.0 mm with 4. The full pipeline’s per-person results:

The p90 column is the error that 90% of joints stay under. The 2D column is keypoint error inside 128-pixel hand crops, before and after our refiner; it falls by about 40% at the median, and by 29–47% per person. Person F is the weakest, at 6.5 mm with 8 cameras and 8.0 mm with 4.

03Skin keypoints and joint centres

Off-the-shelf hand detectors such as MediaPipe are trained on images labelled by people, who click on the visible surface of the hand. A robot hand model, a physics simulator and a dataset like HO-Cap all describe the hand by its joint centres, which sit inside the finger. At the knuckles the two differ by 15–21 mm on HO-Cap. Triangulating skin points perfectly still leaves that offset, which is why our first versions stalled at about 11 mm no matter how many cameras we added.

The first fix works in 3D: a learned correction moves each triangulated point toward its joint centre. Trained on one person it carried over less well to others; trained on seven it brought held-out people from 9.9 to 6.2 mm with 8 cameras.

04Why several cameras

From one camera, how far away a finger is has to be guessed. With two or more calibrated cameras each joint is triangulated, so depth comes from geometry. More views also mean that when an object hides a finger from one camera, another usually sees it, and one bad detection does not have to drag the whole hand.

05Tracking over time

Detectors miss small or partly hidden hands in a full frame. The pipeline uses all the cameras together to recover hands a single view missed, which lowered error by about 1 mm, follows each hand over time so left and right stay consistent without any ground truth, and removes most of the frame-to-frame jitter.

06Our 2D keypoint refiner

The 3D correction can only adjust what the cameras reported. The refiner is our own learned model that fixes the 2D keypoints in each camera before triangulation, moving each point from where the off-the-shelf detector put it toward the joint centre. Like every learned part of the pipeline, it was trained only on the other seven people each time.

07Checking head-camera hand labels

Hands leave the head camera’s frame, overlap and blur when the head turns, so automatic hand labels in first-person video drift. Once the outside cameras have a 3D hand, we can check the head camera’s labels against it and flag the frames where they disagree. With 4 outside cameras and the headset’s tracked head pose, the flags catch about two thirds of grossly wrong head-camera labels at about 70% precision, roughly twice a single-camera check. Details, and what the same 3D labels do for training head-camera models, are in Training head-camera hand models with 3D labels from outside cameras.

08Labels without annotators

The refiner learned from HO-Cap’s labels, which are themselves built from many views. Our own captures can produce training labels the same way, for every camera including the head camera, with no one clicking points. Training on those labels should let later captures in similar settings get by with fewer cameras. We have not yet measured this beyond HO-Cap.

09Limits

• HO-Cap’s cameras are 640×480 lab cameras at fixed positions. Phones and head cameras in a real workplace will behave differently.
• The reference labels are not exact. They are fitted from the dataset’s own camera rig rather than markers, so part of the remaining 5 mm may be theirs.
• Fingers hidden behind an object are the main failure. The refiner helps less there, as the bottle grasp above shows.
• Eight people is a small group. Results varied from 4.3 to 6.5 mm across them with 8 cameras.
• The pipeline’s learned stages were trained one after another on the same seven people, so the later stage saw slightly cleaner inputs in training than it would on new data. The held-out person was never seen.
• Our own captures, a head camera plus phones, are the next test.

10What a capture delivers

• Per frame and per hand: 21 joint centres in millimetres in one world frame, left/right labels; 3D body alongside.
• Per clip: synced video from every camera, camera poses, and an error figure from cameras held out of the reconstruction.
• For first-person data: the frames where head-camera hand labels drift from the outside-camera 3D.
• On request: the motion retargeted to a robot body or robot hand.

Marker-based optical mocap is more precise but needs markers on the hand, which get hidden in grasps; this needs only ordinary cameras around the work area.