# Autelica 3D hands sample

Five desk tasks by one person, each recorded by a head-mounted camera and three fixed cameras around the desk.
Each clip has both hands as 21 joints in a world frame, the head camera's pose on every frame, and a measured error.

## Files per clip

| file | contents |
|---|---|
| `hands_<clip>.mp4` | H.264, 1920 x 1080, 30 fps. Head camera with the 21-joint hands drawn on it (top left), one fixed camera with the same hands (bottom left), a 3D view of both hands and the head camera's pose and path in the world frame (right), and the clip's numbers. |
| `hands_<clip>.json` | Everything below as JSON (null where a value is missing). |
| `hands_<clip>.npz` | The same as NumPy arrays (NaN where a value is missing). |

## Joints

21 per hand, in this order (the common hand-keypoint order: the wrist, then 4 per finger from the base to the tip):

| index | joint | index | joint | index | joint |
|---|---|---|---|---|---|
| 0 | wrist | 7 | index_dip | 14 | ring_pip |
| 1 | thumb_cmc | 8 | index_tip | 15 | ring_dip |
| 2 | thumb_mcp | 9 | middle_mcp | 16 | ring_tip |
| 3 | thumb_ip | 10 | middle_pip | 17 | pinky_mcp |
| 4 | thumb_tip | 11 | middle_dip | 18 | pinky_pip |
| 5 | index_mcp | 12 | middle_tip | 19 | pinky_dip |
| 6 | index_pip | 13 | ring_mcp | 20 | pinky_tip |

Bones drawn in the video: `bones` in the JSON / NPZ. Left and right are the person's own hands. In the video the left hand is
amber and the right hand is blue.

## Coordinate frame and units

- World frame, one per clip: origin at the centre of a fiducial tag lying on the desk, x and y in the desk plane, z up,
  right-handed. All five clips share the same world frame.
- Units: millimetres for every position, including camera translations; seconds for time.
- Cameras: `world_from_camera_mm` is a 4 x 4 matrix that maps camera coordinates to world coordinates. Camera axes follow
  OpenCV: x right, y down, z forward.
- Lenses: OpenCV fisheye model (`cv2.fisheye`, k1..k4) at 1920 x 1080. `K` and `distortion` are given for the head camera and
  for each fixed camera (A, B, C; B is the one in the video).

## Per frame

30 fps. Frame `i` of every array is frame `i` of the MP4. `timestamps_s` starts at 0.

| field (JSON) | NPZ | shape | meaning |
|---|---|---|---|
| `hands.<side>.joints_mm` | `hands_world_mm` | (2, T, 21, 3) | 3D joint positions, world frame, mm. NPZ hand 0 = left, 1 = right. |
| `hands.<side>.confidence` | `confidence` | (2, T) | quality tier per hand and frame (below). |
| `hands.<side>.visible` | `visible` | (2, T) | 1 = the hand was measured this frame (0 = no hand, or a short gap filled by interpolation). |
| `hands.<side>.joints_in_head_view` | `joints_in_head_view` | (2, T[, 21]) | joints that fall inside the head camera's image (JSON: count, NPZ: per joint). |
| `head_camera.world_from_camera_mm` | `head_world_from_camera_mm` | (T, 4, 4) | head camera pose. |
| `head_camera.pose_valid` | `head_pose_valid` | (T,) | 1 = pose available. |
| `timestamps_s` | `timestamp_s` | (T,) | time from the first frame. |

Confidence tiers: 1.0 = tracked across several cameras and passes every plausibility check; 0.7 = measured from fewer views
and passes every check; 0.3 = measured but fails a plausibility check (bone length, spike, joint angle or cross-view
agreement); 0.1 = short interpolated gap, not measured; 0 = no hand. The tiers are categories, not probabilities. For the
strictest set use `confidence == 1`.

## Error report

`error_report.heldout_camera` in each JSON: each fixed camera is left out in turn. The hands measured from the other cameras
are projected into the left-out camera and compared with the hands found in that camera's own image, as a distance at the
hand (mm). The number includes the left-out camera's own 2D error, so it is an upper bound on the error of the 3D hands, which
use every camera. `median_mm` is the mean of the three per-camera medians. The video shows the same number.

## Capture

One person at a desk, four consumer action cameras (one on the head, three on stands about 0.6 m from the desk centre),
recorded at 60 fps and resampled to 30 fps on one timeline. No markers on the hands and no shared clock between the cameras.
People other than the person at the desk were erased from the video (filled in, not blurred), and the fixed-camera view is
cropped to the desk.

## Licence

Evaluation licence, see LICENSE.txt. Questions and full datasets: hello@autelica.com