Handheld phone video is mostly the camera rotating, so the movement between neighbouring frames, which a global structure-from-motion solver needs to place each camera, is tiny and noisy. With its default settings, the solver treated most of that weak evidence as outliers, so its random starting point decided the path: about 3 runs in 10 placed a stretch of frames at the wrong scale. Changing how strictly it rejects outliers gave 10 of 10 good solves on the same clip (5 of 5 on a drone clip) at the same speed, and our replays went from 6 to 10 minutes to about 3.
01What we run
Autelica turns phone videos of an athletic moment into measured 3D motion and a textured replay. With one moving camera, the replay needs the camera’s path, which we solve from the background with structure from motion, placing every camera at once with a global solver. A check then keeps the longest stretch of the path that is smooth in position and rotation; we report the share of frames on it as “consistent”.
02The symptom
On the founder’s own 14-second handheld clip, identical runs gave 0.94, 0.32, 0.34 and 0.65 of frames on a consistent path. When too little of the path was consistent, the pipeline fell back to a denser, slower solve, so the 3D replay took 6 to 10 minutes instead of 3. In production that day, 6 of 9 first attempts failed.
03What it wasn’t
| Suspect | Test | Result |
|---|---|---|
| The mapper | An incremental solver instead of the global one, 3 runs | Still 1 in 3 broken, 2 to 3× slower |
| The lens guess | Focal length × 0.8, 1.0, 1.25, 3 runs each | All three fail about 1 in 3 |
| Too few features | Matches between neighbouring frames | Plenty for every pair across the clip |
| The seed | Fixed random seed | Seed 0 gave 0.97 and then 0.34: the variation is from multithreading |
Every raw solve placed every frame. The difference was the shape of the path: in the broken runs, a stretch of frames sat at about 5× the step size of its neighbours, so the consistency check kept only the largest smooth piece.
04The cause
Most neighbouring frame pairs in this clip were dominated by rotation: the camera mostly turned. Rotation alone says nothing about where the camera moved, so the translation directions between neighbours, which a global solver relies on, are small and noisy. The solver starts from random camera positions and down-weights evidence it considers outliers. With its default setting, much of this weak-but-real evidence was treated as outliers, so it barely pulled on the solution, and the random start decided where a weakly-constrained stretch ended up.
05The fix
Making the solver more tolerant of that weak evidence fixed it: 0 of 10 broken runs on the handheld clip, against 3 of 10 before, and a consistently good path on a drone clip too. Mapping time didn’t change. Discarding low-parallax evidence, the other obvious idea, made it worse: 3 of 3 broken. In production the next three runs on the same clip solved first time, and the 3D replay was ready 2.8 to 3.4 minutes after upload.
06Takeaways
- Run it more than once. A multithreaded solver that is fine most of the time hides a failure rate you only see by repeating identical runs.
- Check how much the camera translates. If most of your frame pairs are rotation-dominated, translation is weakly observed, and a global solver needs help.
- Outlier handling is a modelling choice. Defaults suit typical photo collections; rotation-dominated video can need different ones.
Questions
Why does structure from motion fail on handheld video?
Handheld video is dominated by rotation, so the translation between neighbouring frames is small and poorly observed; global solvers that place all cameras at once from those translations can converge to a wrong path.
How did we fix camera solves on rotation-dominated video?
By making the global solver more tolerant of weak translation evidence instead of treating it as outliers: 10 of 10 good solves on a handheld clip and 5 of 5 on a drone clip, at the same speed.
Does a fixed random seed make a multithreaded SfM solver deterministic?
Not in our runs: with multithreading, seed 0 produced both a good and a broken path on the same input.