JwumaRobotics and embodied AI data, where LiDAR annotation goes wrong
· 2 min read · specification, robotics
Robotics annotation covers point clouds, depth and motion across sequences. Cross-frame consistency matters more than per-frame accuracy, because an object labelled slightly differently frame to frame teaches a model that the world flickers. Specify consistency requirements explicitly, because per-frame QA will not catch this.
The short version
- Cross-frame consistency is the dominant quality dimension.
- Per-frame accuracy checks do not detect cross-frame drift.
- Occlusion handling conventions must be fixed before annotation begins.
- Sensor calibration errors are easily mistaken for annotation errors.
- Sequence-level review is required, not just frame sampling.
Scoping something like this?
We would rather talk about the problem than quote against a volume figure.
See capabilitiesWhy does cross-frame consistency dominate?
Because the model is learning that objects persist. If the same vehicle is boxed slightly differently across consecutive frames, the training signal says the vehicle changed shape, and tracking behaviour degrades in ways that per-frame metrics do not show.
Each individual frame can be within tolerance while the sequence as a whole is unusable. This is the characteristic failure of robotics datasets annotated with image-annotation habits.
How should QA differ from image annotation?
Review sequences, not sampled frames. A random frame sample is nearly blind to drift, because drift is a property of the relationship between frames rather than of any one of them.
Ask specifically how your supplier reviews sequences and what their consistency metric is. If the answer is per-frame accuracy, they are measuring the wrong thing.
- Sequence-level review, not frame sampling
- An explicit cross-frame consistency metric
- Occlusion entry and exit conventions
- Handling of objects leaving and re-entering the scene
What conventions need fixing in advance?
Occlusion, primarily. What happens when an object is partially hidden, fully hidden for several frames, then reappears. Whether the box is interpolated through the occlusion or dropped and a new track started.
Both answers are defensible. An unspecified answer means different annotators choose differently within one dataset, which is worse than either choice.
How do you separate sensor errors from annotation errors?
By checking calibration before blaming annotators. Miscalibration between sensors produces apparent annotation errors that are actually correct annotations of misaligned data.
A supplier who cannot distinguish the two will keep re-annotating data that was never the problem, and bill you for it.
People also ask
What matters most in LiDAR annotation quality?
Cross-frame consistency. An object labelled slightly differently between consecutive frames teaches the model that objects change shape, and per-frame accuracy metrics do not detect it.
How should robotics annotation be quality-checked?
By reviewing whole sequences rather than sampling individual frames, with an explicit cross-frame consistency metric. Frame sampling is nearly blind to drift.
How do I tell sensor calibration errors from annotation errors?
Check calibration first. Miscalibration produces what look like annotation errors but are correct annotations of misaligned data, and re-annotating will not fix it.