Corvus ISR, a provider of wide-area motion imagery (WAMI) exploitation solutions, has published a detailed public tracker benchmark comparing its two tracker models on a synthetic scene with perfect ground truth. This benchmark uses a fixed seed (seed 1337) for scene reproducibility, with a 20-second warm-up followed by a 120-second measured interval, ensuring consistent conditions across tests. The scene parameters—including sensor model, detection generation, and metric definitions—are byte-identical, with the only difference being the tracker model. This rigorous setup emphasizes the importance of methodology in performance assessment.
The two models under comparison are the v1 “greedy nearest-neighbour” baseline, which employs a simple two-pass greedy association with constant-velocity prediction and fixed 2-second coasting, versus the more sophisticated v2 “confirmed-track auction”. The v2 incorporates advanced techniques like three-tier auction association, velocity-consistency gating, noise-scaled reservation price, and confidence-decayed coasting. Such enhancements aim to improve ID consistency and reduce errors, especially under challenging conditions.
Headline results reveal significant reductions in identity switches—an especially strict metric counting every change in track identity, including re-acquisitions and fragmentations. For example, in a baseline scenario with 150 movers at 2 fps, ID switches per minute decreased from 2,042 to 1,183, a 42.1% reduction. Similar improvements are observed in dense scenes with 400 movers, where switches dropped from 14,032 to 8,040, a 42.7% decrease. The models also show resilience under frame-starved conditions, occlusions, and degraded sensor quality, with reductions averaging around 18%. Since detection rates are identical for both models, these performance gains can be attributed solely to the tracker algorithms.
Publishing such failure numbers serves a critical purpose: both models still generate thousands of identity errors per minute under stress, and these metrics are provided transparently—exploiting the synthetic scene’s perfect ground truth. As the scene seed remains fixed, future trackers must be evaluated against the same benchmark, ensuring consistent measurement standards. This approach underscores that published failure metrics foster honest assessment, contrasting with marketing claims that often highlight only successes.
From an engineering perspective, v2 tracker performance is efficient, averaging around 1.2 milliseconds per sensor tick at a scene density of 400 movers, with worst-case processing time around 5 milliseconds—well within real-time constraints (10 ms). This demonstrates the feasibility of deploying advanced tracking algorithms in operational settings. The entire benchmark process is accessible via the live demo, where anyone can reproduce the tests by clicking “Run benchmark”—no registration or NDA required.
Because every pixel in these synthetic scenes is generated, there are no real-world variables—such as actual vehicles or persons—ensuring perfect ground truth. This synthetic approach allows for controlled, reproducible testing that is impossible with real-world data. The methodology underscores why publishing detailed failure matrices matters: they provide an honest, quantifiable measure of tracker robustness under stress, rather than relying solely on success stories.
Ultimately, this transparent benchmarking effort helps foster scientific rigor in the development of tracking algorithms. It demonstrates that even the most advanced models still struggle with thousands of identity errors, emphasizing the ongoing challenge of reliable multi-object tracking. Readers are encouraged to explore the benchmark results and try reproducing them themselves using the live demo. This openness makes it easy to see how different algorithms perform under identical, controlled conditions—an essential step toward improved solutions in the field.
Data Association for Multi-Object Visual Tracking (Synthesis Lectures on Computer Vision)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
