Problem
A robot or a drone moves through a space with a camera and builds a 3D map where every object is an instance: not “points”, but “this chair”, “that table”, each with a descriptor that lets you search for it in natural language. OVO (Open-Vocabulary Online, Berriel Martins et al., RA-L 2025) does this while the camera is moving.
In real conditions, objects don’t always hold together: one gets split in two, two get mixed into one, and after a loop closure the same object can appear twice.
Constraints
- Online. The map is built while moving through the space. No post-processing.
- Cost. OVO is lighter on compute and memory than the offline alternatives. The improvement couldn’t eat into that advantage.
- Two SLAM backbones. OVO integrates with both ORB-SLAM2 and Gaussian-SLAM.
Approach
Before fixing anything, tools to see and measure: a Rerun viewer that rewinds the map, and a test bench that triggers the same drift and the same loop closure on every run. With those, each error can be traced back to the step where it starts. Three fixes:
- Geometry. The map’s points follow every correction the SLAM makes, not just the one from loop closure.
- Assignment. A contest: each point can change owner as evidence builds up view after view, and that decides whether two instances merge, split or stay as they are.
- Fusion. A shared voxel grid: only instances that actually occupy the same space get compared.
Result
- Geometry: +18% semantic accuracy (mIoU 32.4 → 38.3) on a real ScanNet scene, for under a second in total.
- Assignment: better instances in all 8 Replica scenes, including the ones not used for tuning, at no extra time cost.
- Fusion: from over 15,000 compared pairs down to about 500. Five times faster, with the same quality.
Stack and deliverables
CLIP · SAM 2 · ORB-SLAM2 · Gaussian-SLAM · bundle adjustment · Rerun
Where this fits
Systems that work in the paper and break in real conditions: find where they break, measure it, fix it.