MIT researchers build robot memory that flags and revises uncertain object descriptions

A robot that mistakes a chimney for a squirrel can carry that error into later answers about its surroundings. MIT researchers have built a memory system that flags uncertain descriptions and revises them using other camera views. MIT’s November 5 account details UQ-DAAAM, which the team says improves both stored descriptions and answers based on them.

How a bad description becomes a lasting error

The system builds on DAAAM, the team’s earlier approach to robot memory. DAAAM uses a vision-language model—AI that works with images and text—to describe objects captured by onboard cameras. It stores those captions in a 3D map that the robot can consult later.

But a useful-looking caption can be wrong. Distance, partial views, blur and poor lighting can all produce misleading descriptions. Without a correction mechanism, that mistake persists as the robot moves on. In the researchers’ example, a dark rooftop object was incorrectly identified as a squirrel before being recognized as a chimney.

UQ-DAAAM adds a way to identify which memories deserve revision. Its full name is Uncertainty Quantification for Describe Anything, Anywhere, at Any Moment. Rather than accepting every saved caption as reliable, it compares descriptions of the same object across viewpoints.

Choosing which view deserves another look

The uncertainty check combines two signals: the vision-language model’s confidence in its generated words, and agreement between captions from different angles. The system combines them into one score. When uncertainty exceeds a threshold, it selects additional images to help refine the stored description.

The robot already collects images as it travels, so several frames may show the same object. A simple selection rule identifies useful viewpoints without rerunning the vision-language model during that selection. For the chimney, a later frame showing smoke could supply the distinguishing evidence.

The underlying paper, first submitted on June 6, describes a probabilistic guarantee: higher-quality candidate views selected by the method are more likely to reduce uncertainty. That is a likelihood claim about improving the evidence, not a promise that every revised object description will be correct.

A large language model then combines the selected captions into one revised description. It gives greater weight to newer captions and those with less ambiguity. MIT says the streamlined process lets a moving robot continually update uncertain memories in real time.

Contrary to ChatGPT, when we think about physical AI, it is not enough to simply put a disclaimer cautioning that sometimes the robot might be wrong.

Luca Carlone, MIT associate professor and senior author

Better recall, with robot manipulation still ahead

In the results described by MIT, refinement resolved ambiguities for about 92% of objects in the robot’s environment using two additional captions per object. The researchers also reported a 5% improvement in object-description accuracy over the state of the art.

The benefits extended beyond rewriting captions. The paper reports better performance than baselines on OC-NaVQA, a benchmark testing questions about objects, locations and timing. That tests whether the revised memory helps the robot retrieve useful information, rather than merely produce a more confident description.

Harry Zhang and Nicolas Gorlo are the paper’s co-lead authors, with Luca Carlone as senior author. The research is slated for presentation at the Conference on Neural Information Processing Systems. The team next wants to apply UQ-DAAAM to other robots, including mobile machines that manipulate objects.

Another research direction would let uncertainty guide where the robot goes. The team is studying whether the system could identify poorly understood areas worth revisiting, moving beyond selecting existing camera views to gathering better evidence through another visit.