MIT and NYU use AI to turn robot corrections into broader rules

A robot that understands “don’t spill on my laptop” should protect more than one named device. In a November 9 research account, MIT described how FLEX, developed with New York University, turned that warning into avoiding a nearby phone, too. The example was simulated, but it illustrates the study’s goal: learning broader preferences from specific corrections.

FLEX changes the features a robot uses to plan its movements—the objects or properties it treats as important to a task. Rather than keeping an ever-growing list of individual warnings, it can add a concern, turn specific concerns into a broader category, and remove objects that no longer matter. The researchers call this the Flexible Feature Editing Pipeline.

The distinction is between showing a robot what to do and explaining why. A demonstrated path does not necessarily reveal which parts of the scene drove a person’s choice. The paper argues that language is better suited to specifying those meaningful features, especially when people need to make several corrections as a task proceeds.

From a mug to everything fragile

The system combines a large language model—AI trained on text—with the robot’s motion-planning and computer-vision systems. In the version MIT describes, GPT-4o supplies the generalizations. Its role is to connect a person’s named concerns with broader properties, rather than require the person to identify every object separately.

1. Add: A correction introduces a missing concern. If a robot is carrying a bowling ball and someone says, “Don’t drop it on my mug!”, the mug becomes something to avoid. Earlier approaches also added features this way, but repeated corrections can leave a long list of narrowly defined concerns.
2. Generalize: The language model considers the mug alongside other features the person has specified. It looks for connections that support a broader rule, such as avoiding fragile objects. This is the step that lets a warning about one item affect how the robot treats other items that were not named.
3. Mask: The robot’s computer-vision system checks which objects are no longer present and removes them from the feature list. That prevents the representation from retaining every object mentioned earlier, even after the scene changes. The step complements adding and generalizing, rather than simply accumulating more instructions.

MIT says the full three-step process takes about five seconds. The intended interaction is conversational: a person supplies a correction in ordinary language, while the system handles the changes to the features used in planning the robot’s path.

The idea is that there is some higher-level intention behind the words that you say.

Nathan Dennler, study co-author and MIT postdoctoral researcher

What the experiments cover

In the simulated arm tasks, volunteers typed corrections while the robot served coffee, moved a candle or carried a bowling ball. The researchers report that FLEX connected warnings about paper with avoiding flammable objects, alongside the electronics and fragile-object examples. These tasks tested whether a correction could extend beyond its literal wording.

The original paper, submitted October 20, describes a wider evaluation: two-dimensional navigation, Franka robot-arm tabletop tasks, and a real-world study using naturalistic language corrections and real-robot demonstrations. The authors report higher normalized reward—a measure of task performance—than competing feature-editing approaches. They also say comparisons with simpler variants showed that all three editing operations were needed in interactive scenarios.

The work was led by MIT assistant professor Andreea Bobu, with co-authors Audrey Lee, Nathan Dennler and NYU’s Ilia Sucholutsky. It is scheduled for presentation at the Conference on Robot Learning in November. Bobu frames the goal as helping household robots learn what matters to each person from a sentence or two.