Researchers Add Predictive Touch to Robot AI, With Gains Across Six Tasks

A robot that predicts how contact will change—not just what its cameras will see—scored higher across six manipulation tasks in the researchers’ evaluation. Their new system, DexTacWAM, averaged 70.6 against 38.0 for the strongest baseline, turning fingertip touch into part of the world the robot learns to predict.

The project brings together researchers from the University of Illinois Urbana-Champaign, UC Berkeley and Northwestern University. Their DexTacWAM research introduces a world-action model: a system that combines predictions about its surroundings with a component that chooses robot actions. Here, those predictions include both visible changes and contact at multiple fingertips.

A camera cannot see every part of a grip

The design addresses a specific blind spot. Pressure, slipping and grip stability can be hard or impossible to read from cameras, especially when a hand blocks the object. A controller with touch sensors can observe current contact, but that alone does not teach a predictive model how contact will evolve.

DexTacWAM makes that evolution a prediction target. Each fingertip supplies a grayscale image of a deformable marker grid; contact compresses or shears parts of the grid. The system processes each finger independently, then combines five fingers into a compact numerical representation for each hand, using finger identity and hand pose.

Those hand representations enter a pretrained video world model alongside visual information. One pass through the model supplies predictive features to the action component, which predicts contact force, action and the next state. It does not require the video model to finish generating a fully resolved visual prediction before supplying those features.

The test that separates sensing from prediction

The most revealing comparison removes tactile world modeling while keeping the same tactile features and action component. In that four-task experiment, performance fell sharply. The authors use this result to argue that the benefit comes from predicting contact changes, rather than merely adding touch observations to the controller.

The full evaluation used a two-handed robot platform with 22 degrees of freedom—independent movement dimensions. Its six tasks included tool-mediated force control, separating stacked bowls, unscrewing a bottle cap, wiping, placing a cube and transferring a cube between hands. Camera access was identical across methods within each task, keeping that input consistent in the comparisons.

Reusing vision training, shrinking touch inputs

The training method reuses an existing visual encoder to process the fingertip images, keeping its pretrained parameters frozen. Adapting the tactile components required four hours of data from 488 two-handed episodes. The team then trained joint visual-and-touch prediction using roughly 100 demonstrations per task, without an intermediate tactile-training stage for the video model.

A newly initialized action component learned from those same task demonstrations, rather than requiring another collection. The authors also report that visual prediction quality stayed within 0.5 decibels of matched vision-only models. Reusing visual features does not mean vision and touch share one representation space; the project explicitly distinguishes those claims.

Compression reduces the model’s inputs from one camera view plus ten fingertip streams to one camera view plus two hand representations. On a single RTX 4090, training iterations dropped from 3.87 to 1.71 seconds. Inference latency fell from 363.0 to 281.6 milliseconds per chunk.

The compressor retained 89.4% of pre-fusion contact recall. That measures preserved contact detection, not the robot’s task score. The distinction matters: the research pairs faster computation with strong manipulation results, but its headline performance finding remains bounded to the authors’ six-task evaluation on this robot platform.