UC San Diego and Lambda Researchers Report 3D AI Gains on Five Benchmarks

A model asked where an object sits in a room has two jobs: pick out the object and keep the room straight as the view changes. Researchers from UC San Diego and Lambda report that their model, CVP, beat a prior 3D vision-language system on five benchmarks by tackling those jobs separately.

One room, several viewpoints

A camera view alone is a shaky guide to a 3D space. The same object can look different in size, position or orientation from another angle. A model working across views must connect those appearances to a consistent scene. It also has to ignore objects that do not help answer the question at hand.

CVP gives those two problems different treatments. Its target-affinity token is trained to favor objects relevant to the current task. The researchers put visual information from multiple views into a shared 3D coordinate system and build representations of individual objects. During training, a comparison-based objective draws the token toward relevant objects and away from irrelevant ones.

For the wider room, CVP uses an allocentric grid: a compact, overhead map tied to the world rather than to one camera. The system turns that grid into a short text description for the language model. That gives it an explicit account of the scene's layout alongside the multi-view visual information and 3D positions.

The gains span three kinds of task

The team's evaluation covers questions about 3D scenes, locating objects from descriptions, and describing scene contents. Against Video-3D-LLM, CVP scored higher on each of the five reported benchmarks. The scores use different measures, so the result is a set of task-by-task comparisons, not one overall gain.

Three of the reported benchmark comparisons

SQA3D · exact-match accuracy

58.6→62.3

Video-3D-LLM scored 58.6; CVP scored 62.3 on SQA3D.

ScanRefer · Acc@0.25

58.1→62.0

Video-3D-LLM scored 58.1; CVP scored 62.0 on ScanRefer.

Scan2Cap · CIDEr

83.8→90.5

Video-3D-LLM scored 83.8; CVP scored 90.5 on Scan2Cap.

The other two comparisons point the same way. On ScanQA, the CIDEr score rose from 102.1 to 107.1. On Multi3DRefer, F1@0.25 rose from 58.0 to 60.2. The paper, accepted to WACV 2026, describes a model built on LLaVA-Video-7B and Video-3D-LLM's multi-view representation. The team fine-tuned a 7-billion-parameter model across five datasets using eight NVIDIA A100 GPUs supplied by Lambda.

Which part earns its place?

The researchers also tested what each component contributes. Removing or changing parts of the design showed a split: the object-focusing token helped most on visual grounding, where the task is to find the right object. The overhead grid mattered more for question answering and dense captioning, which draw on a broader view of the scene. Those tests support the case for using both, rather than treating the map as a substitute for object selection.

More map detail was not the answer in these experiments. The default grid has six cells by six cells; increasing its resolution brought no additional benefit in the team's tests. That is a useful constraint on the finding: the reported improvement comes from how CVP organizes and selects spatial information, not from demonstrating that a finer map always helps.

The five results establish gains on the evaluated question-answering, object-location and captioning tasks against one named baseline. They do not test a robot navigating a physical room. Whether the same way of organizing a scene holds up when observations and decisions unfold in a changing environment remains an open question.