Salesforce study finds top visual AI assistant finishes 53.4% of timed simulated tasks

Salesforce AI Research detailed a test of visual AI assistants on November 12 that asks whether their advice actually helps someone finish a task. In its simulated environments, the strongest assistant completed 53.4% of tasks before time ran out. Correct guidance alone was not enough: it also had to arrive while someone could act on it.

A recording cannot follow instructions

The November 12 account explains Interactive Visual Gym, or IVG, whose research paper was submitted to arXiv on November 5. Its starting contrast is with evaluations that replay prerecorded video and grade an assistant’s response against an answer key. Those tests can check whether a warning is right, but the person in the recording cannot change course after hearing it.

IVG instead pairs an assistant with a simulated user inside an environment that responds to instructions. The assistant speaks in everyday language rather than issuing robotic movement coordinates. In a cooking task, for example, it might need to catch a wrong ingredient before the virtual cook uses it. Both the user’s actions and the environment can change in response to that guidance.

Researchers can introduce mistakes, plan changes, questions and reminder requests at chosen points. Simulated users also differ in skill, trust in AI, preferred speed and tolerance for interruptions. That lets the test distinguish between helping a novice who wants detailed instructions and helping an experienced user who would rather be left alone.

315Interaction scenarios

IVG-Bench covers cooking, household manipulation and mobile-interface tasks across three environments.

15Assistants evaluated

Each assistant ran on a different model.

Finishing the task is only one score

Gemini 4 Argon powered the highest-scoring assistant, with its completion result averaged across environments. Salesforce says open-weight models generally trailed the leading frontier models. But ranking assistants by finished tasks alone would miss another finding: an assistant can help users finish more often while giving clumsier advice along the way.

Timing creates a particularly clear failure. An instruction to approach a table is useless if the user has already arrived. Salesforce also observed assistants repeating instructions that users had acknowledged, while some open-weight models’ guidance deteriorated as tasks continued. These are failures of the unfolding interaction, rather than simply wrong answers to isolated questions.

Two measures, different questions

• Task success tracks whether the user completes the job within the time limit.
• Interaction quality captures a separate dimension of assistance; a better completion rate does not necessarily mean better guidance along the way.

How closely do simulated users match people?

To check the interaction-quality measure, the researchers had people control the cook in the simulator, work with different assistants and rank them. The automated score agreed with participants’ preferences 78.5% of the time. That is a comparison with human preferences during a simulated cooking task, not a measurement of successful assistance in real kitchens or workplaces.

Salesforce says real participants and simulated users behaved similarly, with one notable difference: the simulated users spoke and replied much more often. That difference qualifies how closely the artificial conversations resemble human ones, even when the scoring broadly aligns with people’s preferences.

The same environments can also supply feedback for training, rather than only grading. In a proof-of-concept experiment, the research paper reports improved task success and interaction quality for Qwen3.5-4B after training in the cooking environment. Salesforce’s account describes rewarding task completion and error prevention, with the assistant becoming better at intervening before mistakes. That result is narrower than the full benchmark: it concerns one model trained in one environment.