Google DeepMind introduces a 200-task benchmark for AI agents doing visual work

Google DeepMind and Artificial Analysis introduced a 200-task benchmark on November 10 for AI agents that must reason across visually complex files and produce finished work. Called the Multimodal Knowledge Agent benchmark, or MKA, it pairs professional assignments with detailed grading criteria, testing more than an agent’s ability to answer text-only questions.

Google says Gemini 4 Carbon ranked second in quality among the models tested, while completing tasks at an average cost six times lower than the top-scoring model. That is a comparison within this evaluation, with each model working under the same tools and limits.

Assignments that require looking at the files

Google’s starting argument is that many knowledge-work benchmarks can be solved with text alone. Professional work often requires something else: cross-referencing architectural blueprints, following engineering schematics, or turning findings into decks, reports and spreadsheets. MKA is designed around that visual component, rather than treating it as optional.

Domain experts across 10 industry areas and 83 occupations helped create the tasks. Participants included architects, biologists, underwriters and data scientists. The assignments were built to assist those specialists in their jobs, grounding the test in work their professions actually require.

Each task represents a multi-step project estimated to take an experienced professional three to eight hours. Agents receive a realistic request and a bundle of files combining visual and textual information. An expert-written rubric then defines how the resulting work will be judged.

The rubrics contain roughly 70 yes-or-no checks per task. Factual accuracy carries most of the weight, but visual design also contributes to the score. The test therefore evaluates both the substance of a deliverable and how it is presented, using criteria written for that assignment.

75%Factual accuracy

Factual accuracy accounts for three-quarters of the rubric’s weighting.

25%Visual design

Visual design accounts for the remaining quarter.

Shared tools, different working habits

Every model runs through Stirrup, the same open-source software framework for operating the agent. Models get identical tools, an isolated computing workspace and the same turn budget—the allowance for successive steps. This holds the working environment constant rather than giving each model a different setup.

The models still take markedly different routes. Google reports averages ranging from 30 to 130 turns per task. It also says experiments removing vision showed models trying, and failing, to compensate for its absence—supporting the benchmark’s emphasis on inspecting visual material rather than relying on text.

An automated grading agent built on a non-Google model checks the deliverables. It opens every file, renders pages, inspects spreadsheet formulas and applies the expert rubric. Google reports that domain experts agreed with this grader 96% of the time when they assessed model outputs themselves.

The scored test stays private

Artificial Analysis runs the full 200-task evaluation and keeps those tasks private. The stated purpose is to prevent the assignments from leaking into training data or being discovered by a model through search. The scored evaluation is therefore separate from the material released for public inspection.

A separate set of 10 public tasks is available on Hugging Face so developers can inspect the format and test their own agents. Google also says model evaluation is available through Artificial Analysis. The public sample offers a way to examine the assignments without exposing the full private test.