Stanford finds leading AI agents finish under half of simulated healthcare tasks

A medical AI agent can describe the right treatment step and still leave the order unplaced. In an October 19 overview, Stanford HAI researchers reported that the strongest agents tested finished fewer than half of complete tasks in two simulated healthcare benchmarks. The tests examine clinical work and administration, checking what agents actually accomplished—not simply what they wrote.

The overview draws together three years of evaluation work. An earlier framework, MedHELM, tested individual responses across healthcare tasks and identified clinical decision support and administration as weaknesses. PhysicianBench and HealthAdminBench take the next step: give agents software, tools and a goal, then inspect the results. These are simulated workflow evaluations, not measurements of outcomes for patients receiving AI-directed care.

Clinical work: the plan is not the order

PhysicianBench contains 100 multi-step tasks adapted from real consultations across 21 specialties. Agents work with synthetically reconstructed patient records inside an electronic health record, or EHR, sandbox. Each task averages 27 tool calls. Across the benchmark, 670 checks cover finding data, clinical reasoning, carrying out actions and documentation.

Among 12 models, GPT-5.5 led with 46% completion on a single attempt. DeepSeek V4-Pro, the strongest open-source model tested, reached 19%. Repeating the exercise exposed another constraint: GPT-5.5 correctly completed only 28% of tasks in all three runs. A successful attempt therefore did not necessarily mean the agent could repeat that success.

The grading distinguishes a written intention from a completed action. Scripts check whether the required orders exist in the record; language-model judgments assess reasoning and notes. Among action-execution failures, 62% for GPT-5.5 and 73% for Claude Opus 4.6 involved writing the correct action in a note without placing the order. Those percentages describe that failure category, not all attempts.

Administration: strong pieces, unfinished workflows

HealthAdminBench moves from consultation work to prior authorizations, appeals and denials, and durable medical equipment orders. Its original paper, submitted in April, describes 135 expert-defined tasks across four simulated interfaces: an EHR, two payer portals and a fax system. Tasks are broken into smaller, verifiable steps, producing 1,698 evaluation points.

Claude Opus 4.6, using its native computer-use mode, led with 36.3% end-to-end completion. Kimi K2.5, the strongest open-weight model tested, reached 15.6%. GPT-5.4 showed why scoring individual steps can give a different impression from checking whether the entire job was done.

82.8%Individual subtasks passed

GPT-5.4 in computer-use mode passed 82.8% of HealthAdminBench’s individual subtasks.

26.7%Whole tasks completed

The same configuration completed 26.7% of whole tasks.

Different tests, a shared information problem

The clinical and administrative completion rates are not a head-to-head ranking. Their environments, model lineups, interfaces and grading methods differ. The researchers’ narrower conclusion is that both independently built tests find substantial difficulty finishing workflows. Across them, agents often fail to carry information between encounters, documents and systems.

HealthAdminBench’s failure analysis makes that coordination problem concrete. Agents encountered several recurring obstacles:

• They gathered information in the EHR but failed to save it for later use in a payer portal.
• They skipped required file uploads and downloads.
• They rarely returned to repair earlier steps, instead exhausting their execution budget.

The researchers argue that better memory and agent architecture—the software organizing a model’s actions—may matter at least as much as larger models. They are contributing both benchmarks to MedHELM, a community-owned framework released under the Apache 2.0 license, and inviting others to add tests for specific healthcare workflows.