Stanford Won Databricks’ Agent Cup, but 18.8% of Questions Stumped Every Team

Stanford won Databricks’ inaugural Grounded Reasoning Cup with 63.3% accuracy, but 18.8% of questions were unsolved by every team. The contest indicates that agent workflow design mattered on a fresh document corpus, while enterprise grounded reasoning remains incomplete.

A transfer test with little time to specialize

Eleven academic teams developed agents on OfficeQA, then applied them to OfficeQA Pro V2, a newly released benchmark. It contains 90 questions based on roughly 1,400 U.S. Treasury PDFs and about 120,000 pages of records spanning 1793 through 2024.

Teams received the corpus 36 hours before the event, giving them time to process and index it while limiting corpus-specific tuning. They competed across six 15-minute rounds of 15 questions; difficulty rose over the event and the final round carried double points.

The contest left room for system design

• Teams had to use their industry partner’s model family, but could choose retrieval, tools, document processing and human-in-the-loop workflows.
• Competition-developed agents averaged 41.1% accuracy on OfficeQA Pro V2.

63.3%Stanford

Stanford answered 57 of the 88 questions it attempted.

41.1%Teams’ average

Competition-developed agents averaged 41.1% accuracy.

18.8%Unsolved by all teams

Nearly one-fifth of questions received no correct answer from a competing team.

Stanford built a playbook, then changed its pace

Stanford’s Claude Opus 4.8 agent used more than 100 reusable skills, including procedures for table localization, financial wording and answer formatting. It also moved from parsed text and markdown-style representations to source PDFs when a parsed version lacked needed context.

The team initially used a second agent to re-extract values, check data lineage and unit scaling, and repair calculations. After earning only two speed bonuses in the first three rounds, Stanford removed that pass, later earned 14 speed bonuses, then restored verification for a final-round resubmission that helped secure the win.

The baseline figures are not a head-to-head result

In its own OfficeQA Pro V2 evaluations, Databricks put provider-specific, out-of-the-box agent harnesses at 26.0% accuracy on average. Across matched models, the company said its Genie harness increased mean accuracy from 37.5% to 52.8%, while its strongest configuration reached 60%.

Those offline baselines are not directly comparable with the live teams: frozen agents ran once across the test set without rounds, pauses or opportunities to react to intermediate failures. Databricks built the benchmark with automated synthesis and verification followed by manual review; it says systems still stumble on parsing, temporal reconciliation, entity scope and category granularity.