Full archive

Benchmark: Latest AI News

Search reporting on AI markets, models, products, companies, policy, and culture.

Showing 1-9 of 9Results matching "Agent harness"

Yesterday

1 story

15 hours ago

InferenceX Puts GB300 First for MiniMax M2.7, but Only on a Fixed Chat Test

The new ranking gives infrastructure teams a comparable speed and cost snapshot, but its single-turn design does not test the multi-step agent work MiniMax highlights for M2.7. MiniMax pitches M2.7 for complex agent harnesses and software-engineering work. GB200 NVL72 came second at 12,515 tokens per second per GPU, a 10% gap.

2 min readRead story

Earlier this week

2 stories

Benchmark Models

Sep 1, 2026

Grok 4.6 Clears 50% on Biosecurity Refusal and Routine Biology Tasks

The reported result rewards a difficult balance: recognizing hazards hidden inside ordinary-looking research requests without shutting down legitimate biological work. Its surveillance score shows that refusal performance is not the whole biosecurity test. The result puts the model’s ability to distinguish intent at the center of a biosecurity contest where excessive blocking can also impair useful work.

3 min readRead story

Benchmark Models

Aug 31, 2026

Qwen3.8-Max Leads Open-Weight Commerce Agent Bench by Two Passes, With Audit Limits

The benchmark checks whether agents leave commerce software and files in the required state, rather than simply narrate a workflow. But configuration choices and unavailable immutable result bundles limit what the narrow ranking can settle. Qwen3.8-Max holds a two-pass lead over DeepSeek V4 Pro in the published open-weight Commerce Agent Bench results, with 156 passes across three agent harnesses.

2 min readRead story

Last week

3 stories

Benchmark Tools

Aug 28, 2026

Perplexity Search API Sweeps Artificial Analysis Test, but Medium Context Wins

The result gives developers a concrete retrieval-quality lead to test, while showing that sending an agent more page text is not automatically the best route to better answers. Perplexity ’s Search API occupied the top three positions in Artificial Analysis’ Search Index, with its medium-context setting scoring 80. Search APIs give an AI agent web results and extracted page material to reason over.

3 min readRead story

Benchmark Models

Aug 28, 2026

Meta’s EvoHarness-RL Takes Qwen3-8B to 96.9% on ALFWorld, Near Claude Opus 4.5

The result is a strong benchmark showing for learned state management, not yet evidence that the framework improves the long-running enterprise workflows it is designed to target. Meta AI and the University of Illinois Urbana–Champaign say their EvoHarness-RL framework let Qwen3-8B reach 96.9% average success on ALFWorld, edging Claude Opus 4.5’s reported 96.4% score. Source: venturebeat.com.

3 min readRead story

Benchmark Models

Aug 24, 2026

NVIDIA Says AVO Took Claude Opus 5 From 30% to Perfect on ARC-AGI-3

The result shifts attention from training larger models to the software that gives them memory, tools and recovery loops. Its clearest limitation is equally important: the perfect score came on ARC-AGI-3’s public environments, while private sets remain the harder test. It used 6,624 actions, which NVIDIA’s cited comparison puts at about 12% fewer actions than the previous most efficient system.

3 min readRead story

August 2026

3 stories

Benchmark Models

Aug 22, 2026

Stanford Won Databricks’ Agent Cup, but 18.8% of Questions Stumped Every Team

The live contest tested whether agents developed on one benchmark could handle an unfamiliar Treasury archive. Stanford led the field, but the unsolved questions show the limits of reliable document reasoning. Stanford won Databricks’ inaugural Grounded Reasoning Cup with 63.3% accuracy, but 18.8% of questions were unsolved by every team.

2 min readRead story

Benchmark Models

Aug 21, 2026

Nvidia’s AVO Clears ARC-AGI-3’s Public Set. Withheld Tests Still Matter.

The result makes a strong case that long-running agent design can change benchmark outcomes. It does not yet show whether AVO transfers to ARC-AGI-3’s withheld competition environments. Nvidia says its AVO agent system completed every level in ARC-AGI-3’s public benchmark set, reaching a 100.00 Relative Human Action Efficiency score with Claude Opus 5 underneath it.

3 min readRead story

Benchmark Models

Aug 21, 2026

Open Models Are Catching the Frontier Faster. Benchmark Scores Aren’t the Whole Contest.

SemiAnalysis sees the lag from closed-model breakthroughs to open-model benchmark parity shrinking across three AI eras. But a comparable model score still leaves a harder product question: who can turn capability into dependable agent work? In the reasoning and agentic eras, open AI models reached closed-model benchmark parity in months, according to SemiAnalysis ’s cross-era analysis.

4 min readRead story