Benchmark: Latest AI News

Yesterday

InferenceX Puts GB300 First for MiniMax M2.7, but Only on a Fixed Chat Test

The new ranking gives infrastructure teams a comparable speed and cost snapshot, but its single-turn design does not test the multi-step agent work MiniMax highlights for M2.7. MiniMax pitches M2.7 for complex agent harnesses and software-engineering work. GB200 NVL72 came second at 12,515 tokens per second per GPU, a 10% gap.

2 min readRead story

Benchmark Models

Aug 31, 2026

ChatGPT o3 Handles Simulated Liquidity Choices, but Consistency Slips in Harder Cases

The simulated wholesale-payments exercise defines a narrow starting point for delegated cash management: routine payment priorities, with human oversight for potentially anomalous patterns. AI agents are moving closer to decisions that determine which payments get made and when. It completed routine prioritization exercises on its own, but referred potentially anomalous payment patterns for human oversight.

2 min readRead story

Last week

5 stories

Benchmark Models

Aug 29, 2026

Google’s WikiSkill Lifts Agent Benchmarks by Giving Models a Memory of Failed Work

The framework turns task traces into reusable instructions instead of changing a model’s training, offering a practical route to more capable agents while leaving uneven task gains and cross-model transfer as constraints. The result comes from a new framework that stores traces of completed work, converts them into a durable knowledge base, and uses that record to revise an agent’s operating instructions.

Aug 28, 2026

Perplexity Search API Sweeps Artificial Analysis Test, but Medium Context Wins

The result gives developers a concrete retrieval-quality lead to test, while showing that sending an agent more page text is not automatically the best route to better answers. Perplexity ’s Search API occupied the top three positions in Artificial Analysis’ Search Index, with its medium-context setting scoring 80. Search APIs give an AI agent web results and extracted page material to reason over.

Aug 25, 2026

OpenAI’s Jalapeño Claims 1.5–1.9x More AI Work Per Watt, Faces 2027 Scale Test

The custom chip gives OpenAI an early efficiency claim against available Nvidia systems, but production qualification, broader workloads and a small initial rollout will determine whether it becomes a meaningful serving platform. The company presented Jalapeño’s first benchmark results at Hot Chips, framing the chip as the opening generation of a longer-term silicon platform. Source: openai.com.

4 min readRead story

Benchmark Models

Aug 24, 2026

NVIDIA Says AVO Took Claude Opus 5 From 30% to Perfect on ARC-AGI-3

The result shifts attention from training larger models to the software that gives them memory, tools and recovery loops. Its clearest limitation is equally important: the perfect score came on ARC-AGI-3’s public environments, while private sets remain the harder test. It used 6,624 actions, which NVIDIA’s cited comparison puts at about 12% fewer actions than the previous most efficient system.

Aug 24, 2026

NVIDIA Says Vera Rubin Delivers 30x More Agentic Throughput per Megawatt, Pending Review

The preliminary comparison shifts attention from fixed prompt benchmarks toward the power required to keep long, tool-using AI sessions responsive. NVIDIA says Vera Rubin NVL72 delivered up to 30 times the throughput per megawatt of GB300 NVL72 for an agentic AI workload at 160 output tokens per second per user. NVIDIA reports the DeepSeek V4 Pro 1.6T comparison on the AgentX workload.

August 2026

5 stories

Aug 23, 2026

InferenceX Adds 11 Telemetry Views to AgentX, Exposing What Benchmark Curves Hide

The new exploration layer makes cache setup, bursty subagents and queue behavior inspectable, but its best-curve design means readers must still examine each point’s underlying configuration before treating a curve as a like-for-like comparison. A throughput curve can make two inference systems look nearly identical. The distinction reflects the workload AgentX is designed to measure.

Aug 22, 2026

SemiAnalysis Says AgentX Drove 50-Plus Upstream Fixes for AI Agents

The claimed contribution is not a faster model kernel. It is a test workload that makes state retention, routing and data movement visible—and leaves open whether its fixes generalize beyond AgentX’s replay matrix. SemiAnalysis says its AgentX benchmark has helped produce more than 50 upstream pull requests across eight layers of AI inference software.

Aug 22, 2026

LinkedIn Measures AI Code Review Against Merged Code

The company’s reported acceptance rate is a useful outcome measure, but the more transferable design is a controllable review pipeline: local rules, pre-posting filters and infrastructure built for monitoring. LinkedIn has built a multi-agent AI code-review platform that evaluates suggestions against the code that ultimately gets merged. LinkedIn reported that 63.9% of suggestions in that evaluation were accepted.

2 min readRead story

Aug 18, 2026

Nvidia Dynamo’s Agentic Speedups Target the Work Around the Model

The reported gains come from changing how serving systems route, retain and stream state under concurrency. The next challenge is whether these AgentX-specific improvements hold across production mixes and unfinished scheduling work. NVIDIA Dynamo’s agentic-serving optimization work treats the router, request plane and cache-ownership ledger as performance-critical infrastructure—not background plumbing.

4 min readRead story

Aug 18, 2026

AgentX Replays Claude Code Sessions to Test AI Serving Systems

The replay dataset is designed to measure infrastructure behind agentic coding workloads, not whether a model writes better code. Its value depends on whether synthetic traces preserve the traffic patterns operators need to serve. Long conversations, repeated context and parallel subagents create a different kind of serving workload than isolated prompts.