Benchmark: Latest AI News

Anthropic’s Fable 5.1 Takes Benchmark Lead, but Its Top Setting Costs More Per Task

Artificial Analysis found Anthropic’s newest general model reached its highest measured intelligence score, while the token use required at maximum effort raised the cost of its own evaluation tasks.

13 hours ago

4 min readRead story

Yesterday

2 stories

GPT-5 and Gemini-3 Recovered Up to 65% of Missed Facts by Thinking Longer

The Google Research and Technion benchmark shifts the diagnosis for some factual errors from missing training data to unreliable access—while leaving open whether the result holds beyond Wikipedia facts.

Yesterday

Grok 4.6 Clears 50% on Biosecurity Refusal and Routine Biology Tasks

The reported result rewards a difficult balance: recognizing hazards hidden inside ordinary-looking research requests without shutting down legitimate biological work. Its surveillance score shows that refusal performance is not the whole biosecurity test.

Yesterday

Earlier this week

2 stories

Qwen3.8-Max Leads Open-Weight Commerce Agent Bench by Two Passes, With Audit Limits

The benchmark checks whether agents leave commerce software and files in the required state, rather than simply narrate a workflow. But configuration choices and unavailable immutable result bundles limit what the narrow ranking can settle.

Yesterday

2 min readRead story

ChatGPT o3 Handles Simulated Liquidity Choices, but Consistency Slips in Harder Cases

The simulated wholesale-payments exercise defines a narrow starting point for delegated cash management: routine payment priorities, with human oversight for potentially anomalous patterns.

Aug 31, 2026

2 min readRead story

Last week

12 stories

Hack The Box Benchmark: Best Human Team Cleared 36 Challenges; Top AI Team Reached 32

The latest benchmark points to a useful division of labor in cyber work: agents can accelerate solving, but elite performance still depends on people choosing paths and checking results.

Aug 30, 2026

Chatbots Debunked Foreign Falsehoods About 75% of the Time, Beating Search Results

The result makes chatbots a potentially stronger starting point for investigating state-spread claims than a search-results page. It does not make their answers self-validating: language, citations and product design still shape the outcome.

Aug 30, 2026

4 min readRead story

Benchmark Products

Sanctuary AI Puts Its Robot Brain on Existing Machines After a 99.5%-Plus Test

The company is pitching dexterous automation that can fit installed robot systems, but its automotive wire-plugging result has not yet become a production deployment.

Aug 29, 2026

Google’s WikiSkill Lifts Agent Benchmarks by Giving Models a Memory of Failed Work

The framework turns task traces into reusable instructions instead of changing a model’s training, offering a practical route to more capable agents while leaving uneven task gains and cross-model transfer as constraints.

Aug 29, 2026

Benchmark Tools

Lemmalog Turns Agent Memory Into Datalog State, Cutting LongMemEval Context 38-Fold

The experimental system moves long-running agent work away from recalling old notes and toward maintaining conclusions with explicit support. Its early benchmark results suggest a substantial context-saving path, while leaving the probabilistic extraction step as the central weakness.

Aug 28, 2026

Benchmark Tools

Perplexity Search API Sweeps Artificial Analysis Test, but Medium Context Wins

The result gives developers a concrete retrieval-quality lead to test, while showing that sending an agent more page text is not automatically the best route to better answers.

Aug 28, 2026

Meta’s EvoHarness-RL Takes Qwen3-8B to 96.9% on ALFWorld, Near Claude Opus 4.5

The result is a strong benchmark showing for learned state management, not yet evidence that the framework improves the long-running enterprise workflows it is designed to target.

Aug 28, 2026

ReViSQL-K2.6 Hits 91.37% on Cleaned SQL Test, but Not a Live Database

The result argues that a focused model, repaired examples, and domain-specific rewards can rival elaborate SQL-agent stacks. Its remaining test is whether that recipe transfers from curated academic data to corporate databases.

Benchmark Infrastructure

Databricks Says Async Save Cut 20B-Parameter PyTorch Checkpoints to 9 Seconds on 32 H100s

The AI Runtime components pair distributed recovery with local-NVMe data caching. The remaining operational test is whether training jobs save their input position as well as their model state.

Benchmark Infrastructure

Heidi Cuts ASR GPUs From 16 to 4 With CUDA MPS, Holding Sub-Second Latency

The production design turns spare capacity in small speech-inference requests into concurrency, but it depends on workload-specific scheduling and protections against MPS failure modes.

Databricks Says 300M Chart-JSON Pipeline Tops Four Multimodal Baselines on Answer Accuracy

The company’s test suggests chart values can become a useful retrieval layer instead of relying solely on page-image embeddings, though its comparison measures answer correctness and includes a synthetic benchmark it built.

Google DeepMind Puts Gemini Flash Lite in a Double-Blind Test to Guard Secret Benchmarks

The pilot replaces a longstanding choice between exposing confidential tests and exposing proprietary model weights. Its value now rests on whether cryptographic separation can make external testing more credible for sensitive use cases.