Full archive

Benchmark: Latest AI News

Search reporting on AI markets, models, products, companies, policy, and culture.

Showing 1-18 of 45

Today

2 stories

5 hours ago

LlamaIndex and Kaggle Launch a Document Benchmark for Cost and Long Files

ExtractBench publishes a broader test for enterprise document extraction, while its launch results include company-reported figures for LlamaIndex’s own product.

2 min readRead story

Benchmark Models

13 hours ago

Anthropic’s Fable 5.1 Takes Benchmark Lead, but Its Top Setting Costs More Per Task

Artificial Analysis found Anthropic’s newest general model reached its highest measured intelligence score, while the token use required at maximum effort raised the cost of its own evaluation tasks.

4 min readRead story

Yesterday

2 stories

Benchmark Models

Yesterday

GPT-5 and Gemini-3 Recovered Up to 65% of Missed Facts by Thinking Longer

The Google Research and Technion benchmark shifts the diagnosis for some factual errors from missing training data to unreliable access—while leaving open whether the result holds beyond Wikipedia facts.

3 min readRead story

Benchmark Models

Yesterday

Grok 4.6 Clears 50% on Biosecurity Refusal and Routine Biology Tasks

The reported result rewards a difficult balance: recognizing hazards hidden inside ordinary-looking research requests without shutting down legitimate biological work. Its surveillance score shows that refusal performance is not the whole biosecurity test.

3 min readRead story

Earlier this week

2 stories

Benchmark Models

Yesterday

Qwen3.8-Max Leads Open-Weight Commerce Agent Bench by Two Passes, With Audit Limits

The benchmark checks whether agents leave commerce software and files in the required state, rather than simply narrate a workflow. But configuration choices and unavailable immutable result bundles limit what the narrow ranking can settle.

2 min readRead story

Benchmark Models

Aug 31, 2026

ChatGPT o3 Handles Simulated Liquidity Choices, but Consistency Slips in Harder Cases

The simulated wholesale-payments exercise defines a narrow starting point for delegated cash management: routine payment priorities, with human oversight for potentially anomalous patterns.

2 min readRead story

Last week

12 stories

Benchmark Models

Aug 30, 2026

Hack The Box Benchmark: Best Human Team Cleared 36 Challenges; Top AI Team Reached 32

The latest benchmark points to a useful division of labor in cyber work: agents can accelerate solving, but elite performance still depends on people choosing paths and checking results.

3 min readRead story

Benchmark Models

Aug 30, 2026

Chatbots Debunked Foreign Falsehoods About 75% of the Time, Beating Search Results

The result makes chatbots a potentially stronger starting point for investigating state-spread claims than a search-results page. It does not make their answers self-validating: language, citations and product design still shape the outcome.

4 min readRead story

Benchmark Products

Aug 29, 2026

Sanctuary AI Puts Its Robot Brain on Existing Machines After a 99.5%-Plus Test

The company is pitching dexterous automation that can fit installed robot systems, but its automotive wire-plugging result has not yet become a production deployment.

3 min readRead story

Benchmark Models

Aug 29, 2026

Google’s WikiSkill Lifts Agent Benchmarks by Giving Models a Memory of Failed Work

The framework turns task traces into reusable instructions instead of changing a model’s training, offering a practical route to more capable agents while leaving uneven task gains and cross-model transfer as constraints.

3 min readRead story

Benchmark Tools

Aug 28, 2026

Lemmalog Turns Agent Memory Into Datalog State, Cutting LongMemEval Context 38-Fold

The experimental system moves long-running agent work away from recalling old notes and toward maintaining conclusions with explicit support. Its early benchmark results suggest a substantial context-saving path, while leaving the probabilistic extraction step as the central weakness.

3 min readRead story

Benchmark Tools

Aug 28, 2026

Perplexity Search API Sweeps Artificial Analysis Test, but Medium Context Wins

The result gives developers a concrete retrieval-quality lead to test, while showing that sending an agent more page text is not automatically the best route to better answers.

3 min readRead story

Benchmark Models

Aug 28, 2026

Meta’s EvoHarness-RL Takes Qwen3-8B to 96.9% on ALFWorld, Near Claude Opus 4.5

The result is a strong benchmark showing for learned state management, not yet evidence that the framework improves the long-running enterprise workflows it is designed to target.

3 min readRead story

Benchmark Models

Aug 27, 2026

ReViSQL-K2.6 Hits 91.37% on Cleaned SQL Test, but Not a Live Database

The result argues that a focused model, repaired examples, and domain-specific rewards can rival elaborate SQL-agent stacks. Its remaining test is whether that recipe transfers from curated academic data to corporate databases.

3 min readRead story

Benchmark Infrastructure

Aug 27, 2026

Databricks Says Async Save Cut 20B-Parameter PyTorch Checkpoints to 9 Seconds on 32 H100s

The AI Runtime components pair distributed recovery with local-NVMe data caching. The remaining operational test is whether training jobs save their input position as well as their model state.

3 min readRead story

Benchmark Infrastructure

Aug 27, 2026

Heidi Cuts ASR GPUs From 16 to 4 With CUDA MPS, Holding Sub-Second Latency

The production design turns spare capacity in small speech-inference requests into concurrency, but it depends on workload-specific scheduling and protections against MPS failure modes.

3 min readRead story

Benchmark Models

Aug 27, 2026

Databricks Says 300M Chart-JSON Pipeline Tops Four Multimodal Baselines on Answer Accuracy

The company’s test suggests chart values can become a useful retrieval layer instead of relying solely on page-image embeddings, though its comparison measures answer correctness and includes a synthetic benchmark it built.

3 min readRead story

Benchmark Models

Aug 27, 2026

Google DeepMind Puts Gemini Flash Lite in a Double-Blind Test to Guard Secret Benchmarks

The pilot replaces a longstanding choice between exposing confidential tests and exposing proprietary model weights. Its value now rests on whether cryptographic separation can make external testing more credible for sensitive use cases.

3 min readRead story