Researchers find complex AI-agent setups offer no edge on ML-engineering benchmarks

More elaborate software did not give autonomous machine-learning engineering agents an advantage in a research comparison published on Apple’s machine-learning site. With the same underlying AI model and time budget, leading open-source agent setups offered no advantage over a single session of a minimally equipped coding agent, the authors report.

The paper, How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?, is by Kirill Brilliantov, Alejandro Hernández-Cano and Emmanuel Abbé, all listed with EPFL affiliations. Brilliantov and Hernández-Cano contributed equally; Hernández-Cano’s work was done while at Apple.

More machinery, or more capable models?

The researchers describe a field where autonomous machine-learning engineering agents have advanced on public leaderboards while their surrounding software has grown more elaborate. These “harnesses” include systems that coordinate multiple agents and dedicated helper agents for retrieving information.

The motivation, they write, often comes from stalled progress over long task cycles and limits in the basic actions available to language models. But improved coding agents using simpler setups have received less attention in this field.

Their baseline gives the model direct access to its execution environment through tools for reading, writing and running shell commands. The simpler setup therefore still lets the agent work directly with code; it is not limited to producing a text response.

Via a series of large-scale systematic ablation studies, we argue that the machinery layers become redundant in the coding agent setting.

Kirill Brilliantov, Alejandro Hernández-Cano and Emmanuel Abbé, in the paper’s abstract

Testing which layers earn their place

Ablation studies remove components to examine their contribution, going beyond a comparison of complete systems. The authors interpret their results as pointing to the underlying language model as the primary driver of performance.

Their conclusion is that effort spent elaborating hand-crafted harnesses around strong models yields poor returns on current ML-engineering benchmarks. That is a finding about the tested tasks, not a verdict on every use of agent software.