IBM says open AI agents found a 52× estimated hardware speedup for $26 in inference

IBM Research says an all-open AI agent team found a quantum error-correction hardware design with a 52× estimated speedup over its baseline, spending about $26 on model inference. Its October 13 study also shows the tradeoff: proprietary models found a 204× speedup, but their inference bill reached $299 under the same search budget.

These are high-level-synthesis estimates of hardware performance, not measured results from a fabricated chip. The prices cover running the models, not the complete hardware-development workflow. IBM evaluated each configuration in one search run, so the findings describe those searches rather than establish a universal model ranking.

A search for faster decoding

The test used Relay-BP, an algorithm that interprets measurements to infer corrections for errors affecting quantum information. That decoding is classical computation supporting quantum error correction. IBM chose a demanding hardware problem: Relay-BP’s repeated message passing offers parallel work, but the baseline’s sparse graph representation and memory-access patterns make that parallelism difficult for synthesis tools to exploit.

IBM’s agent factory organized the search around a coordinator and four parallel workers. The coordinator assigned optimization ideas; workers changed the decoder implementation. Correctness checks and synthesis results then fed the next round. High-level synthesis translates C or C++ implementations into hardware designs, letting the team evaluate how code changes affect latency and hardware resource use.

Each of three configurations received 100 hypotheses to explore: proprietary models throughout, open models throughout, and a proprietary coordinator directing open-model workers. The agents pursued changes with hardware consequences, rather than simply rewriting code:

• Reorganizing message storage helped the synthesis tool recognize independent memory accesses and perform more work in parallel.
• Replacing sequential reductions with tree structures exposed parallel computation while preserving the decoder’s computation.

Cheaper serving, not fewer tokens

The open-model search actually consumed more tokens—the units of text models process and generate. It used approximately 241 million, compared with 216 million for the proprietary search. IBM attributes the lower bill to the economics of serving those tokens, not to doing less model work.

The open model ran on premises through llm-d, an open-source distributed inference framework. An agent factory repeatedly revisits source code, constraints, earlier changes and synthesis reports. llm-d routes requests toward servers holding reusable context, reducing repeated processing. It also separates prompt processing from token generation so resources can be allocated to each independently.

For this workload, IBM reports an approximately 98% prefix-cache hit rate—a measure of reuse of previously processed prompt context. The blended serving cost was about $0.11 per million tokens. The factory ran alongside other agent workloads on shared infrastructure, and open weights let IBM keep the design artifacts under its control on premises.

More attempts brought smaller gains

Doubling the all-open search budget from 100 to 200 hypotheses increased its estimated speedup from 52× to 60×. Additional exploration found a better result, but the improvement was modest compared with the increase in attempts. Lower inference cost did not automatically translate into proportionate hardware gains.

IBM says future work will explore adaptive resource allocation, a wider range of architectural ideas, and selective use of stronger models for difficult changes or coordination. That shifts the next research question from how many attempts are affordable to where extra reasoning or a different model can most usefully improve the design.

The researchers also identify synthesis and verification costs as necessary parts of future evaluation. A cheaper model-serving bill creates room for more exploration only if the rest of the development process can support the added experiments.