InferenceX Adds 11 Telemetry Views to AgentX, Exposing What Benchmark Curves Hide

A throughput curve can make two inference systems look nearly identical. AgentX’s new point-level telemetry is built to show when they are not: one run may keep finding its cached conversation history, while another may repeatedly evict and recompute it. InferenceX now exposes 11 telemetry views, request timelines and conversation flamegraphs for individual results in its agentic-inference benchmark.

The distinction reflects the workload AgentX is designed to measure. The benchmark models multi-turn coding sessions with long context, high reuse of prior prompt prefixes, sub-agent bursts and tool calls. That differs from fixed-sequence tests, where each request has the same input and output length and an aggregate throughput figure can more fairly describe the run.

For an AgentX point, thousands of replayed requests can arrive as conversations grow, cache conditions change and subagents overlap. Similar aggregate throughput can therefore conceal different operational costs. The detailed view separates warmup from profiling: the reported result covers profiling, while warmup reveals the cache-fill behavior required to get there.

InferenceX has also changed what a curve represents. Rather than consistently showing separate curves for configurations with and without features such as speculative decoding, it combines allowed optimizations into one best-available curve for each model, hardware SKU and inference engine. A single line can therefore include points using different techniques, including speculative decoding, disaggregation or KV-cache offload.

That presentation favors a practical question—what is the strongest production result available from a stack—over a clean comparison of one fixed configuration. It makes provenance more important, not less. Each point exposes its container image, parallelism, precision, cache-hit rates, speculator, run metadata and a public CI provenance link.

The point pages include charts for sequence-length distributions, interactivity, time to first token or end-to-end latency, cache utilization, queue depth, cache-hit rate, throughput, prompt-token sources and unique input tokens. Points using KV-cache offload receive a dotted-circle marker and disclose the offload type and engine, plus chip and CPU cache-hit rates.

The new timeline goes further by grouping replayed requests by conversation or worker, with subagents visible under the root conversation. A click can lead from a curve point to the related anonymized conversation and turn. Flamegraphs then divide each turn into cached prefix tokens, uncached input and generated output, while showing overlapping main-agent and subagent work.

SemiAnalysis describes AgentX 1.0 as an Apache 2.0-licensed, open-source benchmark for multi-turn agentic coding at up to one million tokens. Its initial corpus replays 393 anonymized internal Claude Code traces while preserving request scheduling and prefix-reuse patterns. The InferenceXv3 matrix spans more than 1,000 chips and about 2 MW of continuously operated compute, including NVIDIA and AMD systems.

The scale and public instrumentation make the benchmark more auditable than a bare leaderboard. But the telemetry does not eliminate the judgment call at the center of agentic benchmarking: whether a replay corpus and its cache patterns represent a buyer’s own traffic. AgentX makes the assumptions and behavior easier to inspect; it does not make different deployment workloads identical.