AgentX Replays Claude Code Sessions to Test AI Serving Systems

Long conversations, repeated context and parallel subagents create a different kind of serving workload than isolated prompts. AgentX is a benchmark methodology and replay dataset designed to test how those conditions affect throughput, time to first token and interactivity across serving configurations.

InferenceX built AgentX v1.0 from 393 opt-in Claude Code sessions. The release attempts to preserve the shape of an agent workload: request lengths, shared prefixes, branching subagents and the pauses while a user’s tools run.

That distinction is consequential for infrastructure testing. A system that serves a single request quickly can face a different burden when an agent returns to a large prior context, launches parallel work, or waits between turns before sending the next request.

The benchmark keeps workload structure, not its contents

The privacy design is central to the project’s trade-off. AgentX does not publish the original conversations. Instead, it uses deterministic placeholder content to reproduce request lengths, timing, conversation topology and KV-cache reuse, the reuse of previously computed attention state for a matching prompt prefix.

• Sessions must contain at least 20 requests, use Claude Code 2.1.139 or newer, and have no more than 10 concurrent subagents.
• The dataset’s median request has 142,000 input tokens and 444 output tokens; 44% of sessions include subagents.
• Users can choose a full replay set with contexts up to 1 million tokens or a version limited to 256,000 tokens.

The v1.0 processing also removes duplicate requests, client-specific security-monitor and title-generation calls, and reconstructed inputs above 990,000 tokens. Those filters define the replay set before testing begins.

A one-hour test is meant to expose a system’s trade-offs

Each configuration receives a seeded warmup before AgentX profiles it for one hour over a range of concurrent agent clients. Primer requests and additional warmup requests establish cache state, while per-replay cache-bust markers prevent unrelated sessions from sharing prefixes.

The methodology also fixes speculative-decoding acceptance lengths using SPEED-Bench measurements for each model, speculator, draft length and thinking-mode combination. Synthetic tokens can otherwise distort how many draft tokens a system accepts. Systems without standardized DRAM are capped at 3 TB, while listed GB200 NVL72, GB300 NVL72 and TPUv7 systems use installed capacity proportional to their GPU allocation.

AgentX treats concurrency as simultaneous agent clients, not a fixed batch of requests. It is also closed loop: faster configurations complete more requests and may therefore encounter a slightly different mix of work, particularly at low concurrency. That is why the methodology calls for throughput to be read alongside time to first token and interactivity rather than as a standalone speed number.

The boundary is model quality

AgentX is a serving-system benchmark, not a model-quality test. Its client cannot observe provider-side chat templates, proprietary tokenizers, server tools, encrypted reasoning content, or the precise token expansion of images and documents; it substitutes model-specific padding where needed.

That leaves AgentX with a deliberately narrower promise: compare the systems that deliver agent workloads, while avoiding disclosure of the work itself. The key test will be whether its preserved structure is representative enough for operators to make meaningful infrastructure choices from the replay results.