Experiential Open-Sources an Agent Router That Learns From Traffic, but Results Are Unverified

Experiential offers a familiar promise: one endpoint for many model providers. It also makes a harder claim—that its gateway can learn from an agent’s production traffic and steer future work toward a better balance of cost and quality. The open-source project shows how the system works, but its performance claims have not been independently validated.

One control point before the optimization

At its base, Experiential is an open-source model gateway and router for agent workflows. It presents hosted services, bring-your-own-key connections, local models, and custom models through a single OpenAI-compatible API. The hosted service also supports Anthropic’s Messages API format.

That compatibility layer is paired with administrative controls. Teams can decide which users and agents may call particular models, constrain approved use cases, and set spending limits. The documented provider connections include OpenAI, Anthropic, Gemini, Azure, Bedrock, Fireworks, and OpenRouter.

Developers can run the gateway locally, where its data plane listens on loopback and serves the model routes. That makes the gateway usable as an integration layer before a team adopts the project’s more ambitious routing workflow.

The proposed learning loop

Experiential Labs open-sourced its World Model Optimizer in July. The tool can take an agent’s OpenTelemetry traces and turn them into text simulations of the tools and environments it uses. It then tests candidate models on those tasks and builds a K-nearest-neighbor router, which chooses models based on similar past examples.

The router is designed to optimize for cost or quality and includes prompt caching when calculating an effective request price. It also uses confidence gates: when evidence is weak or a new request is unlike its training examples, the system is meant to avoid switching away from the strongest known model.

Experiential also documents a next step beyond routing: traffic collected through the router can be used with Tinker to fine-tune an open-source model. Experiential Labs says customers will own the weights created through that training workflow, though that is the company’s stated ownership model rather than an independently assessed outcome.

The savings case still rests on internal tests

Experiential Labs reported three benchmark outcomes for its router. The results suggest it can trade modest quality loss for substantial savings in one setting, or improve both measures in two others. They were selected and run by Experiential Labs, however, so they do not establish comparable results on a customer’s distinct agent workload.

Company-reported routing benchmarks

• RouterBench: 66.5% lower cost with a 1.7% performance decline.
• TauBench: 44.5% lower cost with 6.3% higher performance.
• Terminal Bench 2: 64% lower cost with 8% higher performance.

A practical privacy and proof boundary

The product’s central input is sensitive operational history, so telemetry choices matter. Aggregate PostHog product telemetry is enabled by default but can be disabled locally; Experiential’s documentation says that telemetry excludes prompts, traces, actions, observations, paths, model names, credentials, and raw customer content.

The bigger question is whether simulations built from past traces reflect a team’s future production work. Experiential Labs has not published customer counts, revenue, or the terms of its claimed savings guarantee. Developers can test the gateway and its optimization loop locally, but the open-source release does not close the evidence gap behind that savings pitch.