OpenRelay Routes AI Inference Across GPUs, TPUs and Trainium Through One API
The early-access network aims to let developers buy performance rather than a particular accelerator, but its cost-savings claims and ability to deliver consistent service across mixed infrastructure remain unproven.
Listen to this story
The audio brief
Story brief
3 key pointsOpenRelay is testing whether a software layer can make fragmented accelerator supply look like one inference market. Its early-access service claims to route 100 billion tokens weekly across 22 locations and eight configurations, while accepting OpenAI- and Anthropic-compatible requests through a stable /v1 API. Developers can also access the pool through dedicated GPU VMs or batch jobs. The main commercial promise...
- 01
The network spans Nvidia, AMD, Google TPU, and Amazon Trainium capacity across 22 physical locations.
- 02
Developers can switch via base URL and API key using OpenAI or Anthropic wire formats.
- 03
OpenRelay claims 10%–20% savings, but has not published workload mix, comparison period, or methodology.
OpenRelay has launched an inference API that sends AI workloads across GPU, TPU and Trainium capacity through a single endpoint. The product promises to shift a developer’s choice from selecting a cloud and chip to setting performance requirements, while OpenRelay’s scheduler chooses available hardware that meets them.
The company says its scheduler continuously benchmarks capacity, then routes each job using cost, latency and availability alongside a customer’s requested latency and throughput. That puts one control layer in front of accelerators from Nvidia, AMD, Google and Amazon, whose hardware otherwise requires developers to manage separate infrastructure choices.
OpenRelay is in early access, though its /v1 API is listed as stable. It supports OpenAI and Anthropic wire formats, so a developer using those formats can test the service by changing a base URL and API key instead of replacing an existing SDK.
OpenRelay says the network processes 100 billion tokens each week.
The company says those workloads run across 22 physical locations and eight accelerator configurations; the operating figures are self-reported.
The service is also trying to assemble the supply behind that endpoint. Data centers, GPU clouds and companies holding reserved capacity can contribute machines. OpenRelay says it takes on workload isolation, orchestration, metering and billing, then shares usage revenue with the hardware operator.
One capacity pool, three product paths
- Inference API: Developers submit workloads through the common endpoint and specify latency and throughput requirements.
- Dedicated GPU virtual machines: OpenRelay offers dedicated GPU VMs from the same underlying capacity pool.
- Batch jobs: The same pool is also available for batch processing, rather than only live inference requests.
The commercial appeal is straightforward: unused or reserved accelerator capacity can become supply for other customers, while developers gain an alternative to building their own multi-provider routing stack. OpenRelay says this arrangement can reduce inference costs by as much as 20%; its website gives a narrower 10% to 20% range for high-throughput workloads.
Those savings are not yet independently legible. OpenRelay has not published the workload mix, comparison period or benchmark methodology behind its percentages, so the result will depend on a customer’s model, traffic pattern and the hardware available when requests arrive.
Cost is only one part of the product test. Different accelerators can require different kernels, runtimes or model formats, and shifting traffic can introduce cold starts or uneven latency. OpenRelay says it handles those differences with continuous benchmarking and automated routing; whether it can maintain service levels across hardware, clouds and operators remains the central unresolved question.
OpenRelay was launched in August by Jaden Wang and Prashant Patel and is part of Y Combinator’s Summer 2026 batch. Its wager is that a reliable routing layer can make fragmented accelerator capacity behave like a single market; early access will show whether the interface is enough to hide the operational differences underneath.
