Oracle publishes AI router that picks cheaper models against a quality target

Choosing the right AI model for every request is the promise. Oracle’s new public reference implementation makes a narrower, measurable choice: select the cheapest model group predicted to clear a quality target. In Oracle’s evaluation, it produced 84% good answers—matching another classifier at lower answer cost, but trailing an always-reasoning-model approach.

Published November 12, the oci-litellm-routing implementation connects LiteLLM’s AI gateway—a shared entry point for model requests—to 20 models on Oracle Cloud Infrastructure’s Generative AI service. The models span OpenAI, Google, Meta, xAI and Cohere. Applications request a model group rather than naming a vendor’s model.

A quality bar, not a correctness guarantee

The eight groups cover fast, balanced, strong, reasoning, code, vision, long-context and multilingual work. The trained classifier estimates each route’s chance of producing a good answer, then chooses the cheapest route above the threshold. That threshold defaults to 75%, and individual requests can specify a different bar.

The classifier does not make every decision alone. Oracle’s design puts three layers between an incoming request and the model that answers it:

• Request rules send images to vision models, inputs above about 30,000 tokens to long-context models, and non-English requests to the multilingual group.
• A code check directs programming requests to the code group. Otherwise, the trained router makes its cost-and-quality selection.
• LiteLLM applies business controls, including tags, key permissions, budgets, fallbacks, retries and latency-based model selection.

For an agent, routing happens on each model call, including calls made after a tool returns a result. The gateway returns the selected route, model and cost in response headers. Developers can therefore inspect individual choices rather than seeing only the final answer and total bill.

What the judge rewarded

Oracle says training used 3,590 model answers to 591 requests across 26 task families. Two models generated the requests; a judge graded anonymized answers for correctness, completeness and instruction-following. Across three labeling rounds, only 66% of requests received the same cheapest passing route every time. The router therefore predicts probabilities rather than learning one fixed winning route.

The held-out evaluation used 73 requests absent from training, with actual OCI answers assessed by that same judge. LiteLLM’s language-model-based classifier also reached 84%, at $0.338 in answer costs. Oracle describes the trained router’s result as 9% cheaper. Those figures measure this test’s answer costs, not a promised customer saving.

Oracle also discloses two demonstration prompts where the router picked the strong group even though the reasoning group passed more often. It reports that the overall ordering held on 150 fresh requests generated by another model and graded by two judges from different vendors. The repository includes scripts for customers to train on their own requests.

A deployable reference, with an unfinished check

The implementation also supports OpenAI Codex as a routed client. A custom provider connects LiteLLM to GPT-5.3-codex through OCI’s Responses API, including tool calls. Applications use LiteLLM virtual keys instead of OCI credentials; those keys can carry budgets and rate limits, while the gateway records spending per key.

Oracle supplies five deployment scripts and instructions for having Codex run them in a customer’s OCI environment. That path includes a preflight check, a proposed build and cost plan, and approval before execution. One material qualification remains: Oracle says a first build in an empty tenancy using the final scripts has not yet been run.