Loading page…
Six endpoints delivered closely matched scores, but the sample cannot rule out small quality gaps. Prices also varied substantially between providers serving the same precision.
Listen to this story
OpenRouter’s November 11 report finds that, in a one-day comparison of Qwen3.8 27B, endpoint and serving differences mattered at least as much as weight precision: scores clustered tightly, while prices and response times varied widely. The 280-task test cannot rule out a one- or two-point accuracy gap, and it covered only one model and six services. For teams weighing quantized models, the results argue for benchmarking their own prompts and providers rather than assuming lower precision is automatically cheaper or faster.
The six endpoints scored 98.3%–99.2%; overlapping uncertainty ranges mean the test did not establish a precision-related accuracy difference.
Darkbloom’s fp4 endpoint solved 26 of 40 multiplication problems, compared with 30–33 elsewhere, but the result did not establish a precision effect.
Among same-precision endpoints, the most expensive cost a median 1.72 times the cheapest; fp8 was cheaper than fp16 for 14 of 24 models.
Paying for higher numerical precision did not buy measurably better answers in OpenRouter’s new comparison of six services running Qwen3.8 27B. Published November 11, the study found no established accuracy difference between 16-bit, 8-bit and 4-bit versions. Its accompanying price analysis found that lower precision was often—but not always—cheaper.
The comparison examines quantization: storing a model’s weights, the numbers it uses to generate answers, in fewer bits. That reduces memory requirements but rounds those numbers, potentially changing answers.
OpenRouter collected the test and pricing data on October 20, 2026. It sent 280 exact-answer tasks to six endpoints—provider-specific services running the model. Cerebras supplied fp16; Parasail, CoreWeave and AkashML supplied fp8; Darkbloom and ModelRun supplied fp4. Requests were pinned to each endpoint, with fallback disabled and reasoning off.
All six endpoints scored between 98.3% and 99.2%, with overlapping statistical uncertainty ranges. That is not proof of equal quality: OpenRouter says a test this size cannot rule out a gap of one or two percentage points.
On 40 separately scored multiplication problems without working, Darkbloom’s fp4 endpoint got 26 right, versus 30 to 33 elsewhere. OpenRouter said that did not establish a precision effect. Hardware, serving software and run-to-run variation could also explain differences; only one fp16 endpoint was tested.
Where multiple endpoints served a model at the same precision, the most expensive cost a median 1.72 times the cheapest.
The price analysis compared each model’s cheapest endpoint at each precision, using three input tokens to one output token. Eight-bit was cheaper than 16-bit for 14 of 24 models, but more expensive for seven. Four-bit beat 8-bit on price for 26 of 33 models and cost more for four.
Qwen3.8 27B’s ten fp8 endpoints ranged from $0.342 to $1.137 per million blended tokens. Speed did not follow precision either: Cerebras fp16 answered in a median 1.3 seconds, versus 2.4 seconds for ModelRun fp4 and 8.4 to 15 seconds elsewhere.
OpenRouter’s inventory covered 1,455 endpoints; 59.5% had no reported precision. Users can restrict services with provider.quantizations, but that also reduces fallback options. With Cerebras pinned and fallback disabled, 93 initial requests failed three times consecutively under upstream rate limits. OpenRouter retried until all completed.
Before moving traffic to 4-bit, OpenRouter recommends testing your own prompts: its quality study covers one model, six services and one day.
Loading discussion...
Join the conversation
Explain when uncertainty alone would justify the higher bill.
Be the first to share a perspective or an experience.
Reader comments
Newest comments first. Replies stay oldest first.