Fireworks AI finds Kimi K3.1 Pro gains come with task-specific trade-offs
The evaluation shows cheaper web research but slower, costlier banking support than Claude Opus 5.5. Routing simulations suggest a role alongside existing models.
Loading page…
The evaluation shows cheaper web research but slower, costlier banking support than Claude Opus 5.5. Routing simulations suggest a role alongside existing models.
Listen to this story
Choosing Kimi K3.1 Pro by aggregate benchmark wins could be misleading: Fireworks AI’s Nov. 17 evaluation found sharp task-level reversals, including a 64% drop on LAB: Contracts and a 22% decline on FrontierMemory, despite gains elsewhere. Kimi cut Legacy-Bench spending 48% and delivered web-search scores near Claude Opus 5.5 at 72% lower model cost, but cost more per successful banking resolution and lagged GPT-6.1 Sol in financial research. Simulated model pairing improved scores, though a live router would first have to classify tasks; none of the findings measure customer deployments.
On Legacy-Bench, Kimi K3.1 Pro completed 62.5% of tasks versus K3’s 54.3%, while spending 48% less.
On banking support, Kimi resolved 63.3% of attempts versus Opus’s 56.0%, but cost $1.51 per success versus $0.80 and took over twice as long.
GPT-6.1 Sol outscored Kimi on Rogo’s finance benchmark, 52.0% to 48.2%, and cost about $0.05 versus $0.17 per question.
Kimi K3.1 Pro beat its predecessor on 14 of 17 shared benchmarks in an evaluation Fireworks AI published November 17. But the results complicate a wholesale switch: some tasks became cheaper or more reliable, while others regressed. Simulations also found benefits from dividing work between Kimi and a competing model.
Fireworks examined 21 benchmark leaderboards, comparing task scores, model spending and execution time where available. It studied task results and agent logs more closely in banking support, company-record research, financial research and legacy-code maintenance. The findings are benchmark results and simulations, not measured customer-deployment outcomes.
Legacy-code work offered a clear improvement over Kimi K3. On Factory’s Legacy-Bench, which covers fixes, features and migrations in older programming environments, K3.1 Pro completed 62.5% of 97 tasks, against K3’s 54.3%. Model spending was 48% lower. That combines more finished work with a smaller model bill, rather than trading one for the other.
Web research produced one of K3.1 Pro’s strongest cost comparisons with Claude Opus 5.5. On the Parallel Search Capability Leaderboard, it finished 1.2 score points behind Opus while costing 72% less per question. These are model costs for that benchmark, not a general pricing comparison.
Financial research favored another competitor. On Rogo’s Big Finance Benchmark, GPT-6.1 Sol scored 52.0% at about $0.05 per question. K3.1 Pro scored 48.2% at $0.17. Matching Opus’s score at a lower cost therefore did not make Kimi the strongest score-and-cost option among the models compared.
Banking support reversed the cost story. K3.1 Pro resolved 63.3% of attempts on Sierra’s τ³-Banking benchmark, versus Opus’s 56.0%. But each successful resolution cost $1.51 versus $0.80, counting spending on failed attempts, and average execution took more than twice as long. Kimi processed roughly three times as many input tokens per conversation, outweighing its lower token prices.
Fireworks’ routing simulations assigned each of 461 tasks to one of two models; the models did not check or retry each other’s work. Selection used recorded scores and costs from other tasks in the same category, excluding the task’s own result. A hypothetical $20 value per score point set the balance between performance and spending.
Pairing K3.1 Pro with GPT-6 Astra or GPT-6.1 Sol scored higher than either member alone. That advantage held in 98% and 89% of 2,000 task resamples, respectively. The Opus pairing cost less than Opus alone in every resample, but beat Kimi alone in only 55%—a weaker case for combining them to improve scores over Kimi.
The practical hurdle is classification: the simulation already knew each task’s category. A production router would need to obtain or predict it. The resampling measured sensitivity to the task sample, not variation between repeated model runs.
Loading discussion...
Join the conversation
Explain when a better chance of resolution is worth the wait.
Be the first to share a perspective or an experience.
Reader comments
Newest comments first. Replies stay oldest first.