Superpower Daily
Perplexity Puts Open-Weight Models Behind One API, With OpenAI...Model-Serving Platforms Reach 6.1% of AI-Spending Businesses, W...Gemini Omni 1.1 Flash Adds 40-Second Scene Extensions and 4K Fi...Databricks Says 300M Chart-JSON Pipeline Tops Four Multimodal B...Nvidia’s $63.1 Billion Receivables Add a Cash-Flow Caveat to It...Moonbug Starts AI Tests for Cocomelon Workflows With Legal and...Perplexity Puts Open-Weight Models Behind One API, With OpenAI...Model-Serving Platforms Reach 6.1% of AI-Spending Businesses, W...Gemini Omni 1.1 Flash Adds 40-Second Scene Extensions and 4K Fi...Databricks Says 300M Chart-JSON Pipeline Tops Four Multimodal B...Nvidia’s $63.1 Billion Receivables Add a Cash-Flow Caveat to It...Moonbug Starts AI Tests for Cocomelon Workflows With Legal and...
Modelspublished

GLM-5.3’s Cheap Retries Put Fable 5’s Coding Premium Under Pressure

Together AI’s benchmark run suggests first-attempt parity is not enough to justify Fable 5’s price for general coding. But a Rust and serialization advantage, plus benchmark-specific cost assumptions, leave room for targeted routing.

By 2 min read
GLM-5.3’s Cheap Retries Put Fable 5’s Coding Premium Under Pressure
Sign in to save and react

Listen to this story

The audio brief

About 1:36
0:001:36
Read transcript
GLM-5.3 has turned a near-tie on first attempts into a significant cost challenge for Claude Fable 5. In Together AI’s DeepSWE evaluation, Fable led pass-at-one by just 69.7 percent to 69.0 percent, a gap Together AI considered within the models’ error bands. But after allowing four attempts per task, GLM reached 87.6 percent pass-at-four, versus 84.1 percent for Fable. The reported rollout cost was $3.99 for GLM, compared with $21.63 for Fable. That changes the practical question: not simply which model produces the best first patch, but which one reaches a passing patch most economically. The test covered 113 software-engineering tasks, with 904 rollouts overall. GLM won five of eight domains, including JavaScript, concurrency and durability. Fable still had clear strengths in Rust, and in data modeling and serialization, suggesting a specialty-based router could make sense. Together AI estimated that sending tasks to GLM first, then escalating after a verifier rejection, could deliver 81.1 percent accuracy at $10.74 per task. But that is a modeled result, relying on conditional independence and verifier behavior. The comparison also has an accounting caveat: GLM’s cost was index-only, while Fable had more infrastructure errors. The key constraint is whether those cost assumptions hold in production—and whether specialty routing can capture Fable’s narrower advantages without routinely paying for both models.

Story brief

3 key points

Together AI's DeepSWE evaluation suggests retry economics can outweigh a near-tied first-shot score: GLM-5.3 reached 87.6% pass@4 versus Claude Fable 5's 84.1%, while reported rollout costs were $3.99 and $21.63, respectively. A GLM-first cascade was estimated at 81.1% accuracy for $10.74 per task, but this is not a clean production benchmark: GLM's cost used index-only accounting, Fable had more infrastructure...

  1. 01

    At pass@1, Fable led 69.7%-69.0%; Together AI considered the 0.7-point gap within the models’ error bands.

  2. 02

    GLM won five of eight reported domains, including JavaScript (90% versus 75%) and concurrency and durability (62% versus 45%).

  3. 03

    Fable led Rust (85% versus 70%) and data modeling and serialization (88% versus 79%), favoring specialty-based routing.

Claude Fable 5 narrowly led GLM-5.3 on the first coding attempt in Together AI’s DeepSWE run. Once retries entered the picture, GLM pulled ahead while carrying a reported $3.99 rollout cost, against $21.63 for Fable. The result shifts the practical choice from a single benchmark score to the cost of reaching a passing patch.

The first patch is not the decision

The comparison tested both models at maximum effort on 113 DeepSWE software-engineering tasks, with four trials per model and 904 rollouts in total. Fable reached 69.7% pass@1 under official scoring, compared with 69.0% for GLM; Together AI characterized the 0.7-point difference as within the models’ error bands.

Pass@k measures whether at least one of several attempts passes a task’s hidden test suite. GLM’s advantage grew at higher attempt counts: 81.1% versus 77.1% at pass@2, and 87.6% versus 84.1% at pass@4. Its reported return was 17 solves per $100, versus three for Fable.

Same first-shot range, sharply different stated cost

0169.0% pass@1

GLM-5.3

GLM-5.3 posted 69.0% pass@1 at a reported average cost of $3.99 per rollout.

0269.7% pass@1

Claude Fable 5

Claude Fable 5 posted 69.7% pass@1 at a reported average cost of $21.63 per rollout.

Specialties, not a broad portfolio

The aggregate scores conceal distinct task strengths. GLM led on JavaScript, 90% to 75%, and on concurrency and durability, 62% to 45%. Fable led on Rust, 85% to 70%, and data modeling and serialization, 88% to 79%. GLM won five of the eight reported task domains; Fable won three.

The routing case is narrow

  • The models had a 0.65 per-task correlation and together solved 106 of 113 tasks. Their overlap limits the coverage benefit of routinely running both.
  • Together AI calculated that a GLM-first cascade, escalating after verifier rejection, reached 81.1% accuracy at $10.74 per task. That calculation assumes a verifier decides escalation and conditional independence given the task.

The benchmark has operational boundaries

The official scores excluded infrastructure errors. Fable had 16 such errors and GLM had one; counting them as failures put Fable at 67.3% pass@1 and GLM at 68.8%. Fable’s cost matched recorded trajectory totals, while GLM’s was index-only because per-turn trajectory files for its batch were unavailable publicly at analysis time.

Sources

  1. together.aiGLM-5.3 vs. Claude Fable 5 on DeepSWE: Cost, Coding, and Routing

Related coverage

Continue reading