Superpower Daily
ReViSQL-K2.6 Hits 91.37% on Cleaned SQL Test, but Not a Live Da...Databricks Files $5B Equity Offering With Nearly All of It Alre...Snowflake Open-Sources Semi-Persistence With Sub-Second Single-...Cisco Rolls Out MyAgent to 90,000 Employees for Supervised Cros...Midjourney Opens V8.2 Image-Editing Tests With Up to Four Refer...Harvard Uses 7 AI Faculty Avatars in a $699 Founder Bootcamp, W...ReViSQL-K2.6 Hits 91.37% on Cleaned SQL Test, but Not a Live Da...Databricks Files $5B Equity Offering With Nearly All of It Alre...Snowflake Open-Sources Semi-Persistence With Sub-Second Single-...Cisco Rolls Out MyAgent to 90,000 Employees for Supervised Cros...Midjourney Opens V8.2 Image-Editing Tests With Up to Four Refer...Harvard Uses 7 AI Faculty Avatars in a $699 Founder Bootcamp, W...
Modelspublished

ReViSQL-K2.6 Hits 91.37% on Cleaned SQL Test, but Not a Live Database

The result argues that a focused model, repaired examples, and domain-specific rewards can rival elaborate SQL-agent stacks. Its remaining test is whether that recipe transfers from curated academic data to corporate databases.

By 3 min read
ReViSQL-K2.6 Hits 91.37% on Cleaned SQL Test, but Not a Live Database
Sign in to save and react

Listen to this story

The audio brief

About 1:39
0:001:39
Read transcript
ReViSQL-K2.6 reached 91.37 percent on a cleaned SQL benchmark—but not on a live corporate database. The result is notable less because of the score alone than because it exposes how much benchmark quality can shape AI evaluation. Researchers audited roughly 2,500 examples from BIRD Train and found a problem in 61.1 percent of them. Incorrect reference SQL appeared in 52.1 percent, while flawed questions and incorrect outside knowledge also showed up; those categories overlapped. They used several rounds of expert review to create BIRD-Platinum, then fine-tuned Moonshot AI’s Kimi-K2.6 through the Tinker training service from Thinking Machines. With one generated answer—called greedy decoding—the model scored 88.55 percent on Arcwise-Plat-SQL, an expert-verified version of BIRD Mini-Dev. Generating 16 answers and choosing the majority result raised that to 91.37 percent. But the extra accuracy comes at a price: about 56 cents per task, versus 3.5 cents for greedy decoding. The training also uses reinforcement learning with execution-based rewards, designed to distinguish sound logic, equivalent SQL wording, and compliance with supplied constraints. The code, repaired data, and training instructions are available, but checkpoint and software-license details remain unclear. The key limitation is the deployment gap. No test covered permissions, changing schemas, company-specific terminology, or the complexity of a real corporate database. The open question is whether this data-repair and reward recipe transfers beyond curated academic tasks.

Story brief

3 key points

ReViSQL-K2.6’s strongest result—91.37% on Arcwise-Plat-SQL—comes from a repaired, expert-verified benchmark and 16-answer self-consistency, not a production database trial. Researchers found issues in 61.1% of roughly 2,500 audited BIRD Train examples and built BIRD-Platinum for training. The release suggests data quality and execution-aware reinforcement learning can materially shift text-to-SQL results, but buyers...

  1. 01

    An audit found incorrect reference SQL in 52.1% of sampled BIRD Train examples; issue categories overlapped.

  2. 02

    Accuracy rose from 88.55% greedy decoding to 91.37% with 16-sample self-consistency.

  3. 03

    The 16-sample configuration costs about $0.56 per task, versus $0.035 for greedy decoding.

ReViSQL-K2.6 posted a 91.37% score on a corrected SQL benchmark, but the headline number is also a warning about the data behind many AI evaluations. The model was trained on expert-reviewed examples and tested on a curated academic set, not deployed against a live company database.

Start with the data

The project fine-tuned Moonshot AI’s Kimi-K2.6 through Thinking Machines’ Tinker training API. ReViSQL-K2.6 reached 88.55% accuracy with greedy decoding, which generates one answer, and 91.37% when it generated 16 answers and selected the majority execution result on Arcwise-Plat-SQL.

That evaluation is an expert-verified version of BIRD Mini-Dev. Before building the training set, the researchers audited about 2,500 examples from BIRD Train and found that 61.1% had at least one identified problem. Incorrect reference SQL appeared in 52.1% of sampled examples; flawed questions appeared in 26.2%; and incorrect external knowledge appeared in 18.2%. Those categories overlapped.

The audit behind the score
61.1%Audited BIRD Train examples with a problem

The researchers reported at least one identified issue in 61.1% of approximately 2,500 audited BIRD Train examples.

52.1%Incorrect reference SQL queries

Reference-query errors were the most common reported issue in the sampled examples; issue categories overlapped.

91.37%ReViSQL-K2.6, 16-sample self-consistency

The 91.37% result was measured on Arcwise-Plat-SQL using 16-sample self-consistency.

Train for the failure modes

The researchers turned the audit into BIRD-Platinum, a corrected training set produced through multiple rounds of expert review and conflict resolution. That matters for text-to-SQL because a system can be marked wrong when a benchmark’s reference query is wrong, or marked right for reproducing a flawed answer.

Its training approach also differs from a workflow that routes a request through separately prompted agents for schema selection, query writing, repair, and ranking. ReViSQL uses reinforcement learning with execution-based rewards, then adds checks intended to address two SQL-specific weaknesses: whether a query is logically equivalent to the reference and whether it follows supplied background constraints.

What the reward tries to distinguish

  • A query that returns the expected result through sound logic rather than an accidental or faulty route.
  • A query that is logically equivalent to the reference even if its SQL text differs.
  • A query that respects the task’s supplied background constraints.

The bill moves into preparation

The reported cost also depends on which decoding path is used. Greedy decoding cost $0.035 per task, while the 16-sample configuration cost about $0.56 per task. The latter improves accuracy by generating more candidates, so it still uses additional inference compute even without a multi-stage agent pipeline.

The project released its code, corrected data, and training recipe. Users can reproduce training and inference commands, but must train their own checkpoint from the Kimi-K2.6 base model. The released materials do not establish whether a downloadable ReViSQL-K2.6 checkpoint or software license is available.

The central limitation is the deployment boundary. Corporate databases introduce access controls, changing schemas, organization-specific terms, and far more columns than a curated benchmark. ReViSQL’s results do not yet show how the model handles those conditions, because the work included neither a customer deployment nor an evaluation on a live corporate database.

Related coverage

Continue reading