ReViSQL-K2.6 Hits 91.37% on Cleaned SQL Test, but Not a Live Database

ReViSQL-K2.6 posted a 91.37% score on a corrected SQL benchmark, but the headline number is also a warning about the data behind many AI evaluations. The model was trained on expert-reviewed examples and tested on a curated academic set, not deployed against a live company database.

Start with the data

The project fine-tuned Moonshot AI’s Kimi-K2.6 through Thinking Machines’ Tinker training API. ReViSQL-K2.6 reached 88.55% accuracy with greedy decoding, which generates one answer, and 91.37% when it generated 16 answers and selected the majority execution result on Arcwise-Plat-SQL.

That evaluation is an expert-verified version of BIRD Mini-Dev. Before building the training set, the researchers audited about 2,500 examples from BIRD Train and found that 61.1% had at least one identified problem. Incorrect reference SQL appeared in 52.1% of sampled examples; flawed questions appeared in 26.2%; and incorrect external knowledge appeared in 18.2%. Those categories overlapped.

61.1%Audited BIRD Train examples with a problem

The researchers reported at least one identified issue in 61.1% of approximately 2,500 audited BIRD Train examples.

52.1%Incorrect reference SQL queries

Reference-query errors were the most common reported issue in the sampled examples; issue categories overlapped.

91.37%ReViSQL-K2.6, 16-sample self-consistency

The 91.37% result was measured on Arcwise-Plat-SQL using 16-sample self-consistency.

Train for the failure modes

The researchers turned the audit into BIRD-Platinum, a corrected training set produced through multiple rounds of expert review and conflict resolution. That matters for text-to-SQL because a system can be marked wrong when a benchmark’s reference query is wrong, or marked right for reproducing a flawed answer.

Its training approach also differs from a workflow that routes a request through separately prompted agents for schema selection, query writing, repair, and ranking. ReViSQL uses reinforcement learning with execution-based rewards, then adds checks intended to address two SQL-specific weaknesses: whether a query is logically equivalent to the reference and whether it follows supplied background constraints.

What the reward tries to distinguish

• A query that returns the expected result through sound logic rather than an accidental or faulty route.
• A query that is logically equivalent to the reference even if its SQL text differs.
• A query that respects the task’s supplied background constraints.

The bill moves into preparation

The reported cost also depends on which decoding path is used. Greedy decoding cost $0.035 per task, while the 16-sample configuration cost about $0.56 per task. The latter improves accuracy by generating more candidates, so it still uses additional inference compute even without a multi-stage agent pipeline.

The project released its code, corrected data, and training recipe. Users can reproduce training and inference commands, but must train their own checkpoint from the Kimi-K2.6 base model. The released materials do not establish whether a downloadable ReViSQL-K2.6 checkpoint or software license is available.

The central limitation is the deployment boundary. Corporate databases introduce access controls, changing schemas, organization-specific terms, and far more columns than a curated benchmark. ReViSQL’s results do not yet show how the model handles those conditions, because the work included neither a customer deployment nor an evaluation on a live corporate database.