Sakana AI finds open-weight models lag on its new Japanese finance benchmark
Web and terminal access improved all 12 models tested, but the strongest open-weight entry still trailed. Sakana identifies knowledge of Japanese rules as the main weakness.
Sakana AI’s JFinBench release makes Japanese financial-rule accuracy measurable across 321 questions from six certification exams. Results for 12 models show that research tools improve scores without erasing differences: Kimi K3.1 Pro narrowed its gap to the proprietary leaders from 12.4 points without tools to 4.4 with tools. That comparison uses each model’s best tested software setup, not one shared configuration. JFinBench’s refreshable construction process and exam-date controls are designed to keep results relevant as laws change and old questions risk entering training data.
01
Tax-accountant questions were a clear weak spot: the highest score was 79.4, while Claude and GPT scored 96–100 on each other exam.
02
In trials with the shared Terminus-2 setup, 67% of models fetched a web page, most often from Japan’s e-Gov law-search service.
03
The benchmark covers exams held from May to August 2026 and includes 193 multiple-choice questions, 128 written answers, and 12 image-based questions.
Giving AI models the ability to look up financial rules helped—but did not erase their differences. Sakana AI has released JFinBench, a Japanese finance benchmark built from 321 certification-exam questions. Its study of 12 models found proprietary systems ahead of open-weight rivals, both with and without access to outside tools.
The lead changes; the divide remains
Without tools, Claude Fable 5 led the evaluation, followed by GPT-6 Astra and Gemini 3.8 Flash. Kimi K3.1 Pro was the strongest open-weight entry, finishing 12.4 points behind Claude. With tools, GPT moved into first place, while Kimi remained the leading open-weight model, 4.4 points behind.
Those overall scores give each of the six exams equal weight. In the main no-tools evaluation, models answered through a single API call, and the headline results average three runs at maximum supported reasoning effort. Prompts specified the exam date and the applicable legal cutoff, so answers were judged against the rules in force for that exam.
The average also hides a particularly hard category. Claude and GPT scored between 96 and 100 on every exam except the tax-accountant exam. There, the highest score was just 79.4. Even the leading models therefore had substantially more room to improve on tax questions than on the other qualifications.
Looking up a rule is not the whole task
Sakana’s error analysis points to missing knowledge of Japanese rules and institutions as the most common failure cause for many models. Calculation mistakes were uncommon. Some models also spent their entire output allowance reasoning without producing a final answer, including entries from the MiMo, Mistral, Qwen and GLM families.
Tools offered a way to fill those knowledge gaps. Models could use a Linux terminal and the web to research laws and perform calculations over multiple steps. In trials using the shared Terminus-2 setup, 67% fetched a web page. The most frequent destination was Japan’s government law-search service, e-Gov.
Every model improved, but the rankings stayed largely similar. The authors interpret that pattern as evidence that knowledge already held inside a model still helps it decide what to search for and how to apply what it finds. Access to information did not make the competing systems interchangeable.
The tool-enabled comparison is between each model’s best tested software setup, not one identical setup for everyone. Sakana tested multiple agent frameworks and reported the highest score per model, averaging three runs with a six-hour limit per attempt. Published answer keys were forbidden; audits caught eight of 11,556 attempts referring to them and assigned those attempts zero.
A test designed to be rebuilt
JFinBench draws from exams held between May and August 2026, covering accounting, tax, financial planning and banking. Of its 321 questions, 193 are multiple-choice and 128 require written answers. Twelve include images, such as land diagrams. Written responses are graded by a language-model judge, using answer equivalence, partial credit or essay criteria as appropriate.
Freshness is part of the design. The authors warn that old questions can become outdated as laws change, or enter model training data and reward memorization. Their construction process is intended to make replacement easier: a vision-language model transcribes exam pages, then a coding agent writes exam-specific conversion programs and generates review pages.
A human reviewer compared every question with the original PDF, flagging problems such as wrongly split choices or missing reference material. The agent revised its code and regenerated the questions for another check. Sakana says that verification took about two hours per exam, roughly 12 hours in total—not 12 hours for the entire development effort.
Sakana has published the paper, dataset, code and leaderboard through its JFinBench release. Adding another exam edition requires new exam-specific conversion code and human verification, while retaining the shared transcription script. The released construction pipeline gives researchers a way to refresh the test alongside the financial rules it measures.
Sources
arxiv.orgJFinBench: A Continuously Refreshable Japanese Financial Benchmark from the Latest Certification Exams
Reader comments
Newest comments first. Replies stay oldest first.