Dharma-AI says a tailored AI vocabulary cut essay-grading inference cost 77%

Dharma-AI says rebuilding the vocabulary of its Portuguese essay-grading model cut measured inference cost by 77% while improving its benchmark score. Its VocabX method targets the text that costs most to generate—not simply the text that appears most often—without switching the underlying Qwen3-4B base model.

The team published its research preview on Hugging Face on October 21. The paper is still under review, and the authors warn that its numbers may change before publication. The central result comes from a production grader that returns a structured JSON assessment: a grade and comment for each competency.

To isolate the vocabulary change, Dharma-AI trained seven grader versions using the same base model, data, training recipe and seed. Each handled the same 500 requests on an RTX 5090 GPU using the vLLM serving software. Cost was calculated from measured runtime; the quality score combined grading accuracy with detection of off-topic or rule-breaking essays.

The grader’s reported before-and-after results

Average response length

350→101

tokens

Average response length across the 500 test requests, as reported by Dharma-AI.

Grader benchmark score

0.703→0.810

The team’s combined grading and essay-screening score improved with the revised vocabulary.

Why writing gets a bigger weight

A tokenizer divides text into the pieces a language model processes. The grader inherited a general-purpose vocabulary, despite working in one language and one response format. Repeated assessment phrases and JSON structure were split into multiple tokens. Meanwhile, the model scored thousands of vocabulary entries it rarely needed whenever it generated another token.

Not every token carried the same cost. Input text is processed together, while responses are generated one token at a time. In the grader’s measured setup, generating a token cost 7.4 times as much as reading one. That ratio varies with hardware, batch size, context length and the task; it is not a universal multiplier.

VocabX therefore weights candidate tokens by estimated serving savings in the system prompt, user message or response. It rebuilds word fragments and turns recurring phrases, lines and structured text into single “macro tokens.” Separate budgets constrain those layers. In the grader, the output vocabulary shrank from about 152,000 entries to about 65,000.

The grade needed its own token

The most compact version was not the best grader. An early vocabulary fused each competency’s grade with the surrounding JSON structure. Each grade-and-format combination became a different token, forcing the model to choose the assessment and its packaging together. Its benchmark score fell to 0.689, below the production grader’s 0.703.

Giving the grade its own token added two tokens per response but restored—and improved—quality. It also proved cheaper in the runtime measurement: the fused version had reduced cost by 73.3%, less than the revised version. The authors’ lesson is that compression can backfire when it bundles away the very decision the model exists to make.

Broader compression gains, estimated costs

Dharma-AI also compared VocabX with three reference methods across contracts, medical papers, scientific papers, news and Python code. Tests covered Qwen3, Llama-3.1 and Gemma-3, with matching mining and test documents and vocabulary ceilings. VocabX beat the strongest reference method on compression in all 15 combinations, by 1.2 to 9.3 percentage points.

Those wider cost comparisons were estimates, unlike the grader’s runtime-based measurement. The pruned VocabX version led in estimated inference-cost reduction across all 15 combinations. On contracts, it reduced tokens by 62.6% using 113,426 vocabulary entries; the best in-place tokenizer-expansion configuration reached 61.8% with 207,803 entries. The forthcoming paper is set to include full experiment tables and discarded interventions.