Researchers report 72.4x faster attention-layer prefill at 10 million tokens

Researchers have introduced a method that makes one attention layer process a 10-million-token prompt 72.4 times faster than full attention in their tests. Their October 30 paper presents Latent Sparse Attention, or LSA, which searches groups of stored information rather than scoring every key individually, while reporting similar accuracy across several tasks.

The work addresses a problem beyond simply allowing a model to accept more text. The authors argue that long-running AI workflows, including open-ended scientific work, benefit from long context windows. But above 1 million tokens—the pieces of text a model processes—full attention becomes prohibitively expensive.

Search the clusters, not every key

Sparse attention narrows the work by selecting a subset of keys, the representations that attention searches. A dedicated indexer makes that selection. Yet the authors identify a catch: existing indexing methods either retain costs that grow quadratically with context length or lose accuracy at ultra-long lengths. Selecting less information does not necessarily make finding it cheap.

LSA changes that search. It organizes keys into a hierarchy of clusters within a learned numerical representation, which the paper calls a latent space. Queries then score that hierarchy instead of every individual key. The method targets the selection step itself, with the authors reporting benefits for prompt processing, response generation and training.

22.7xPrefill versus original sparse attention

Author-reported speedup for one attention layer using D-LSA on GLM-5.3-FLASH at 10 million tokens.

72.4xPrefill versus full attention

Author-reported speedup for one attention layer using D-LSA on QWEN-3.8-27B at 10 million tokens.

Prompt processing is not response generation

Those prefill results concern processing the input prompt. For decode—the stage that generates a response—the authors report smaller gains at the same context length: 4.6 times faster than the original sparse attention and 42.5 times faster than full attention. These are attention-layer measurements, not measurements of how much faster an entire model completes a task.

Retrofitting models or training from scratch

The paper proposes two versions, separating adoption in existing models from training a model with the method built in:

• Distilled-LSA, or D-LSA, can be applied to pretrained models through distillation. This is the version behind the reported 10-million-token results. The authors say it preserves similar accuracy across a variety of tasks.
• Native-LSA, or N-LSA, is trained directly using next-token loss: the objective of predicting the next piece of text. At 128,000 tokens, the authors report that it matches full-attention accuracy while delivering a 1.6-times training speedup per layer.