Superpower Daily
Snowflake Open-Sources Semi-Persistence With Sub-Second Single-...Databricks Files $5B Equity Offering With Nearly All of It Alre...OpenAI’s 1,200 Test Agents Built a Covert Network and Breached...Model-Serving Platforms Reach 6.1% of AI-Spending Businesses, W...Harvard Uses 7 AI Faculty Avatars in a $699 Founder Bootcamp, W...Perplexity Puts Computer Behind MCP, Opening 400+ Services to A...Snowflake Open-Sources Semi-Persistence With Sub-Second Single-...Databricks Files $5B Equity Offering With Nearly All of It Alre...OpenAI’s 1,200 Test Agents Built a Covert Network and Breached...Model-Serving Platforms Reach 6.1% of AI-Spending Businesses, W...Harvard Uses 7 AI Faculty Avatars in a $699 Founder Bootcamp, W...Perplexity Puts Computer Behind MCP, Opening 400+ Services to A...
Toolspublished

AWS’s RAG Cost Cut Comes With a 19% Latency Bill

AWS’s two-model pattern sharply reduced the context reaching its answer model in one benchmark. The tradeoff is another inference step, a modest quality decline, and results that may not transfer to a company’s own documents.

By 3 min read
AWS’s RAG Cost Cut Comes With a 19% Latency Bill
Sign in to save and react

Listen to this story

The audio brief

About 1:30
0:001:30
Read transcript
AWS says it can cut a retrieval-augmented generation pipeline’s cost to 67 percent of baseline—but the bill comes due in latency. Its query-aware compression pattern adds Claude Haiku between retrieval and Claude Sonnet. Haiku reads the question and the retrieved document chunks, then extracts only relevant passages, word for word, along with their chunk identifiers. Sonnet uses that smaller evidence set to generate the answer, preserving material that can support citations instead of relying on a summary. In AWS’s benchmark, the approach reduced the context sent to Sonnet to 12 percent of the original amount and cut total cost by 33 percent. But the extra model call increased end-to-end latency by 19 percent, while composite quality reached 97.5 percent of the uncompressed baseline. Adding reranking improved the scorecard: cost fell to 64 percent, Sonnet’s input fell to 10 percent, latency rose 12 percent, and quality reached 97.6 percent. The test covered more than 500,000 documents, nine enterprise source types, and 500 questions. AWS also reports a lower hallucination measure, but warns that the results may not generalize. The central risk is omission: a small model can remove evidence needed for a multi-step answer. Both calls run in one AWS Lambda function through the Amazon Bedrock Converse API. The practical question is whether a team’s own documents make that filtering reliable enough to justify the latency tradeoff.

Story brief

3 key points

AWS has published a query-aware RAG pattern for Amazon Bedrock that inserts Claude Haiku between retrieval and Claude Sonnet, extracting verbatim evidence before answer generation. In its 500-question benchmark, compression lowered total cost to 67% of baseline and primary-model input to 12%, but raised end-to-end latency 19% and delivered 97.5% composite quality. Adding reranking improved those figures to 64% cost,...

  1. 01

    Haiku extracts query-relevant spans and chunk IDs, preserving citation evidence rather than summarizing retrieved content.

  2. 02

    Evaluation covered 500,000+ documents, nine enterprise source types, and 500 questions; AWS warns results may not generalize.

  3. 03

    Compression-only hallucination measure fell from 51% to 44%; rerank-plus-compression reached 38%, but extraction omissions remain a risk.

AWS is not promising that RAG can become cheaper without a sacrifice. Its query-aware compression pattern cuts the context sent to the main answer model, but AWS’s benchmark also shows a 19% latency increase and composite answer quality at 97.5% of the baseline. The proposal is a targeted trade: spend a small-model call to avoid paying a larger model to read material the question does not need.

A second model becomes an evidence filter

The design adds one step after retrieval and before answer generation. A retriever first returns the top-k document chunks; AWS then has Claude Haiku read the user’s question and those chunks, extract only query-relevant verbatim spans, and pass that compressed evidence to Claude Sonnet for the final answer.

Both invocations run in one AWS Lambda function through the Amazon Bedrock Converse API. The key constraint is extraction rather than summarization: preserving source wording and chunk identifiers is intended to keep evidence available for downstream citations while removing irrelevant text.

The savings depend on a simple price-and-volume equation. The smaller model must process all retrieved text and produce the shortened context, but the higher-priced answer model reads only that shorter result. AWS says the pattern is most favorable when retrieved context is large, the price gap between the models is wide, and substantial material can be removed without losing needed evidence.

AWS’s benchmark tradeoff
67% of baselineCost with compression

AWS measured compression costs at 67% of its baseline pipeline cost.

12% of baselineContext sent to primary model

Compression reduced tokens sent to the primary model to 12% of baseline in AWS’s benchmark.

+19%End-to-end latency

The additional compression step increased latency by 19% versus the baseline.

The token result is large; the evaluation is bounded

AWS tested more than 500,000 documents across nine enterprise source types and 500 questions in 10 categories. It compared an uncompressed baseline with compression and rerank-plus-compression, using an LLM judge to score correctness, completeness, citation accuracy and conciseness against reference answers; it also tracked faithfulness against the evidence given to the final model.

On that setup, compression reduced cost by 33% and cut primary-model input tokens by 8.6 times. The quality score was slightly lower than baseline, and AWS says completeness and citation accuracy declined slightly while conciseness improved. That makes the result a cost-performance measurement, not evidence that the filter improves every answer dimension.

Where the decision gets harder

  • Latency-sensitive applications need their own measurement: the compression call is an added step, even if the final model has less context to process.
  • Questions requiring multi-step reasoning can depend on an evidence chain staying intact, which AWS identifies as a deliberate compression-design concern.
  • AWS says its benchmark represents one corpus, domain and query distribution, so performance can differ with other documents, queries and model choices.

Reranking improves the scorecard, not the deployment verdict

AWS’s combined rerank-plus-compression pipeline went further in the benchmark: cost fell to 64% of baseline and primary-model tokens to 10%. It also posted 97.6% of baseline composite quality and 12% higher latency, rather than the compression-only pipeline’s 19% increase.

The benchmark’s hallucination measure moved in the favorable direction, from 51% in the baseline to 44% with compression and 38% with reranking plus compression. But the mechanism also creates a new operational responsibility: teams must validate that an extractor does not omit the evidence a final answer needs. AWS recommends adapting the prompts to the document set and question types rather than treating the example pipeline as a drop-in default.

Sources

  1. aws.amazon.comReduce RAG costs on Amazon Bedrock with query-aware compression | Amazon Web Services

Related coverage

Continue reading