RAG 與推論工程
AWS Uses a Small Model to Compress RAG Evidence First: Input Tokens Cut to One-Tenth, but Request Latency Rises 12%–19%
The new architecture adds a low-cost model call between retrieval and the primary model, retaining only verbatim passages relevant to the question and their source identifiers. AWS’s internal tests recorded a roughly 30% cost reduction, but the evaluation relies on a single enterprise corpus and an LLM judge.
AWS has published a query-aware compression pattern that can be inserted into an Amazon Bedrock RAG pipeline. Rather than reducing the number of retrieved results, it adds a small model after high-recall retrieval. Lambda first sends the question and all top-k chunks to Claude Haiku, instructing it to extract relevant passages verbatim, preserve chunk IDs, and neither summarize nor rewrite them. It then passes the compressed evidence to Claude Sonnet to generate the answer. Both calls use the Bedrock Converse API, so the models can be replaced with pairings from the same family, such as Nova Micro and Nova Pro. The pattern can also be combined with reranking, prompt caching, or Intelligent Prompt Routing.
This cascade is economical only when the cost of having the small model read the complete retrieved context is lower than the savings from reducing the number of tokens read by the primary model. If the original retrieved context contains `R` tokens and the compression ratio is `c`, the primary model’s input falls from `R` to `R/c`, but the system adds `R` input tokens for the small model and approximately `R/c` output tokens. AWS tested the approach using more than 500,000 documents across nine categories of enterprise content and 500 questions. Compression alone reduced the tokens sent to the primary model to 12% of the baseline and lowered cost to 67%, while increasing end-to-end latency by 19%. Reranking before compression produced figures of 10%, 64%, and a 12% latency increase, respectively. The aggregate score across four quality metrics retained about 97.5% of the baseline, while the share of answers containing at least one claim unsupported by a source fell from 51% to 44%, and to 38% with reranking.
In implementation, the real risk is evidence loss, not API integration. Setting `temperature=0` cannot guarantee complete extraction, and multi-hop questions, negative conditions, tables, and cross-chunk references particularly require independent validation. Teams should run A/B tests using identical retrieval results while measuring completeness, citation accuracy, faithfulness, cost, and P95 latency. Narrow questions with contexts exceeding roughly 5,000 tokens are the most likely to benefit, whereas short contexts or subsecond interactive services may be slowed by the additional model call.