推薦系統
NVIDIA Rewrites KV and Beam Search for Its Generative Recommendation Stack, Making 5K-Context Inference 2.27× Faster
NVIDIA has assembled HSTU, dynamic embeddings, and Semantic ID-based generative recommendation into an executable PyTorch reference stack. It rewrites the caching and decoding paths for recommendation workloads characterized by long histories, generation of only a few tokens, and extremely large beam widths, though the results are currently limited to NVIDIA hardware and synthetic configurations.
Generative recommendation is not simply a matter of connecting a chat model directly to a product catalog. It treats user history as a sequence and predicts the next item or action. Real-world workloads often contain thousands of historical tokens, output only two or three Semantic IDs, yet must maintain 128 or 256 beams simultaneously. This differs from the multi-user, long-running autoregressive chat workloads at which vLLM and SGLang excel, and it causes conventional paged KV caches to store large shared prefixes repeatedly.
NVIDIA’s open-source `recsys-examples` brings two paths into a single PyTorch stack. In the HSTU path, TorchRec manages user, item, and event embeddings. DynamicEmb allocates rows only for IDs that actually appear and uses score-based eviction between GPU HBM and pinned host memory. The dense network integrates Megatron-Core, FBGEMM attention, and fused CUDA kernels. In tests on two DGX H100 systems, NVIDIA reports that applying these optimizations incrementally increased model FLOP utilization from 7.65% to 31.40%.
The Semantic ID path splits the cache into `ContextKV`, shared by all beams; `BeamKV`, used briefly by individual branches; and `BeamPath`, which records only ancestry relationships. It also adds continuous batching, CUDA Graph replay, and catalog-constrained top-k. On a single H100 with Qwen3-1.7B, a 5,000-token context, batch size 4, beam width 256, and three output tokens, offline latency fell from SGLang’s 349.857 ms to 154.224 ms—a 2.27× speedup. Online throughput increased from approximately 10.7 to 19.7 requests/s.
The value of this code lies in showing why recommendation models cannot simply reuse chat-serving architectures, but it does not demonstrate improved recommendation quality. The tests assume an ideal GPU cache hit rate and use a single Qwen model size and NVIDIA accelerators. For large catalogs, SSD or remote parameter-server cache behavior, data-update consistency, and real-world P99 latency must still be retested before deployment.