推論基礎設施
SQD Disaggregates Decoding by Attention Type, Breaking with Dense-Model Partitioning for Subquadratic Model Inference
A new study introduces SQD, which assigns decoding workloads according to the memory and compute characteristics of quadratic and subquadratic attention. The paper reports up to a 56% gain in tokens per joule, though its Rubin/LPX figures still come from analytical models rather than a physical cluster.
A systems study included in arXiv’s latest batch on September 14 argues that existing disaggregated inference approaches are largely designed around dense Transformers, placing prefill and decode—or attention and FFN—on separate devices. When models instead use sparse, linear, or sliding-window attention, however, this partitioning may no longer align with their data-movement patterns and arithmetic intensity.
The research team proposes SubQuadratic Disaggregation (SQD). For content-based sparse attention, the system separates the top-k selection stage, which must scan the entire KV cache, from attention plus FFN computation, which processes only the selected positions. For linear- and sliding-window-attention models, it isolates the small number of dense-attention layers while keeping the remaining subquadratic layers and FFNs on the other side. The key contribution is not a new attention algorithm, but a scheduler that understands the dynamic working sets of different layers: the former requires high-capacity, indexable memory, while the latter has a more stable working set and is better suited to SRAM-based accelerators.
In a modified heterogeneous-system proxy setup using eight B200 GPUs, the authors measured average improvements in tokens/J of 53%, 31%, and 56% for GLM 5.2, Nemotron 3 Ultra, and Gemma 4 31B, respectively, relative to the strongest GPU-only baseline. Using a fixed-power analytical model of Rubin paired with LPX, they also estimate that SQD can tighten the latency bound by 1.2–1.5× and increase throughput by as much as 3.6× compared with existing attention–FFN disaggregation.
The engineering implication is that subquadratic attention may force serving frameworks to incorporate model architecture into hardware placement decisions rather than distinguishing only between prefill and decode. The current results, however, combine a proxy platform with analytical models, and the paper’s page does not list publicly available code. Cross-device communication, batch-size variation, top-k indexing overhead, and real-world tail latency still need to be reproduced on actual Rubin/LPX systems or comparable hardware.