AI 基礎設施
OpScale Breaks LLM Scaling Down to the Operator Level, Cutting Average GPU Use for Qwen2 Serving by 35%–50%
Instead of replicating an entire model, OpScale scales only the operators currently creating bottlenecks—such as attention and MLP—based on traffic and sequence length. The authors replayed 929,000 requests on A100 and GB200 clusters, using up to 36.3% fewer GPUs while meeting the same latency targets, but the prototype code has not yet been released.
LLM serving systems typically use a complete model replica as the unit of autoscaling. However, loading 70B-scale weights takes around 10 seconds, making it difficult to keep up with traffic spikes that occur on a seconds-level timescale. Moreover, attention, linear projection, normalization, and MoE operators respond differently to batch size, sequence length, and memory constraints. The new OpScale system therefore treats a model as an operator DAG, replicating or removing only the nodes currently limiting TTFT or TBT.
The system first profiles each operator type offline, measuring execution time, weight and temporary memory requirements, communication volume, and performance under different SM allocations. An online controller then uses a millisecond-scale greedy algorithm to adjust batch sizes, operator replicas, tensor-parallel partitioning, and device placement. Its estimated resource cost is within 8% of a brute-force oracle. Physical placement uses best-fit packing that accounts for HBM, SMs, NVLink/InfiniBand communication, and co-location interference. The execution layer partitions SMs using [CUDA Green Contexts](https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__GREEN__CONTEXTS.html), and connects operator replicas through reusable streams, dynamic KV memory, and shortest-queue routing.
The research team rewrote roughly 17,000 lines of Python on top of [nano-vLLM](https://github.com/GeeeekExplorer/nano-vllm) and evaluated the system using Qwen2-7B, Qwen2-57B-A14B, and production traffic traces. The dense model used an average of 7.1 GPUs, compared with 11.2–14.3 GPUs for the baseline systems, while achieving a 98.4% SLO attainment rate versus 88%–95% for the baselines. The MoE model used an average of 10.8 GPUs and achieved a 98.1% SLO attainment rate. Scaling a single operator took just 0.03 seconds on average, and even when scaling all operators simultaneously, P99 remained below 0.45 seconds. Model-level scaling took an average of 10.68 seconds. Under a fixed SLO, GPU usage fell by as much as 36.3%, while cluster power consumption dropped by 14%–28%.
These results reframe serving elasticity from “launch another model replica” into a dataflow scheduling problem, an approach that is particularly appealing for mixed-model workloads and bursty coding traffic. However, the comparison reproduced other scaling strategies on a shared nano-vLLM backend rather than deploying each system’s native implementation directly. The gains also depend on high-speed interconnects and GPU spatial partitioning, and the authors acknowledge that improvements are larger within a GB200 NVL domain. More importantly, OpScale itself has not yet been released, so engineering teams still cannot independently validate the 17,000-line prototype or assess the costs of fault recovery and multi-tenant isolation.