GPU Topology Study Adjusts Memory Allocation, Measures 14.3% Gain in H200 Decode Throughput

A GPU scheduling study submitted on September 21 suggests that the same number of streaming multiprocessors (SMs) can deliver different performance depending on their physical placement and distance from memory. By incorporating topology and non-uniform memory access (NUMA) into resource configuration, the authors increased H200 decode throughput by 14.3% in a specific Qwen3-8B test. [Paper record](https://arxiv.org/abs/2609.24270)

The study uses microbenchmarks to probe the mapping between compute resources and memory, keeping private state for prefill and decode within their respective partitions while allocating shared weights and the KV cache across partitions. The prototype is integrated into mini-SGLang, with a prefill batch size of 4 and sequence length of 4K, and a decode batch size of 128 and average KV length of 512. Compared with a baseline that already accounts for cluster compatibility, decode throughput improved by 14.3% on H200 and 10.4% on B200, while prefill gains were limited. [Methods and experiments](https://arxiv.org/html/2609.24270v1)

The prototype requires modifications to the CUDA driver so that two MIG instances can share mapped memory, so it cannot be treated as a ready-to-deploy option. The paper lists an open-source code release as future work. For now, the methods and test conditions can be examined, but the figures cannot yet be independently verified using a publicly available prototype. [Implementation limitations](https://arxiv.org/html/2609.24270v1)

Current CUDA documentation provides additional context for interpreting the results: Green Contexts can partition SM resources, but when using Thread Block Clusters, co-scheduling requirements must also be specified; supplying only the number of compute units is insufficient. The documentation also states that even separate SM resources and work queues do not guarantee concurrent execution. Resource partitioning, compute cluster compatibility, and actual execution overlap therefore need to be checked separately. [Official CUDA documentation](https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/green-contexts.html)

Measurement conventions in profiling tools also need to be consistent. The Nsight Compute documentation explains that newer versions scale attributable metrics according to the number of SMs used by a Green Context. If resource allocations overlap, some metrics may include contributions from other contexts. Comparisons of utilization should therefore record the profiling tool, driver, and resource configuration together. [Profiling documentation](https://docs.nvidia.com/nsight-compute/ProfilingGuide/)

For inference service teams, the study offers a new direction for diagnosis. The next engineering step is to hold the model, batch size, and context length constant, measure decode throughput, time to first token, and tail latency separately, and then compare configurations. A speedup in a single stage alone cannot establish whether the service as a whole completes more requests.

Future developments to watch include reproduction across GPUs once the prototype is released, and whether official interfaces can reproduce equivalent configurations and gains. Until then, deployment teams should treat the results as the authors’ measurements under specific conditions, retain records of hardware topology and software versions, and avoid extrapolating directly to an entire fleet.