ServingStudio Introduces an Open Inference Performance Workbench for Simulation, Agent-Driven Changes, and GPU Validation

The University of Washington’s SyFI Lab introduced ServingStudio on September 24, bringing performance simulation, code changes, and hardware validation for LLM inference serving into one workflow. Engineers specify a model, request distribution, GPU configuration, and performance target. An agent then establishes a baseline, explores settings, and implements candidate solutions in the inference framework. [Official introduction](https://syfi.cs.washington.edu/blog/2026-09-24-introducing-servingstudio/)

The workbench consists of a simulator, an agent, and a visualization interface. The simulator uses measured GPU kernel times to predict serving performance. The agent compares predictions with execution traces, investigates whether gaps come from kernels, communication, or idle time, and then changes the code. Changes must pass correctness checks before throughput and latency are measured under a full workload to decide whether to keep or redo them. From an engineering perspective, this can narrow the search space for expensive hardware experiments; any gains still need to be confirmed through measurement. [Project documentation](https://github.com/SyFI-ServingStudio/ServingStudio)

A public case study reveals a detail in SGLang: completing automatic tuning at startup does not mean that MoE kernels inside a prefill CUDA graph have also been tuned. Graph mode uses output placeholder tensors with different shapes, producing different tuning keys and potentially falling back to a backup strategy. The patch performs tuning along the actual execution path before graph capture, avoiding an optimization that would apply only to the regular eager-execution path. [Patch proposal](https://github.com/sgl-project/sglang/pull/38560)

The authors tested GLM-5.2 NVFP4 using four B200 GPUs with four-way tensor parallelism: 240 requests, concurrency of 24, and 4,096 input tokens plus 8 output tokens per request. After warm-up, they repeated the measurement three times. Median runtime fell from 44.873 seconds to 42.503 seconds, and input throughput increased by 5.58%. This was a specific workload with long inputs and short outputs. The two test runs used different GPU groups on the same host, and the groups were not swapped. Initial tuning on the graph path also took about 91 seconds; it became much faster only after the cached results were reused. [Test conditions and results](https://github.com/sgl-project/sglang/pull/38560)

As of the review, the patch was still a draft PR. The identical aggregate scores on 64 GSM8K questions also do not establish that outputs matched question by question. [Validation limitations](https://github.com/sgl-project/sglang/pull/38560) When adopting the change, pin the model and software versions, then retest with your own input lengths, concurrency, and tail latency. The workbench pins component versions through submodules and requires Linux and several build tools. Core profiling and tests with a real framework still require an NVIDIA GPU and a compatible CUDA environment. [Deployment requirements](https://github.com/SyFI-ServingStudio/ServingStudio)

The team also plans to extend prefix-cache simulation across GPUs, host memory, and remote storage, and to release a public database of kernel performance. These remain future work. Evaluators should check whether the available timing data covers their model, precision, and hardware combination, then track upstream review and prediction error across different workloads. [Future plans](https://syfi.cs.washington.edu/blog/2026-09-24-introducing-servingstudio/)