SWE-Serve Adds Live Serving Tests, Exposing a Validation Gap in Coding Agent Patches

Researchers from NVIDIA and the University of California, Berkeley released the SWE-Serve paper on September 22, followed by a technical overview the next day. The benchmark turns 83 merged SGLang pull requests into 53 tasks covering model integration, decoding, caching, scheduling, and serving APIs, requiring coding agents to make changes across components in an existing codebase. [Official announcement](https://developer.nvidia.com/blog/how-swe-serve-exposes-the-gap-between-local-tests-and-live-serving/)

The central finding comes from 19 tasks that launch actual model services. The researchers held the same set of 627 agent-generated patches fixed, changing only whether end-to-end tests were included in scoring: the pass rate was 45.9% with full validation, rising to 69.4% when serving tests were removed. In other words, roughly one-third of the patches that passed the other checks still failed live serving validation. This difference in scores reflects test coverage and should not be interpreted as an improvement in model capabilities. [Paper](https://arxiv.org/html/2609.26777v1)

For each task, the researchers first verify that the unmodified code fails tests for the new functionality while passing regression tests, then require the reference patch to pass all checks. Agent-generated code is scored on its runtime behavior and does not need to match the reference solution. This allows the benchmark to detect issues with model loading, public interface responses, and state across requests, rather than checking only individual functions. [Evaluation methodology](https://research.nvidia.com/benchmarks/swe-serve)

The study uses mini-SWE-agent to evaluate 11 models across 31 combinations of models and reasoning settings, with three complete runs per configuration and a limit of 210 minutes and 350 steps per task. Access to the public internet and upstream repositories is restricted during execution, while the access needed to obtain model weights is retained. The highest mean single-run pass rate is approximately 75%, which still represents only performance on this task set with this agent framework. [Experimental setup](https://arxiv.org/html/2609.26777v1)

Anyone reproducing the results should note that the public runner requires the explicit `--closed-book` flag to match the leaderboard’s network restrictions. Omitting it produces an open-book evaluation, which should not be treated as the same test condition. The project also requires pinning a release tag, because tasks and validators may differ between tags. [Execution instructions](https://github.com/NVIDIA/swe-serve)

The engineering takeaway is to include loading actual model weights, request paths, and regression behavior in acceptance testing for agent-generated patches. However, the benchmark currently covers only SGLang, running on a CPU or a single H100; it has not evaluated multi-GPU or multi-node deployments. Passing the benchmark also does not mean a patch meets the requirements for merging or production deployment. Future work should examine reproducibility on other inference engines and whether results remain stable after validator revisions. Researchers should also distinguish insufficient test coverage from implementation errors made by agents, rather than attributing every failure to model capabilities. [Scope and limitations](https://developer.nvidia.com/blog/how-swe-serve-exposes-the-gap-between-local-tests-and-live-serving/)