Back Home

GitHub Repo

SGLang Proposes Benchmark Fix to Prevent Streaming Errors from Being Counted as Successful Generations

On November 14, the SGLang community reported that after a native streaming request is rejected, benchmarks may still count it as successful and include its output tokens. Another contributor validated the proposed fix with a CPU-simulated endpoint, but it has not been merged; its impact on official releases remains to be confirmed.

Baycrest · CC BY-SA 2.5 · Image source
zh-Hant

SGLang’s Python serving benchmark has a request-counting bug. A report filed on November 14 says that when the native /generate interface sends HTTP 200 headers and then returns an error in the stream because the input exceeds the context limit, the test client may still mark the request as successful and count the requested output length toward generation totals. Issue report

The reporter’s logs show a server context limit of 16,384 tokens, but an input exceeding 130,000 tokens was received. Twenty rejected requests were recorded as successful, adding up to 20,000 output tokens, while the actual output after re-tokenization was zero. This means reported generation totals may come from request settings, so engineers cannot determine whether a test was valid based on success counts alone. Reproduction logs

A fix proposed the same day detects top-level error events in the stream, preserves the server’s error message, and excludes failed requests from successful-token and latency statistics. Another contributor independently tested it using a local HTTP 200 streaming endpoint, covering immediate failure, partial text followed by failure, and normal completion. With the fix applied, only one of three mixed requests was counted as successful, and partial latency records for failed requests were excluded from the aggregate. Fix and validation discussion

The official documentation describes online serving benchmarks as tools for measuring time to first token, time per output token, and throughput, separate from offline tests that bypass HTTP. It follows that misclassifying streaming errors can directly affect capacity assessments and hardware comparisons. When running long-context stress tests, engineers should cross-check server rejection logs, actual output, and success counts. Official benchmarking documentation

As of this review, the proposal remains open, and the page also shows several CI failures, so it cannot yet be considered an official fix. The available evidence focuses on the native streaming benchmark client and does not establish that all backends or previously published results are affected. Follow-up should track maintainer review, the merge, and inclusion in a release. Proposal status

Sources

  1. Native serving benchmark counts a context-length stream error as a successful request
  2. Reject native generation stream errors
  3. Benchmark and Profiling