vLLM Community Reports AMD Prefill Race That May Trigger GPU Memory Errors

On September 27, the vLLM community filed a race condition report for the AMD inference backend: when AITER MLA, an FP8 KV cache, and asynchronous scheduling are used together, prefill may trigger GPU memory errors. The case came from a development environment with eight MI355X GPUs, and the report also includes a standalone single-GPU reproducer. The scope of impact on released versions remains to be confirmed. [Issue report](https://github.com/vllm-project/vllm/issues/58886)

The reporter traced the cause to scheduler metadata shared across batches. When the host rewrites the buffer for the next batch of requests, GPU kernels from the previous batch may still be queued and can therefore read the work allocation table for a different batch. The existing copy operation did not account for the ordering of the current execution stream, and synchronizing afterward was too late to prevent the overwrite. [Proposed fix](https://github.com/vllm-project/vllm/pull/58887)

This illustrates a basic tradeoff in inference performance optimization. The vLLM documentation says asynchronous scheduling can reduce GPU idle gaps and improve latency and throughput. AMD’s documentation notes that asynchronous transfers can overlap with computation, with streams and events used to coordinate execution order. Under this mechanism, the lifetime of shared buffers is also part of service correctness. [vLLM configuration documentation](https://docs.vllm.ai/en/latest/configuration/engine_args/), [AMD asynchronous execution documentation](https://rocmdocs.amd.com/projects/HIP/en/latest/how-to/hip_runtime_api/asynchronous.html)

The proposed fix uses two pinned host staging buffers to generate scheduler data, then copies it to the GPU on the current stream and waits for a copy-completion event before reusing the staging buffer. In the author’s test, a workload that originally failed on request 111 completed all 1,024 requests after the fix was applied. The related crash test used substitute weights, so it cannot establish the quality of the model’s responses. [Fix and test results](https://github.com/vllm-project/vllm/pull/58887)

The author also reproduced the same failure with DeepSeek-V3. A default serving test on ROCm 7.2 did not crash, but the standalone case and a modified queue configuration could still trigger the failure, so downgrading cannot be considered a definitive fix. These results currently come from the submitter and have not been independently verified; the effect of silent numerical errors on accuracy has also not been adequately quantified. [Reproduction details and limitations](https://github.com/vllm-project/vllm/issues/58886)

As of this review, the fix has not been merged. Disabling asynchronous scheduling avoided the failure in the author’s case. Deployments that match the conditions can compare behavior in a test environment first, but should measure the performance cost themselves. [Proposal status](https://github.com/vllm-project/vllm/pull/58887) Engineering teams should preserve image tags, backend selection, and cache precision so failures can be tied to specific configurations. Validation should cover new requests joining an existing decode batch, monitor completion rates and error logs, and then compare throughput. Follow-up should track maintainer review, the release that includes the fix, and regression results with real weights across long inputs and varied concurrency workloads.