Transformers Reports Continuous Batching Performance Issue; Unused KV Cache Capacity May Also Slow Generation

Hugging Face project member albertvillanova submitted a reproduction on October 20: with Transformers’ generate_batch in continuous batching mode and Flash Attention, runtime may increase with KV cache capacity, even when actual requests use far less than the cache can hold. The report covers stable version 5.20.2 and development version 5.21.0.dev0; the issue remains open. Issue report

The test used an H100 80GB and Qwen2.5-0.5B-Instruct, with 64 prompts and up to 128 generated tokens per prompt. Allocating 50% of available memory to the cache took about 11 seconds per run after warm-up; increasing the allocation to 90% took about 18.7 seconds, while setting num_blocks=64 took about 1.9 seconds. The profiling trace showed that much of the GPU time was spent in prefill attention calls, whose K/V input shapes spanned the full cache capacity; decode calls had low cost. This is the reporter’s diagnosis and cannot yet be generalized to all models and attention backends. Test and trace data

The performance difference also extends to reinforcement learning training. A separate TRL issue reports that, when Transformers continuous batching is enabled, GRPO and RLOO default to allocating half of available GPU memory to the cache for each generation. In a small five-step GRPO test, the default settings took 59 seconds, while using 64 blocks reduced that to 15 seconds. This comparison applies to a specific generation configuration and is not a general speedup ratio for the training framework as a whole. TRL report

The official documentation lists cache capacity, the per-batch token budget, and scheduling as separate controls; continuous batching adjusts requests at each step to improve GPU utilization. This case is a reminder for engineers to include capacity settings in performance tests. Based on the current results, workloads with short prompts, small models, or low concurrency may be especially worth checking for the cost of over-allocation. Continuous batching documentation

Next, watch whether the prefill path is changed to process only the effective cache and whether TRL adjusts its allocation strategy. Reducing the number of blocks can be a starting point for local comparisons, but long sequences and concurrency capacity must be retested. Although the reported SDPA runtime did not change noticeably with cache size, it was still slower than Flash Attention with a small cache, so it should not automatically be treated as the best alternative.