GitHub Repo
Transformers Reports Continuous Batching Performance Issue; Unused KV Cache Capacity May Also Slow Generation
An October 20 Transformers report found that, with Flash Attention in continuous batching, increasing the cache allocation may increase runtime even when most of the cache is unused. Related TRL tests also showed slower training; no official fix release has been confirmed yet.