GitHub Repo
Transformers Community Proposes Cache Offloading Fix for Crashes in All-Sliding-Window Models
A Transformers fix proposal dated October 24 says models using only sliding-window attention may fail on the first forward pass with a specific cache offloading setting. It includes a GPU reproduction and before-and-after comparison, but had not been merged as of the review.
On October 24, the Transformers community submitted PR #49955 to address offloaded_static interrupting generation under specific model configurations. The submitter reports that the first forward pass may raise a ValueError when every attention layer uses a sliding window. Affected configurations include Phi-3-mini and Mistral-7B-v0.1. This is a newly proposed fix, not an officially released correction.
Cache offloading moves some KV state to the CPU to reduce GPU memory requirements, then prefetches it before computation. The official cache strategy documentation explains that static caches offload only non-sliding-window layers by default, because sliding-window caches are size-limited and the cost of moving them may outweigh the memory savings. This default differs from dynamic caches, so engineers should not assume that enabling offloading moves every layer to the CPU.
According to the proposal, an all-sliding-window configuration leaves the list of layers to offload empty, but the prefetch function still tries to find the next target, triggering an exception. The fix returns immediately when no layers need offloading. This means the sliding-window cache for that configuration remains on the accelerator after the fix; “no longer crashes” should not be interpreted as providing additional memory savings. The proposal describes the cause and change.
The submitter demonstrated that output from the fixed version matched that of a regular static cache, using a reduced Phi-3 configuration and a tiny Mistral model on an RTX 2060 SUPER. This supports correctness for the specific path, but does not establish deployment performance for full models. Teams using GPUs with limited memory should watch for maintainer review and inclusion in a release, then compare generation output, peak GPU memory, and throughput on their actual models. For now, treat this as a community fix proposal backed by reproduction evidence. Reproduction and test results