Transformers cache offloading reportedly changes Qwen3.8 outputs on first reuse

On September 19, Hugging Face Transformers received a report of a cache issue affecting a hybrid model: with CPU offloading enabled, Qwen3.8-27B produced a different next token the first time its cache was reused. The report includes a minimal reproducer running Transformers 5.17.0 and PyTorch 2.14.0 on a single RTX PRO 6000 Blackwell GPU, using BF16 and SDPA. [Issue report](https://github.com/huggingface/transformers/issues/48947)

Cache offloading trades data transfer time for lower GPU memory usage: most layers’ historical states reside on the CPU, with the next layer’s state prefetched for computation and moved back afterward. The official documentation lists offloading as an option when GPU memory is insufficient and allows users to create `DynamicCache(offloading=True)` manually. Engineering checks should cover both throughput and output consistency after states move between devices. [Cache documentation](https://huggingface.co/docs/transformers/v5.17.0/en/kv_cache)

The two inputs in the report each contain 4,096 random tokens. The initial prefill outputs were identical, but the next forward pass produced a maximum absolute difference of 2.117 in the logits, and the highest-scoring tokens differed. This exposes a blind spot in tests that check only prefill: the model can start normally and use less GPU memory while its generation results have already changed. These figures come from the submitter and have not been independently reproduced. [Test code and output](https://github.com/huggingface/transformers/issues/48947)

The hybrid architecture broadens the scope of validation. The model configuration lists 48 linear-attention layers and 16 full-attention layers. The source code for this Transformers version shows that linear-attention layers store convolutional and recurrent states, while full-attention layers store key-value tensors. Verifying transfers of conventional KV tensors alone therefore does not cover every state. However, this does not yet establish which type of state is responsible for the issue. [Model configuration](https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/config.json), [cache implementation](https://raw.githubusercontent.com/huggingface/transformers/v5.17.0/src/transformers/cache_utils.py)

The reproducer subsequently uses greedy decoding. Once the first token diverges, the two execution paths receive different inputs, so any widening differences afterward cannot be attributed entirely to offloading. Given this design, further investigation should keep the input at each step identical across both paths, then compare states and outputs layer by layer to separate cache errors from divergence in the generation paths. Acceptance testing should also hold the model revision, precision, attention backend, and input length fixed, while retaining a baseline without offloading. Comparing only the final text can conflate numerical errors with changes in generation; retaining scores at each step can help narrow the investigation. These validation recommendations are based on the reproducer.

At the time of verification, the issue remained open, with no visible maintainer confirmation or link to a fix. Deployment teams can add output consistency checks across multiple steps to their offloading acceptance tests and track subsequent reproduction efforts and tests of fixes. The available evidence covers only the combination described above and does not establish that all models or all offloading modes are affected. [Issue status](https://github.com/huggingface/transformers/issues/48947)