推論系統
Recirculation Feeds Deep Transformer States Back, Reducing Gemma 3 Perplexity by Up to 35%
Google DeepMind proposes mixing the deep residual stream from one token into shallow layers for subsequent tokens at inference time, creating cross-token recurrence in off-the-shelf Gemma 3 models. The method adds no model weights, but turns parallel prefill into sequential processing, and its real-world latency has yet to be fully measured.
Researchers from Google DeepMind and elsewhere have introduced Recirculation, an attempt to address a structural limitation of standard Transformers: their inability to continuously update internal state. Conventional models process each token from shallow to deep layers. Representations that have already undergone disambiguation or integration in deeper layers cannot flow back to shallow layers to influence early processing of the next token. With the new method, when the model processes the next token, a deep residual stream from the previous step is normalized by its L2 norm and mixed into a designated shallow layer using a coefficient α. This is not simply depth looping, in which the same set of blocks is rerun; instead, it creates recurrence across both model depth and token time.
The researchers froze the original model weights and searched for the source layer, destination layer, and mixing coefficient on pretrained Gemma 3 1B, 4B, and 12B models. Fixed Recirculation improved perplexity across model sizes on nine of ten language-modeling datasets. On PG-19, the 12B model reduced perplexity by 35.40%, although performance on Lambada worsened by 2.81%. The basic version improved six of eight single-token evaluations on Gemma 3 4B, generally by less than one percentage point. An adaptive version, which tunes the recirculation coefficient using a small amount of data, reduced overall perplexity by roughly 23% relative to the baseline and improved GSM8K accuracy by about 21% in relative terms.
Its engineering appeal is that it requires neither retraining nor additional parameters, and it barely changes the critical path of token-by-token decoding. The trade-off lies in prefill: the state must be updated sequentially by token, so the entire prompt cannot be processed in parallel as it can with a standard Transformer. This could substantially increase time to first token for long-context serving, and the paper does not yet provide end-to-end measurements of throughput, GPU memory usage, or performance across different accelerators. The results are also concentrated on Gemma 3, and some state-tracking tests deteriorated on the instruction-tuned 12B model. Key next steps are determining whether the results generalize to other model families, whether coefficients can be selected automatically without data, and whether runtimes can reduce the cost of sequential prefill through chunking or speculative techniques.