模型架構與推論
T-LoopFormer Lets Each Token Choose Its Own Recurrence Depth and Splits the KV Cache by Loop
The new architecture repeatedly applies a three-layer shared block, allowing a router to decide per token whether to exit early or continue latent reasoning. In experiments with roughly 1 billion parameters, the paper reports better latency and accuracy than a fixed-depth looped Transformer, though the approach has yet to be validated on frontier models or mainstream inference engines.
Released on September 14, [T-LoopFormer](https://arxiv.org/abs/2609.15160) brings dynamic computation from choosing a depth for an entire sequence down to the individual token level. A conventional looped Transformer repeatedly reuses the same set of weights, creating greater effective depth with fewer parameters. The problem is that easy and difficult tokens are typically forced to undergo the same number of iterations. T-LoopFormer’s lightweight router assigns a recurrence depth based on each token’s initial hidden state: easy tokens exit early, while more difficult ones continue through the shared block. Because the decision depends only on the state already available to that token, it preserves autoregressive causality.
Dynamic exits do not necessarily reduce real-world latency if a full KV cache is still retained at every depth. The researchers therefore maintain a separate cache for each recurrence step, keeping only the tokens that remain active at that depth and restricting attention to the corresponding cache. Training and inference also use gather–scatter operations to skip items that have already exited. On the FineWeb-Edu validation set, about 61% of tokens exited after two or three iterations, while roughly 15% continued for seven or eight. This suggests that a fixed depth both overcomputes easy tokens and may underestimate the needs of long-tail tokens.
The experiments trained NanoGPT-like decoders on 100 billion tokens from FineWeb-Edu-100B, comparing a non-recurrent 24-layer baseline with roughly 1 billion parameters against a model whose three-layer shared block could repeat up to eight times. Under the paper’s defined 24× FLOPs setting, T-LoopFormer averaged 44.20% across ten zero-shot tasks, slightly above the 24-layer baseline’s 43.80%. Its latency per output token was 0.197 milliseconds, compared with 0.441 milliseconds for LoopFormer. The authors have released an [MIT-licensed implementation](https://github.com/YuMingQian1234/T-LoopFormer) that remains close to NanoGPT, making its routing and cache logic easier to inspect.
These figures still come from a controlled research setting and do not represent real-world throughput with vLLM, SGLang, or long-batch serving workloads. When the maximum depth was reduced to 12× or 6× FLOPs, the model also underperformed comparable non-recurrent baselines. The next questions are how the approach behaves under larger-scale pretraining, what routing divergence costs look like across different hardware, and whether dynamic token sets can maintain high utilization in GPU kernels.