RWKV7-G1j Scales an Attention-Free Recurrent Model to 13.3B While Maintaining Fixed State During Long-Sequence Decoding

The RWKV team has released `RWKV7-G1j-13.3B-20260831`, scaling the attention-free recurrent architecture of RWKV-7 “Goose” to 13.27 billion parameters. This Apache 2.0 base model has 61 layers, a hidden size of 4,096, a vocabulary of 65,536, and a 16,384-token training context. Its weights are provided in BF16, and its training data combines web, code, synthetic, instruction, dialogue, and reasoning content.

The key technical point is not simply the higher parameter count, but that the decoding state does not grow with sequence length as a Transformer’s KV cache does. RWKV-7 updates a fixed-size state using a generalized delta rule with vector-valued gating, in-context learning rates, and a relaxed value-replacement rule. As a result, the theoretical state memory and compute cost per generated token remain constant, while training can still be parallelized. This is attractive for long-running local agents, streaming workloads, and memory-constrained deployments. However, “fixed state” does not mean fixed total GPU memory usage: weights, batch size, data type, and execution backend still dominate actual requirements.

This release lowers the barrier to experimentation. The model repository includes remote code for Transformers 5.15+, a fast tokenizer written in Rust, chat and tool-calling templates, and interfaces supporting `SFTTrainer`, PEFT LoRA, and recurrent-state continuation. It also provides a PyTorch fallback and optional TileLang acceleration. The project warns that the full TileLang path may alter floating-point operation ordering, so numerical consistency must be validated for each GPU, dtype, shape, and checkpoint. Loading through Transformers still requires `trust_remote_code=True`; production deployments should audit the code and pin the revision.

The limitations are equally clear. This is a base model: the chat template defines only the prompt format and does not indicate that the model has undergone instruction alignment. The model card does not publish standard capability, safety, bias, or long-context evaluations for this 13.3B checkpoint. Speculative decoding that requires recurrent-state rollback must also preserve additional snapshots. The next things to watch are independent benchmarks, real-world long-sequence throughput, and whether general-purpose serving systems such as vLLM and SGLang can reliably realize the memory advantages of fixed state.