GitHub Repo
Transformers Community Reproduces Mamba Cache Continuation Anomaly; Multi-Token Inputs May Lose Recurrent State
A Transformers community report on October 31 says that reusing the cache with Mamba-1 for multiple new tokens may restart state computation from zero, producing incorrect predictions. Another participant reported reproducing the issue, but as of the time of review, no fix commit or maintainer confirmation was visible.
Transformers received a Mamba-1 correctness report on October 31: after the model processes a text segment and retains its cache, sending multiple new tokens at once may produce outputs that diverge from processing the full sequence. Another community participant said they reproduced the issue on the main branch that same day. As of the time of review, the issue remained open, with no linked fix PR. Public issue
This concerns the practical semantics of the cache interface. The official documentation says that cache_params contains the state-space model state and convolution state; passing it should let new input continue from the existing context without resending old tokens. Engineering teams implementing chunked prefill or conversation continuation through this interface should therefore verify that numerical results are consistent across chunk boundaries. Mamba documentation
The report used Transformers 5.20.2, CPU, and float32, without installing either of the two native acceleration packages. The author compared full-input processing, token-by-token continuation, and chunked continuation with mamba-130m-hf. The maximum logit difference was about 0.000263 for token-by-token continuation and about 115.897 for chunked continuation; the top-scoring token matched in 71.4% of cases for the latter. These figures come from a small reproduction case and cannot be directly translated into a drop in real-task accuracy. Reproduction code and results
The public code shows that the state-update branch is entered only when recurrent state exists and the sequence length is one. Multi-token input instead takes the full-scan path, which is called without the existing recurrent state. The reference scan implementation initializes state to zero, then overwrites the cache after processing. This supports the reported root cause, but does not mean that every backend has been tested. Mamba implementation
For deployment teams, a useful check is to run the same input through full, chunked, and token-by-token paths and compare logits, rather than merely checking that requests succeed. Follow-up should track whether a fix carries forward the existing state, adds regression tests, and addresses the accelerated kernels. Community members also suggested checking other models that share similar code, but the scope of affected models remains to be verified. Follow-up discussion