Transformers Community Reports Prompt Lookup Decoding Anomaly; Outputs May Change with CFG and SynthID

Transformers has a new inference correctness report: enabling prompt lookup decoding, a feature intended to speed up generation, may produce a different token sequence when combined with classifier-free guidance (CFG) or SynthID text watermarking. The reporter says the issue can be reproduced with version 5.21.0 and a specified commit on the main branch. The issue remains open, with no linked fix PR. Community report

Prompt lookup decoding uses repeated n-grams in the input to propose candidate tokens, which the model then verifies. It is suited to tasks such as summarization, where the output overlaps substantially with the input. The official documentation describes it as a variant of speculative decoding that can be used with greedy decoding and sampling. The key value of this optimization is reducing the cost of step-by-step inference while preserving generation semantics, so output differences under deterministic settings warrant investigation. Official inference documentation

The report runs a small randomly initialized Llama model on CPU with do_sample=False, comparing prompt lookup disabled against prompt_lookup_num_tokens=3. The two control runs—standard greedy decoding and repetition penalty—produce the same results. With guidance_scale=1.5, the outputs diverge starting at the second new token; in the SynthID run, they differ from the first new token. These are reproduction results for specific settings and cannot yet be generalized to all models or generation modes. Test code and outputs

The reporter’s root-cause analysis points to processor state: candidate filtering and verification make additional calls to logits processors, potentially advancing their internal state even for rejected draft positions. The official documentation confirms that the CFG processor uses a KV cache for the unconditional branch, while SynthID maintains token context and watermark-related state. This explains why both warrant further checks, but does not mean the maintainers have confirmed the root cause of this report. Processor documentation

For engineering teams, the report suggests that acceleration tests should also check output consistency. Teams using these combinations can compare tokens with prompt lookup enabled and disabled on their own models, and track whether compatibility restrictions or state rollback mechanisms are introduced. There are currently no data on watermark detection rates, formal model quality, or performance impact, so the report does not support claiming that SynthID has failed.