GitHub Repo
Transformers Community Reports Cross-Device SynthID Watermark Differences; the Same Seed May Not Recreate the Detection Signal
A Transformers community report on October 11 found that CPU and Apple MPS can build different SynthID text watermark sampling tables even with the same key and seed. Small-scale tests suggest cross-device scoring may fall to the unwatermarked baseline, but the full detector and newer package versions remain unverified.
On October 11, the Transformers community reported a cross-device reproducibility issue with SynthID-Text: the same key and random seed may correspond to different watermark signals on CPU and Apple MPS. The report includes a minimal reproduction using Transformers 4.46.3, PyTorch 2.8.0, and an Apple M3 Pro. The issue remains open, with no related fix proposal yet. Community report
SynthID’s implementation computes indices from the token sequence and key, then retrieves g-values from a precomputed sampling table to adjust token probabilities during generation. This means the generation and scoring sides must reconstruct the same table; storing only the key and seed may not be enough to ensure consistency across hardware. Transformers technical documentation
In the report, the CPU and MPS sampling tables matched at a rate of about 50.19%, while g-values for the same tokens matched at about 49%. The author also tested Qwen2.5-1.5B-Instruct on ten watermark outputs of 400 tokens each: the mean g-score was 0.59 to 0.66 when scored on MPS, but fell to 0.49 to 0.50 when scored on CPU, close to the unwatermarked control group. However, this was a signal-statistics test; it did not test the Bayesian detector or CUDA, so it cannot directly establish the false-negative rate of a full detection system. Tests and limitations
PyTorch’s official documentation already notes that even with the same seed, CPU and GPU results are not guaranteed to be reproducible. From an engineering standpoint, this suggests watermark deployments should include sampling-table consistency in generation-to-detection compatibility checks and record device and package version information. PyTorch reproducibility notes
The author recommends building the table on CPU first and then moving it to the target device, but that would change tables already used in non-CPU environments, requiring a way to preserve compatibility with historical text. The author only notes that v5.19.0 still uses the same initialization method; that version has not been tested. The next things to watch are cross-device regression tests, a mechanism for selecting legacy tables, and validation results from the full detector. Proposed fix direction