GitHub Repo
Transformers Community Reproduces SAM3 Positional Encoding Precision Issue; bf16 Loading May Change Segmentation Results
A Transformers technical report on October 23 said that loading SAM3 with bf16 can reduce the precision of some positional coordinates, potentially causing objects near the score threshold to be omitted. Another participant reproduced the coordinate error with a small CPU model, but the official fix and impact on overall accuracy remain unconfirmed.
Hugging Face Transformers received a report of a SAM3 precision regression on October 23: after loading the model with dtype=torch.bfloat16, rounding errors may occur in the coordinates used by rotary positional embeddings (RoPE) in global attention layers. The reporter reproduced the issue in versions 5.20.2 and 5.17.0, and said the same behavior did not appear in versions 5.13.0 through 5.16.1. These are community test results and should not be treated as a complete list of affected versions. Technical report
The issue centers on how position_ids are created. Global attention uses scaled, fractional coordinates; if the tensor is created using the default bf16 dtype, values lose precision before being passed to the RoPE calculation. Another participant reproduced coordinate errors in global layers with a small, randomly initialized CPU model. Their analysis also noted that windowed layers use integer positions and therefore do not show the same numerical shift. Reproduction and analysis
This difference may carry through to the final output. In some images, the original report observed object scores crossing the filtering threshold and suggested running in fp32 or recreating the position coordinates in fp32 as temporary workarounds. There is not enough evidence to translate individual cases into an overall accuracy drop, nor to generalize the issue directly to other video models. Reported results
SAM3 can find objects matching a concept from text or image examples and return instance and semantic segmentation results. The official examples set score and mask thresholds during post-processing. The engineering concern, therefore, is that a continuous numerical deviation may become a discrete difference in the number of detected objects after thresholding. This is a technical interpretation based on the documented workflow and the report. Official documentation
For teams using SAM3 in annotation or image analysis pipelines, this incident is a reminder to include raw scores, samples near thresholds, and object counts in upgrade validation, rather than checking mask appearance alone. Tests should also hold prompts, resolution, and post-processing settings constant while comparing versions and dtypes. Key items to watch are whether position coordinates are explicitly kept in fp32, whether regression tests cover bf16 loading, and when a fix reaches an official release. As of the review, the issue remained open and listed no fix PR.