Transformers Community Reports SAM 3 Prompt Structure Difference; Text-Only Path May Omit Geometry CLS Token

A prompt structure consistency question has emerged in the Transformers port of SAM 3. An issue posted on October 14 reports that Sam3Model skips the geometry encoder when given only text, without geometric prompts such as boxes, so the subsequent DETR encoder and decoder receive only text tokens. The report used Transformers 5.19.0, and its author noted that the same path was present on the main branch at the time. Community issue

This path corresponds to a common use case. SAM 3 can locate objects in an image that match a text concept and produce instance and semantic segmentation results. The first example in Hugging Face’s official documentation passes an image and the text “ear” to the processor, then calls the model; users do not need to provide a bounding box. The report therefore concerns a basic inference workflow explicitly shown in the documentation. Official documentation

The author used model hooks to log module calls and prompt lengths: in the text-only case, the geometry encoder was not called and DETR received 32 tokens; with text and a box, it was called once and received 34. According to the Meta code and configuration cited in the issue, the reference implementation runs the geometry encoder even when there is no actual geometric prompt, appending a learnable CLS token. The same text-only case is therefore expected to have 33 tokens. The technical question is whether the prompt sequence at inference time may differ from the structure used to train the checkpoint. Reproduction code and reference path

On the same day, a participant said they had proposed a fix that removes the outer conditional and adds a unit test; another participant independently reproduced the issue on macOS. However, as of this review, the issue remains open, and the public discussion is still asking maintainers to confirm the expected behavior. It cannot yet be considered an officially completed fix, and no full segmentation evaluation is available to quantify any quality difference. Fix and reproduction discussion

For engineers building annotation or vision evaluation workflows, this is a reminder that reproducing results across frameworks requires checking more than weights and preprocessing. Next steps include tracking the maintainers’ determination, the version that includes any fix, and mask and score comparisons using fixed images, text, and post-processing thresholds. The sequence length evidence available so far is not enough to conclude that all text-only segmentation results will be inaccurate.