GitHub Repo
Transformers Community Reports CLAP Preprocessing Anomaly That May Change Audio Retrieval Results by Batch Size
A Transformers community report on October 28 said that, in fusion mode, CLAP randomly marks a short audio clip as long, affecting embeddings and retrieval results. The author has proposed a fix and regression tests, but the change had not been merged as of the time of review, and the measurements still need independent verification.
On October 28, the Hugging Face Transformers community reported a CLAP audio preprocessing issue: when none of the clips in a batch exceeds the length limit, ClapFeatureExtractor in fusion mode still randomly selects one clip and sets is_longer to true. Single-item inference therefore always takes the fusion branch, while larger batches may select a different clip each time. Issue report
CLAP uses contrastive learning to connect audio and text, and can be used for audio retrieval and classification. The official documentation describes fusion as a multi-view approach for long audio: it combines three randomly cropped segments with a downsampled version of the full spectrogram. Short audio is padded and then duplicated four times. This makes how preprocessing selects the fusion branch an important factor in evaluating implementation consistency. Official documentation
The reporter tested text-to-audio search using laion/clap-htsat-fused, 957 AudioCaps clips, and ten-second crops. The author reported a Recall@1 (R@1) of 0.232 at batch size one and approximately 0.315 to 0.322 at batch size eight. After forcing the short-audio flag to false, the single-item result rose to 0.349. These are community-reported figures under specific test conditions and cannot yet be generalized to all CLAP workloads. Test setup and results
The author submitted PR #50101 the same day. It removes the code that randomly overwrites the flag, so the branch is determined by audio length, and adds regression tests for single-item, eight-item, and mixed-length batches. The author reported that 17 local tests passed and four integration tests failed, adding that the latter also fail on the unmodified main branch. As of the time of review, the PR remained open with no maintainer review outcome, so it should not be considered an official fix. Proposed fix
For engineers maintaining audio vector indexes, this report suggests recording batch size and preprocessing version as part of reproducibility records. Based on the mechanism described above, if the fix is adopted, existing short-audio embeddings may change. Before upgrading, compare vectors and search rankings for a fixed sample set, then assess whether the index needs to be rebuilt. Follow-up should track maintainer confirmation and inclusion in an official release.