Transformers Community Reports Distributed Evaluation Discrepancy; Duplicate Tail-Batch Samples May Affect Validation Loss

On October 26, the Transformers community raised an evaluation correctness report: when running multi-process evaluation with Trainer.evaluate, the final eval_loss may differ from the result of processing the same global batch in a single process if duplicate samples must be added at the end of the validation set. The author reported observing the discrepancy in versions 5.20.2 and 4.57.6; as of review, the issue remained open, with no maintainer confirmation or related fix. Report and reproduction code

The key is the order of loss aggregation and deduplication. According to the report’s analysis, each process first produces a scalar loss for the whole batch, then repeats it according to the batch size and passes it to gather_for_metrics. The official Accelerate documentation explains that this interface can remove duplicate items from the final batch in distributed environments. However, if duplicate samples have already been included in the batch loss or normalization denominator, removing output items cannot restore their original contribution. This is the author’s proposed explanation of the root cause and remains to be verified by the project. Accelerate API documentation

The report tests a randomly initialized, small Qwen3 model: three sequences contain four, two, and one valid prediction targets, respectively. Two processes produce a loss of about 1.687, compared with about 2.103 for one process. The author also tested a dataset whose size is evenly divisible by the global batch size; results matched or differed only by floating-point precision, suggesting that the tail batch and number of targets are conditions worth investigating. However, the test used separate CPU processes and a FileStore process group, rather than a typical multi-GPU job launched with Gloo or NCCL, so it does not establish that all deployments are affected. Test conditions

For engineering teams, this is a reminder to check data sharding when comparing validation loss across configurations. If teams choose checkpoints based on small loss differences, they can compare single-process and multi-process results using the same fixed model and global batch size, while recording tail-batch samples and valid token counts. The author proposes collecting each sample’s loss sum and target count, then aggregating after deduplication. Follow-up work will need to determine whether this preserves existing metric definitions and add regression tests for production backends. The author also disclosed using AI to help write and review the report. Proposal and disclosure