GitHub Repo
Transformers Community Reports Duplicate Work During ZeRO-3 Loading, with Each Process Potentially Reading and Converting the Full Checkpoint
A report in the October 11 Transformers development branch says that enabling ZeRO-3 may still leave each process loading the full checkpoint, adding resource pressure during startup. The report includes a two-device minimal reproduction and results from a partial patch, but there is no official fix or comprehensive memory measurement yet.
On October 11, the Transformers community reported an issue with distributed model loading: even when DeepSpeed ZeRO-3 shards model parameters, from_pretrained() may still cause every process to read the full checkpoint and repeat weight conversion. The report uses a specified commit of Transformers 5.19.0.dev0, PyTorch 2.10.0, and DeepSpeed 0.18.4. The issue remains open, with no related fix PR listed. Original report
This concerns two distinct memory lifecycles. ZeRO-3 partitions parameters, gradients, and optimizer states during training. The official documentation explains that parameters are automatically gathered and repartitioned during forward and backward passes. So the sharding of training state cannot be taken as a guarantee of reduced memory use during checkpoint loading. DeepSpeed documentation
The reporter created three safetensors shards from a randomly initialized, two-layer Qwen3 MoE model, then loaded them with two processes. No pretrained model download or training run was needed. Function counts showed that both ranks read all three weight files and performed one conversion each; only subsequent parameter writes were limited to rank 0. The report points to the non-quantized ZeRO-3 loading path and proposes having rank 0 read and convert the weights, then distribute the necessary information. Reproduction and code path analysis
For large-model fine-tuning, this report suggests that engineers should separately monitor host memory use and loading time during startup. This is a practical implication inferred from the loading mechanism; the report does not establish that memory use scales with the number of processes. The minimal reproduction completed normally, and operation counts do not equal physical disk traffic; memory mapping and page cache can both affect the results.
The reporter says a partial patch brings nonzero ranks’ read and conversion counts to zero, and checks consistency of parameters and persistent buffers. Maintainer review and broader validation are still needed. Follow-up areas include multi-node synchronization, buffer distribution, and whether released versions are affected. When investigating, also check the file format: Hugging Face documentation says DeepSpeed-native sharded checkpoints cannot be passed directly to from_pretrained(), so a format issue should not be confused with this duplicate-loading report. Checkpoint documentation