GitHub Repo
Transformers Community Proposes ZeRO-3 Loading Fix to Reduce Repeated Reads and Conversion of Full Weights Across Processes
Proposed on October 14, the Transformers patch centralizes ZeRO-3 checkpoint reads and weight conversion for non-quantized models on rank 0, reducing host memory pressure. The author included a two-process regression test and short training results on a large model, but the proposal has not been merged.
On October 14, the Hugging Face Transformers community proposed PR #49584, which addresses the repeated reading of full checkpoints and weight conversion by each distributed process when loading non-quantized models with DeepSpeed ZeRO-3. At the time of review, the proposal remained open, with no indication that it had passed review or been included in an official release.
The original report used Transformers 5.19.0.dev0 and a randomly initialized, two-layer Qwen3 MoE as a minimal reproduction: both ranks read three weight files and performed one conversion, but only rank 0 later copied checkpoint values into the model. These temporary tensors are not part of the ZeRO-3-sharded model parameters, so the loading stage may still incur duplicated allocations even after the model is sharded. Issue report and reproduction code
The official documentation explains that ZeRO-3 shards parameters, gradients, and optimizer states, and can also offload parameters to the CPU. This case is a reminder for deployment teams to assess host memory requirements during initialization separately from sharding strategies used during training. DeepSpeed ZeRO documentation
The proposal has rank 0 read and convert the weights, then broadcast the keys, shapes, and data types. Other processes create meta-tensor placeholders and use the existing parameter allocation mechanism. Buffers not sharded by ZeRO-3 are broadcast separately. Read and conversion errors are also propagated to other ranks to prevent processes from getting stuck waiting for communication. Patch design
The author reports that an eight-device GLM-5.3-Flash-BF16 test completed loading and 15 training steps with the patch, and includes tests for parameter and buffer consistency and error propagation. However, a memory guard script stopped the unpatched run early, so the two runs do not provide a full peak-memory comparison. Engineers should watch for maintainer review, validation across models, and a release, and measure loading costs in their own environments. Validation results and limitations