GitHub Repo
Transformers Reports Remote Code Cache Race That May Expose Incomplete Files
A November 16 Transformers issue report says processes sharing a cache while loading remote model code may import Python modules before copying is complete. The report includes an offline reproduction and suggests atomic replacement, but maintainers have not confirmed the issue or released a fix.
On November 16, a member of the Transformers community reported a race condition in model loading: when multiple processes simultaneously load a model that requires trust_remote_code and share a module cache, some processes may read Python files before they have finished being written, occasionally raising NameError or SyntaxError. The report targets 5.24.0.dev0 on the main branch; the affected stable versions have yet to be confirmed. Issue report
According to the reporter’s analysis, get_cached_module_file first checks whether the destination file exists, then copies remote code to HF_MODULES_CACHE with shutil.copyfile. After the first process creates the file, another process may pass the existence check, skip the copy, and begin importing it immediately. The filename is visible at that point, but its contents may still be incomplete. This creates a failure window in multi-rank startup flows that share a cache. Technical details
The report includes an offline, CPU-based reproduction that deliberately slows the copy to widen the race window. It shows that a second call can return when only about half the file has been written. This provides evidence for a reproducible mechanism, but is not enough to estimate how often the problem occurs in real-world clusters, and does not cover different file systems. Reproduction case
This loading path matters for custom models. Hugging Face’s official documentation explains that custom models can upload their model and configuration code to the Hub and then be loaded with trust_remote_code=True. The engineering takeaway is that multi-process deployments need to check that the dynamic code cache is fully populated, in addition to preparing model weights. This is a deployment implication drawn from the reported mechanism. Custom models documentation
The reporter suggests writing to a temporary file in the same directory, then replacing the destination with os.replace. Python’s documentation states that the operation is atomic when successful, but may fail across file systems. Follow-up should track whether upstream proposes a fix, adds concurrent-loading tests, and validates behavior on shared storage. At the time of review, the issue remained open, and the page listed no related fix PR. Python documentation, issue status