GitHub Repo
Transformers Community Proposes Question-Answering Model Loading Fix to Prevent Reinitialization of Underlying Pretrained Weights
A Transformers community report on October 26 said some question-answering models may reinitialize underlying weights when loading language model checkpoints because parameter name prefixes do not match. The proposed fix includes comparison results using a small Llama model and a GPU, but it has not been merged, and the full scope of impact remains unconfirmed.
On October 26, the Transformers community proposed a weight-loading fix for an issue in which underlying parameters may not be carried over when converting causal language model checkpoints to question-answering models. The original report used development version 5.21.0.dev0; loading Llama-2-7b through either LlamaForQuestionAnswering or the automatic model interface produced loading reports showing many unused underlying weights and expected parameters missing. Original report
The report says the checkpoint stores underlying parameters with the model.* prefix, while the question-answering model expects transformer.*, preventing the loader from mapping them. The official documentation explains that unused weights are discarded, while parameters not initialized from the checkpoint require further training. For fine-tuning workflows, this means engineers must distinguish normal initialization of a new question-answering output layer from the failure to load underlying pretrained parameters. If the latter goes unnoticed, an experiment may lose its intended pretrained starting point. Model loading documentation
The proposed fix adds dedicated name conversions for twelve question-answering model classes, mapping model. to transformer., and defines the reverse conversion to preserve compatibility with existing question-answering checkpoints. The author reports that, after the fix, a small Llama test had only the question-answering output layer missing, with all twenty underlying tensors matching; a half-precision check on an RTX A5000 also showed zero difference in underlying hidden states. These results were provided by the submitter and have not been independently verified. Proposed fix
As of the time of review, the proposal remains open, with CI awaiting maintainer approval. The author also disclosed using AI to help prepare the changes. The available evidence is insufficient to conclude that all stable releases or listed models are confirmed to be affected. Engineering teams can retain loading reports and use the documented output_loading_info option to inspect missing and unexpected parameters, then compare whether underlying tensors were carried over. Next, they should track maintainer review, regression tests across models, and the release of an official fix. Proposal status, loading inspection interface