Transformers Proposes Restructuring Generation Model Exports So Prefill and Decoding Share a Computation Graph

Hugging Face Transformers contributor IlyasMoutawwakil proposed restructuring generation model exports on November 2, routing prompt prefill and subsequent step-by-step decoding through the same decoder computation graph. The design aims to reduce the number of graphs and integration code that a runtime must load, while avoiding separate exports of the same weights for the two stages. As of this review, the proposal has not been merged. Proposal

The key difference concerns input shape. A dynamically exported graph can receive the entire prompt at once, then process one new token at a time. With a statically exported graph, the query axis is fixed to a single token, so the prompt must also be fed in one token at a time. Sharing a computation graph therefore does not mean every deployment can prefill the prompt in one pass. If a model requires a separate prefill graph, the proposal will raise an explicit error rather than export another copy of the weights. Design and limitations

These changes affect deployment artifacts directly. Optimum Intel's official documentation explains that an OpenVINO export produces an XML file describing the topology and a BIN file containing the weights; by default, stateful models also keep the KV cache inside the model. This shows why graph partitioning and cache boundaries affect how a runtime loads, invokes, and manages a model. Successful file conversion alone is not enough. Official export documentation

The proposal also defaults to outputting logits only for the final position, avoiding vocabulary scores for every position in a long prompt. Full outputs can still be requested for assisted decoding. Quantization calibration covers both prefill and decoding because their activation ranges differ. For deployment engineers, a key question to validate is whether output consistency and quantization quality can be maintained after reducing duplicate artifacts. Technical details

Integration hurdles remain: this PR is stacked on top of a multimodal runtime branch, and the proposal explicitly states that runtime() will reject generation export artifacts until generation runtime support is added. Next, watch whether the dependent changes are merged, whether prefill and decoding across different models match the original runtime's results, and what the actual loading costs and latency are. For now, the design goals should not be treated as achieved speedups. Integration status