vLLM reports progress on hardware-agnostic layers, preserving compilation paths across accelerators

On September 22, vLLM developers reported progress on hardware-agnostic model layers and shared H100 test results, addressing compatibility concerns as the inference engine shifts toward hardware-specific optimizations. The initial foundational code was merged on August 13, but not all models have been ported. The [official announcement](https://pytorch.org/blog/hardware-agnostic-models-in-vllm/) and [merge record](https://github.com/vllm-project/vllm/pull/49458) document different milestones.

The issue is that “flat models,” which use hand-written fused operations tailored to specific models and accelerators, are gradually changing the design of shared layers. External accelerator plugins that rely on full computation graph compilation may consequently need to maintain their own model implementations. Less commonly used models also need a portable path. The [design proposal](https://github.com/vllm-project/vllm/issues/44219) therefore calls for retaining full-graph `torch.compile` support and interfaces for replacing operators, while isolating generic implementations from hardware-specific branches.

For external accelerators, full-graph compilation gives the backend access to the computational structure of the entire model, which it then converts into a form executable on the target hardware. Plugins can also replace individual layers to adjust memory layouts. Keeping these interfaces in the shared codebase helps reduce the maintenance burden on vendors as they track model updates and rewrite the same logic. [Architecture overview](https://pytorch.org/blog/hardware-agnostic-models-in-vllm/)

The first merged implementation integrates with the Transformers backend and adds components including RMSNorm and SiluAndMul. The layer resolver first looks for a hardware-agnostic version. If a layer has not yet been ported, it falls back to the existing vLLM implementation and logs the fallback. Successfully starting the service therefore does not mean the entire model is using the portable path. [Implementation record](https://github.com/vllm-project/vllm/pull/49458), [layer resolver documentation](https://docs.vllm.ai/en/stable/api/vllm/model_executor/models/transformers/layers/)

The configuration name also needs checking: the blog example uses `USE_HW_AGNOSTIC=1`, while the code documentation uses `VLLM_USE_HW_AGNOSTIC=1` alongside `--model-impl=transformers`. Based on the current layer resolver code, engineers should use the latter and inspect the logs to confirm which components were actually replaced. [Configuration and fallback logic](https://docs.vllm.ai/en/stable/api/vllm/model_executor/models/transformers/layers/)

The authors report that the geometric mean of total token throughput across three recent models on H100 was within 3.4% of the comparison path. The article compares the Transformers backend with this feature enabled and disabled; the results cannot be generalized into a performance guarantee for all hardware and models. [Test details](https://pytorch.org/blog/hardware-agnostic-models-in-vllm/)

The generic implementation of DeepSeek V4 remains under review. Teams maintaining support for alternative accelerators should next track layer coverage, plugin interfaces, and continuous integration tests. When adopting the feature, they should pin versions and separately validate numerical outputs, time to first token, and throughput. [Implementation under review](https://github.com/vllm-project/vllm/pull/45470)