推論系統
Tenstorrent Plugs Into vLLM: A Separate Phased Scheduler and Data-Parallel Path for Mesh Accelerators
A new open-source plugin supports multiple language and vision models on Tenstorrent hardware through the same OpenAI-compatible API, without forking the vLLM core. It also exposes current limitations: prefill and decode cannot be mixed in the same step, while LoRA, speculative decoding, and multi-host serving are not yet supported.
The Tenstorrent team has released `vllm-tt-plugin`, which registers hardware, model architectures, workers, and schedulers through vLLM’s standard platform and general-purpose plugin interfaces. As long as the environment can import TT-Metal’s `ttnn`, vLLM will recognize the Tenstorrent platform; externally, it continues to use the existing OpenAI-compatible endpoints and request formats. The key design choice is to keep hardware-dependent code inside the Apache 2.0-licensed plugin, avoiding the long-term maintenance burden of a vLLM fork that falls behind upstream.
The plugin currently covers Llama, Qwen, Mistral, Gemma, DeepSeek V3, GPT-OSS, and selected vision-language architectures. Model implementations do not reside in the plugin itself; they are provided by TT-Metal as hand-written TTNN kernels. The plugin registers them based on the architecture names declared in each checkpoint. Consequently, a “recognized model” does not mean that every combination of device, precision, and context length has been validated. Deployers must still consult TT-Metal’s compatibility matrix for models and commit versions.
The main difference lies in scheduling. Upstream vLLM can combine chunked prefill and continuous decoding within the same step, whereas the Tenstorrent path currently designates each step exclusively as either prefill or decode and switches between the two. During steady-state decoding, a depth-two queue lets the host schedule the next step while asynchronously reading back results from the previous one. When the batch shape changes, prefill resumes, structured output is used, or sampling methods do not match, the pipeline must first be drained. Galaxy systems also use single-process lane data parallelism to prevent cross-process distribution and aggregation at every step from negating the benefits of parallelism.
The current installer builds vLLM 0.26.0 from source, and compatibility tags for older releases are not continuously backported. LoRA, speculative decoding, prompt logprobs, and multi-host serving are not yet supported, while prefix caching and asynchronous decoding require models to explicitly declare those capabilities. The next areas for engineering teams to watch are mixed prefill and decode, cross-host scaling, and real-world throughput data. This release demonstrates architectural feasibility, but provides no like-for-like cost or latency comparison with GPUs.