OpenVINO 2026.4 Brings Multi-Token Draft Decoding to CPUs and GPUs, Unifies NPU Performance Tracing

Intel’s open-source inference toolkit OpenVINO released version 2026.4 on September 16. Rather than simply expanding its model catalog, this release brings more generative AI optimizations to general-purpose CPUs, integrated GPUs, and NPUs. OpenVINO GenAI now provides Multi-Token Prediction (MTP) speculative decoding for Gemma 4, Qwen3.5, and Qwen3.6, allowing the draft model to propose multiple candidate tokens at once for verification by the main model. The Qwen GPU path also adds a preview implementation of DFlash, while EAGLE-3 vision-language models can use Top-K tree drafting instead of chain drafting, which retains only a single path. The [official release notes](https://docs.openvino.ai/2026/about-openvino/release-notes-openvino.html) claim improvements in throughput and latency but do not include comprehensive data spanning different hardware and context lengths.

Hardware coverage has also expanded significantly. Gemma-3n support has been added for CPUs, while the CPU and GPU paths now support models including Qwen3-VL-4B, Qwen3-ASR, Muse Glimmer 30B, Qwen3.8 27B, Gemma 4 12B, and DeepSeek OCR-2. NPU support now includes FLUX.2-Klein 4B and Kokoro-82M. Xe3 integrated GPUs receive long-context optimizations for Gemma 4, while NPUs can process prompts ranging from 1K to 32K tokens for MoE models such as Qwen3-30B-A3B and Gemma4-26B-A4B. Instrumentation and Tracing Technology has also been extended to NPUs, enabling VTune to monitor all three processor types through a single toolchain.

The server and API layers include practical migration considerations as well. Node.js adds an `ASRPipeline` with streaming and performance metrics. Model Server can unload idle models to reduce memory usage, although the feature remains in preview. Python’s `AsyncInferQueue` fixes a potential deadlock during callback cleanup. The C API adds non-variadic property-setting functions and fixes failures when importing blobs larger than 2 GB. The production package is available through `pip install openvino==2026.4.0`, while the [GitHub release page](https://github.com/openvinotoolkit/openvino/releases/tag/2026.4.0) confirms the version and signed commit.

Deployers should not interpret “support” as meaning the same level of maturity across all hardware. NPU dynamic shapes have currently been validated only for vision models with defined lower and upper bounds. DFlash, visual tokens, and idle model unloading remain in preview. Teams should benchmark TTFT, TPOT, peak memory usage, and acceptance rate using their own prompt lengths, quantization formats, and concurrency levels. They should also note that the project is considering making C++20 the default compilation standard in version 27.0, which may require early updates to older toolchains and custom plugins.