推論系統
Cohere Packs the Entire Decoding Step Into a Single CUDA Megakernel, Boosting End-to-End H100 Throughput by 25% to 41%
Cohere has open-sourced a dedicated inference engine for North Mini Code that replaces per-operator launches and full-GPU synchronization with a persistent megakernel. Official benchmarks show higher end-to-end decoding throughput than vLLM on a single H100, though support is currently limited to specific models, hardware, and small batch sizes.
Cohere has released an inference server for North Mini Code. Rather than further tuning individual GEMM or attention kernels, its core approach uses a persistent CUDA megakernel to execute the entire decoding forward pass. Traditional engines sequentially launch kernels for RMSNorm, QKV, attention, MoE, and other operations, with each boundary requiring synchronization across the entire GPU. For memory-bandwidth- and latency-bound workloads such as low-batch autoregressive decoding, launch gaps and workloads too small to fill an entire wave leave SMs idle.
The new engine reserves one thread block on each H100 SM. The host builds a fine-grained task table in advance, assigning GEMM tiles and attention splits to available SMs. Data dependencies are represented using counters in global memory, allowing consumers to execute as soon as their own inputs are ready instead of waiting for the entire grid. North Mini Code’s attention and MoE paths can also run concurrently from the same normalized input, while idle SMs can take over tail-end work from the other path. The system retains continuous batching, a paged KV cache, prefix caching, preemption, tool calling, and an OpenAI-compatible API.
In Cohere’s official tests on a single H100 using BF16 and batch size 1, the megakernel reached 292 tokens per second—about 62% of the estimated bandwidth ceiling and 1.58× faster than vLLM 0.24. After adding real-world prompts, prefill, and dynamic batching, the average decoding-throughput gain across five workloads narrowed to between 1.25× and 1.41×. The results show that service-layer scheduling and synchronization design can still deliver substantial gains without relying on quantization.
The limitations are also clear: the engine currently supports only H100 SM90a, CUDA 13, BF16, and batch sizes from 1 to 8, while its scheduler is tailored specifically to North Mini Code. Prefill still uses PyTorch kernels and pauses decode, and sampling options remain incomplete. Engineering teams should next watch whether the approach can generalize to other MoE models, support mixed prefill and decode workloads, and maintain its advantage across different request distributions and at higher concurrency. The existing figures come primarily from the publisher, and independent reproductions are not yet available.