Ollama Pre-release Ports MLX Custom Kernels to CUDA; Limited Tests Show Nemotron Prompt Throughput More Than Doubling

Ollama released the v0.40.5-rc0 pre-release on October 14, incorporating MLX CUDA kernel ports merged the previous day. The update targets the execution path on NVIDIA GPUs: some custom operations that previously had only Metal implementations fell back to computation graph operations on CUDA. The new release adds corresponding kernels, allowing the affected models to use dedicated implementations. Release notes

The port covers mamba2_scan, depthwise_conv_silu, gated_delta, and related state and bias variants. The author also adjusted thread-boundary checks and scalar reads, and moved some cached values into per-warp registers. This trades repeated normalization calculations for avoiding shared memory and synchronization barriers. These details show that porting across backends still requires adapting to the GPU’s execution and memory models. Technical proposal

The author compared the main branch with the modified branch on Linux with an RTX 6000 Ada. For NVFP4, prompt processing for nemotron3:33b rose from 323.9 to 679.0 tokens per second, while generation speed increased from 112.3 to 149.6. For nemotron-3.5-lightning:30b-a3b, prompt processing rose from 313.3 to 695.6, and generation from 113.2 to 129.7. Prompt-stage gains were about 110% and 122%, respectively; generation-stage gains were about 33% and 15%. These are limited tests reported by the author and cannot be directly extrapolated to all models or online services. Benchmark data

The official MLX documentation already provides an interface for compiling CUDA custom kernels at runtime from source strings, with options to specify input layouts and dynamic shared memory. The technical significance of this change is applying that existing interface to specific model operations in Ollama, closing a backend performance gap. MLX documentation

Deployers should next track whether the changes make it into a stable release and retest first-token latency and sustained generation speed using their own prompt lengths, quantization settings, and GPUs. Numerical consistency also needs validation: the proposal notes that the reduction order differs between fused kernels and the computation graph, so bf16 results may differ by one least significant bit. A reviewer also explicitly said they had not verified CUDA execution locally. The current evidence therefore supports performance improvements in specific configurations, but does not yet guarantee results across devices. Validation notes