AI 系統與 GPU
PTXBench Tests LLMs Writing H100/B200-Specific PTX Directly; No Model Reliably Matches State-of-the-Art Kernel Libraries
PTXBench checks whether generated kernels are correct, whether specified instructions actually execute, and how their performance compares with cuBLAS, cuDNN, and FlashInfer. The results show that generating instructions for new architectures does not necessarily translate into effective acceleration, with B200 attention kernels proving especially difficult.
The [PTXBench paper](https://arxiv.org/abs/2608.17379), released on August 18, evaluates LLM-generated kernels at the level of architecture-specific PTX rather than merely checking whether CUDA code compiles. The benchmark covers GEMM and attention workloads on the H100 Hopper and B200 Blackwell. Each trajectory may call the model up to eight times. Correct kernels are warmed up and then measured across 50 latency runs, with performance compared against cuBLAS 13.1, cuDNN 9.20, or FlashInfer 0.6.14.
The benchmark is specifically designed to prevent false positives in which a kernel only appears to use a new instruction. The system first disassembles the cubin and searches for the specified SASS instruction, then uses Nsight Compute to verify that the instruction was actually executed under the real workload. This excludes instructions hidden inside dummy kernels that are never launched. Even passing this check does not prove that the instruction performs useful computation or delivers a speedup.
None of the four evaluated models consistently matched state-of-the-art libraries across workloads. Claude Opus 4.8 reached 1.012× cuBLAS performance on B200 GEMM, while Gemini 3.1 Pro reached 0.892×. However, on two Blackwell attention backward-pass tasks, Gemini-generated CUDA/PTX kernels achieved only 0.133× and 0.015× the baseline, while the same model's Triton implementations reached 0.484× and 0.436×. Qwen3.6-27B produced only one correct B200 GEMM kernel, and it did not execute the selected Blackwell instruction.
The authors also created Fixit supervised training data from failed kernels, compiler or runtime feedback, and teacher-generated repairs, then fine-tuned Qwen3.6-27B. Performance improved on some tasks, but generalization depended heavily on data coverage, workload balance, and the quality of the reasoning teacher. The [open-source environment](https://github.com/zhang677/PTXBench) includes MiniPTXAgent, isolated evaluation images, and auditable trajectories. Engineering teams should next focus on reproduced results across GPUs and additional shapes, along with race checking. For now, eight rounds of search and a limited set of workloads are insufficient to represent production-grade kernel optimization.