NVIDIA Publishes B200 Confidential Inference Benchmarks, Retaining Over 96% of Throughput on Specific Workloads

NVIDIA published B200 confidential computing inference benchmarks on September 22: running DeepSeek-R1 on eight GPUs with protection enabled retained 96.1% to 98.2% of the throughput measured without protection, while average latency per output token increased by 1.2% to 4.3%. The new material consists of benchmark results and deployment analysis; the two TensorRT LLM patches cited in the article were already merged in February. [Official benchmarks](https://developer.nvidia.com/blog/enabling-private-high-performance-production-ai-inference-with-nvidia-confidential-computing/)

The tests used NVFP4 weights, an FP8 KV cache, and eight-way tensor parallelism, with input and output lengths of 32K and 1K, respectively, and between one and sixteen concurrent requests. The environment combined Intel TDX with TensorRT LLM 1.3.0rc22. These conditions limit how the percentages can be interpreted; they cannot be used to extrapolate costs for other models, short requests, or services with high concurrency. [Test configuration](https://developer.nvidia.com/blog/enabling-private-high-performance-production-ai-inference-with-nvidia-confidential-computing/)

Performance tuning began with transfers between the CPU and GPU. The GPU cannot directly access protected virtual machine memory, so data must pass through encrypted staging buffers. As a result, commonly used pinned memory loses some of its advantages for asynchronous transfers. The relevant patch makes memory pinning conditional on whether confidential computing is enabled and adds an asynchronous worker to handle sampling data readbacks, reducing the likelihood that copy operations will block the main scheduler. [Memory and readback patch](https://github.com/NVIDIA/TensorRT-LLM/pull/11573)

Another change addresses the timing mechanism used by autotuning. CUDA events may provide unstable timing signals in this protected mode, affecting the selection of kernel implementations. An alternative path uses a small CUDA kernel to read the GPU’s global timer, then calculates elapsed time on the host. The timing backend is selected according to whether protection is enabled, with an environment variable override available to help engineers compare results. [Timing patch](https://github.com/NVIDIA/TensorRT-LLM/pull/11657)

For framework authors, these implementations suggest a troubleshooting sequence: first check whether data transfers are blocking scheduling, then examine whether timing tools are causing the tuner to select the wrong kernel, and finally compare the time spent waiting on communication between GPUs. This engineering advice is inferred from the patches. Focusing solely on GPU kernel speed may overlook changes in host-side behavior introduced by protected mode, so existing optimization settings also need to be revalidated.

Multi-GPU communication remains subject to limitations: this type of B200 configuration does not support NVLS multicast, and communication algorithms must be tuned to the topology and message size. These results come from vendor testing, and average token latency does not establish whether time to first token or tail latency meets service targets. Deployment evaluations should toggle protection while keeping the model, version, and workload identical, and separately measure end-to-end request latency. Similar throughput supports a performance assessment; it cannot substitute for security validation. [Scope and limitations](https://developer.nvidia.com/blog/enabling-private-high-performance-production-ai-inference-with-nvidia-confidential-computing/)