Google Tests Double-Blind Model Evaluation in an H100 Secure Enclave, Keeping Model Weights and Confidential Test Sets Hidden from Each Other

Google DeepMind, MLCommons, OpenMined, AVERI, and Singapore’s AI Safety Institute have completed a proof of concept for double-blind model evaluation: the model provider does not receive the test prompts, while the evaluator cannot access the model weights or inference code. The experiment paired Gemini 2.5 Flash Lite with held-out AILuminate safety prompts that had never been processed by the model, and also tested harmful-content elicitation prompts tailored to the Singaporean context.

The parties first reviewed the interfaces and execution policies, then streamed encrypted assets over mTLS to a Google Cloud A3 Confidential VM. The host used Intel TDX to encrypt system memory, an NVIDIA H100 Confidential GPU to protect GPU memory, and OpenMined PySyft to manage job submission, approval, and execution. Remote attestation signed measurements of the firmware, kernel, container, and application images. The weights and test set were decrypted only after both parties confirmed that the execution environment matched the approved configuration. Once the evaluation was complete, only restricted aggregate metrics were released, and the ephemeral keys and execution environment were destroyed along with the enclave.

The technical significance lies not in another Gemini score, but in enabling closed models to undergo previously unseen, highly sensitive tests covering cybersecurity, CBRNE, self-harm, violence, and other risks—without allowing the test set to enter the provider’s logs or subsequent training data. The same approach could also let banks, governments, and research institutions test external models without relinquishing data sovereignty.

A secure enclave is not entirely “trustless.” Root keys from CPU and GPU vendors, along with proprietary low-level firmware, remain part of the trust foundation. A vulnerability in any layer of the trusted computing base could compromise confidentiality or integrity. The technical report also notes that the main bottleneck has shifted from the reported compute overhead of less than 5% to legal agreements, per-party code approval, and manual coordination. Engineering teams should next focus on reproducible builds, limits on output leakage, attestation revocation mechanisms, and whether the workflow can scale to evaluations requiring activation probing or per-token log-likelihoods rather than only black-box outputs.