AI 安全／多模態模型
Attacking VLMs by Backpropagating Through the Vision Encoder Alone Cuts H100 Generation Time From Over 20 Minutes to About 100 Seconds
New research avoids backpropagating through the full multimodal model, instead modifying the input image so that the vision encoder’s representation diverges from the original image or approaches a specified target. Untargeted attack success rates exceeded 93% across four open VLMs, but the evaluation relies on a single LLM judge and does not test black-box transferability.
A study submitted on August 19 narrows the computational scope of vision-language model attacks to the vision encoder. The targeted attack first obtains the vector of a target image, then updates the source image’s pixels over 50 iterations of I-FGSM to minimize the mean squared error between the adversarial image’s representation and the target vector. The untargeted version instead maximizes the distance between the adversarial image’s vector and that of the original image. Because gradients do not need to pass through the language model or the full generation path, the approach reduces memory and compute requirements.
The researchers tested Qwen2.5-VL-3B, Granite Vision 3.2 2B, FastVLM 7B, and Phi-3.5 Vision on 1,000 randomly selected ImageNet image pairs. With a perturbation bound of ε=0.05, untargeted attack success rates ranged from 93.0% to 99.8%. Targeted results varied more widely: Granite reached 41.15%, while Phi-3.5 achieved only 2.0%. When ε was increased to 0.20, Granite, FastVLM, and Qwen reached approximately 45.91%, 29.5%, and 25.0%, respectively. The results suggest that disrupting semantics is generally far easier than steering outputs toward specified content. They also show that the vulnerability of different vision encoders cannot be inferred directly from general image-understanding scores.
The system measurements are also notable. On an H100 80GB GPU, backpropagation through the full VLM consumed approximately 44% to 47% of GPU memory, while backpropagation through the encoder alone reduced usage to 13% to 16%. Qwen’s full-model path still had not completed successfully after more than 20 minutes, whereas the encoder-only path finished in about 100 seconds. For deployment engineers, image-integrity checks, input compression or randomization, and encoder-level stress testing should be included in VLM security assessments.
However, the paper uses Granite-4.0-micro to compare one-sentence image descriptions, so the reported success rates may incorporate judge bias. It also lacks a formal human evaluation of how perceptible the perturbations are at each ε value. The attack requires access to model gradients and feature-layer selection tailored to each architecture. The researchers have not yet demonstrated cross-model transferability, attacks against closed APIs, or robustness through real-world camera capture and re-encoding pipelines.