Adversarial Defense with Preference Optimization (AdPO)

Introduction

A large vision-language model (LVLM), such as LLaVA [1], answers questions about images. It has two main parts: an image encoder, usually the vision transformer of CLIP [2], which turns the image into features, and a language model, which reads those features together with the text prompt and writes the answer. The weak point is the image encoder. Because an image is made of continuous pixel values, an attacker can add noise to it that is too small for a person to see, but chosen so carefully that the encoder’s features change completely. The model then describes something that is not in the image, or writes whatever text the attacker wants, such as a wrong medical diagnosis or a phishing link.

The standard defense is adversarial training: generate such attacked images during training and teach the model to handle them. For vision-language models, the existing methods, such as TeCoA [3] and FARE [4], fine-tune only the CLIP image encoder, and they work on its features alone. They make the encoder more robust, but the model loses accuracy on clean images: it starts to miss fine details, or to describe things that are not there.

In 2025, a team from the University of Science and Technology of China and Alibaba’s Tongyi Lab proposed AdPO (Adversarial defense based on Preference Optimization) [5]. Its idea is that adversarial training and preference optimization want the same thing. Adversarial training wants the model to keep its correct behavior on clean images and to reject the misleading behavior caused by attacked images. Preference optimization, such as DPO [6], raises the probability of a preferred output and lowers the probability of a rejected one. So AdPO trains the model with DPO to prefer its description of a clean image over its description of the attacked image, and adds a second loss that makes the attacked image produce the clean description.

In this post, we first go through how the attacked images are made, then recall the DPO loss, and then build AdPO’s two losses and the training procedure, ending with the limitations. The DPO post covers the loss AdPO builds on.

Adversarial Images

Let $\varphi$ be the image encoder and $x_m$ a clean image. An adversarial image $x_{\text{adv}}$ is a copy of $x_m$ in which every pixel has been moved by at most a small amount $ϵ$. This is the ${\ell }_{\infty }$ constraint $\Vert x_{\text{adv}}-x_m{\Vert }_{\infty }\le ϵ$. AdPO uses $ϵ=2/255$ or $ϵ=4/255$, so no pixel moves by more than $2$ or $4$ steps on the usual $0$ to $255$ scale, far too little to see.

Within that limit, the attacker looks for the image whose features are as far as possible from the features of the clean image [5]:

$$x_{\text{adv}}=\arg \underset{\Vert x_{\text{adv}}-x_m{\Vert }_{\infty }\le ϵ}{\max }\Vert \varphi (x_{\text{adv}})-{\varphi }_{\text{org}}(x_m){\Vert }_2^2\text{\hspace{1em}(1)}$$

Here ${\varphi }_{\text{org}}$ is the original encoder, before any adversarial training, and $\varphi$ is the encoder being trained. If the features of the attacked image are far from the clean features, the language model, which only sees the features, receives a very different picture. This is an untargeted attack: it does not aim for a particular wrong output, it only pushes the features away. AdPO uses it during training because it needs no labels and no target text, and it reliably produces images the model gets wrong [5].

The maximization in Eq. (1) is solved with projected gradient descent (PGD) [7]. Starting from the clean image, each step moves every pixel by a small step size $\alpha$ in the direction that increases the feature distance, and then clips the result back into the allowed box around $x_m$:

$$x^{(k+1)}={\Pi }_{\Vert x-x_m{\Vert }_{\infty }\le ϵ}(x^{(k)}+\alpha \cdot \text{sign}({\nabla }_x\Vert \varphi (x^{(k)})-{\varphi }_{\text{org}}(x_m){\Vert }_2^2\left)\right)\text{\hspace{1em}(2)}$$

The $\text{sign}$ keeps only the direction of the gradient for each pixel, so every pixel moves by exactly $\alpha$, and the projection $\Pi$ cuts back any pixel that has moved more than $ϵ$ away from the clean image. AdPO runs $10$ steps of Eq. (2).

FARE uses the same distance as its defense: it trains the encoder to make $\Vert \varphi (x_{\text{adv}})-{\varphi }_{\text{org}}(x_m){\Vert }_2^2$ small, so that the attacked features stay close to the clean ones [4]. The attack in Eq. (1) maximizes this distance, and FARE minimizes it. This works only at the level of features. It never checks what the language model actually says, and a feature that the distance treats as unimportant may carry exactly the detail the answer needs. AdPO instead trains on the model’s output.

The Starting Point: DPO

AdPO is built on the DPO loss. Given an input $x$, a preferred response $y_w$, and a rejected response $y_l$, DPO trains the policy ${\pi }_{\theta }$ to raise the probability of $y_w$ and lower that of $y_l$, both measured relative to a frozen reference model ${\pi }_{\text{ref}}$, as in Eq. (9) of the DPO post:

$${\mathcal{L}}_{\text{DPO}}=-{\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}\left[\log \sigma \left(\beta \log \frac{{\pi }_{\theta }(y_w\mid x)}{{\pi }_{\text{ref}}(y_w\mid x)}-\beta \log \frac{{\pi }_{\theta }(y_l\mid x)}{{\pi }_{\text{ref}}(y_l\mid x)}\right)\right]\text{\hspace{1em}(3)}$$

Each log ratio, multiplied by $\beta$, is the implicit reward of a response, and the loss pushes the reward of $y_w$ above the reward of $y_l$. In standard DPO, both responses answer the same input $x$. AdPO changes this.

Preferred Image Optimization

For a vision-language model, the input has two parts, $x=\{x_m,x_{\text{text}}\}$: the image and the text prompt. AdPO uses a simple prompt, such as “What is the content of the image?”, and builds the preference pair from the model’s own answers [5]:

• The preferred response $y_w$ is what the model writes for the clean image $x_m$.
• The rejected response $y_l$ is what the model writes for the adversarial image $x_{\text{adv}}$.

The pairs are generated online, during training, from unlabeled images. No one writes captions, and the model is never trained on text written by someone else, so its own way of writing is kept. Since the attack moves the features, the description of the attacked image is usually wrong, and it serves as the rejected response.

The key change from Eq. (3) is that the two responses are now scored under different images. The preferred response is scored on the clean image, and the rejected response on the adversarial image [5]:

$${\mathcal{L}}_P=-\mathbb{E}\left[\log \sigma \left(\beta \log \frac{{\pi }_{\theta }(y_w\mid x_m,x_{\text{text}})}{{\pi }_{\text{ref}}(y_w\mid x_m,x_{\text{text}})}-\beta \log \frac{{\pi }_{\theta }(y_l\mid x_{\text{adv}},x_{\text{text}})}{{\pi }_{\text{ref}}(y_l\mid x_{\text{adv}},x_{\text{text}})}\right)\right]\text{\hspace{1em}(4)}$$

The reference model ${\pi }_{\text{ref}}$ is the model with the original encoder. Only the image encoder is trained. The language model stays frozen, so every change in the probabilities comes from a change in the image features. Read this way, the two halves of Eq. (4) do two different jobs. The first raises the probability of the correct description given the clean image, which protects the clean performance that TeCoA and FARE lose. The second lowers the probability of the wrong description given the attacked image, so the encoder learns to stop producing features that lead to it.

Adversarial Image Optimization

Eq. (4) keeps the clean performance, but on its own it does not make the model much more robust [5]. The reason is a known weakness of DPO on multimodal data, called unconditional preference [8]. The preferred and the rejected responses are both text, and they differ in their wording. DPO can lower the reward of $y_l$ by learning which kind of text to disfavor, without looking much at the image at all. The loss goes down, but the encoder has not learned to handle the attacked image.

So AdPO adds a second loss that ties the attacked image directly to the right answer. It trains the model to produce the clean description $y_w$ when it sees the adversarial image [5]:

$${\mathcal{L}}_A=-\sum _{t=1}^T\log {\pi }_{\theta }(y_w^t\mid x_{\text{adv}},x_{\text{text}},y_w^{0:t-1})\text{\hspace{1em}(5)}$$

This is the usual language-modeling loss, summed over the $T$ tokens of $y_w$: each token $y_w^t$ is predicted from the attacked image, the prompt, and the tokens before it. The loss is computed only on the answer, not on the prompt. Since the language model is frozen, the only way to make the clean description likely again is for the encoder to give the attacked image features that look like those of the clean image. Unlike FARE, though, “look like” is measured by what the language model says, not by the distance between the features.

The AdPO Objective

The full objective adds the two losses, with a weight $\lambda$ [5]:

$${\mathcal{L}}_{\text{AdPO}}={\mathcal{L}}_P+\lambda {\mathcal{L}}_A\text{\hspace{1em}(6)}$$

Eq. (4) keeps the clean behavior and Eq. (5) adds the robustness, and $\lambda$ trades one against the other. The authors found that a larger $\lambda$ makes the model more robust, and a smaller one keeps more of its clean performance. They use $\lambda =1$ and $\beta =0.1$.

Training the full LLaVA-7B in this way would be expensive, since every step runs the language model on two images and generates two responses. AdPO therefore trains on a small model, TinyLLaVA [9], which pairs the same CLIP ViT-L/14 encoder with OpenELM-450M [10], a language model of $450$ million parameters. Only the encoder is updated, so the trained encoder can then be plugged into larger models that use the same CLIP encoder, such as LLaVA-1.5-7B and OpenFlamingo-9B [11], without any further training. The images come from ImageNet, without their labels. Putting everything together, with the encoder trained on Eq. (6):

The attack in line 3 is made against the current encoder $\varphi$, so as the encoder becomes more robust, the attacks change with it.

Limitations of AdPO

AdPO’s cost is higher than that of FARE: each step generates two responses and runs a language model, even if a small one, where FARE only compares features. The preferred response is the model’s own description of the clean image, so any mistake the model already makes on clean images is learned as correct. The training attack is untargeted and limited to small ${\ell }_{\infty }$ noise, and the defense covers only the image, not attacks through the text prompt. The method also depends on the variant of preference optimization: the authors found that IPO, KTO, and Step-DPO work about as well as DPO, but SimPO does worse, possibly because it has no reference model [5].

AdPO showed that adversarial training can be written as a preference: the model’s answer on a clean image is preferred over its answer on an attacked one, and training on that preference, together with a loss that makes the attacked image give the clean answer, keeps the model accurate while making it robust.

References

1. Liu, H., Li, C., Li, Y., & Lee, Y. J. (2024). Improved Baselines with Visual Instruction Tuning. CVPR.
2. Radford, A., Kim, J. W., Hallacy, C., et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. ICML.
3. Mao, C., Geng, S., Yang, J., Wang, X., & Vondrick, C. (2023). Understanding Zero-Shot Adversarial Robustness for Large-Scale Models. ICLR.
4. Schlarmann, C., Singh, N. D., Croce, F., & Hein, M. (2024). Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models. ICML.
5. Liu, C., Gui, T., Liu, Y., & Xu, L. (2025). AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimization. arXiv:2504.01735.
6. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS.
7. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2018). Towards Deep Learning Models Resistant to Adversarial Attacks. ICLR.
8. Wang, F., Zhou, W., Huang, J. Y., Xu, N., Zhang, S., Poon, H., & Chen, M. (2024). mDPO: Conditional Preference Optimization for Multimodal Large Language Models. EMNLP.
9. Jia, J., Hu, Y., Weng, X., Shi, Y., Li, M., et al. (2024). TinyLLaVA Factory: A Modularized Codebase for Small-scale Large Multimodal Models. arXiv:2405.11788.
10. Mehta, S., Sekhavat, M. H., Cao, Q., et al. (2024). OpenELM: An Efficient Language Model Family with Open Training and Inference Framework. arXiv:2404.14619.
11. Awadalla, A., Gao, I., Gardner, J., et al. (2023). OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models. arXiv:2308.01390.