Tag
DPO
Posts and notes tagged with DPO.
- September 30, 202612 min read
Adversarial Defense with Preference Optimization (AdPO)
How AdPO turns adversarial training into a preference optimization problem: a vision-language model learns, with DPO, to prefer its own description of a clean image over its description of an attacked one, and to describe the attacked image as if it were clean.
- July 21, 202614 min read
Simple Preference Optimization (SimPO)
How SimPO simplifies DPO: it replaces DPO's reward with the average log-likelihood that the model uses to generate text, which removes the reference model, and adds a target reward margin to separate preferred and rejected responses.
- May 13, 202619 min read
Direct Preference Optimization (DPO)
How Direct Preference Optimization (DPO) aligns language models with human preferences without a reward model or reinforcement learning, by turning the RLHF objective into a simple classification loss. Part 5 of a series on policy optimization for LLMs.