Tag
Preference Optimization
Posts and notes tagged with Preference Optimization.
- July 21, 202614 min read
Simple Preference Optimization (SimPO)
How SimPO simplifies DPO: it replaces DPO's reward with the average log-likelihood that the model uses to generate text, which removes the reference model, and adds a target reward margin to separate preferred and rejected responses.
- May 13, 202619 min read
Direct Preference Optimization (DPO)
How Direct Preference Optimization (DPO) aligns language models with human preferences without a reward model or reinforcement learning, by turning the RLHF objective into a simple classification loss. Part 5 of a series on policy optimization for LLMs.