Tag
LLM Alignment
Posts and notes tagged with LLM Alignment.
- July 21, 202614 min read
Simple Preference Optimization (SimPO)
How SimPO simplifies DPO: it replaces DPO's reward with the average log-likelihood that the model uses to generate text, which removes the reference model, and adds a target reward margin to separate preferred and rejected responses.
- May 13, 202619 min read
Direct Preference Optimization (DPO)
How Direct Preference Optimization (DPO) aligns language models with human preferences without a reward model or reinforcement learning, by turning the RLHF objective into a simple classification loss. Part 5 of a series on policy optimization for LLMs.
- March 21, 202627 min read
Reinforcement Learning from Human Feedback (RLHF)
How Reinforcement Learning from Human Feedback (RLHF) trains language models from human preferences: collecting comparisons, training a reward model, and optimizing the model with PPO under a KL penalty. Part 4 of a series on policy optimization for LLMs.