Tag
Advantage Function
Posts and notes tagged with Advantage Function.
- March 10, 202635 min read
Proximal Policy Optimization (PPO)
How Proximal Policy Optimization (PPO) achieves the stability of TRPO with a simple clipped objective and first-order optimization, explained step by step: the advantage function, the value model, Monte Carlo and temporal difference estimates, GAE, the probability ratio, and clipping. Part 3 of a series on policy optimization for LLMs.
- February 14, 20268 min read
Reinforcement Learning for LLMs: SFT to Policy Gradients
Why supervised fine-tuning is not enough to align large language models, how reinforcement learning maps onto an LLM, and why the vanilla policy gradient is unstable. Part 1 of a series on policy optimization for LLMs.