Tag
Proximal Policy Optimization
Posts and notes tagged with Proximal Policy Optimization.
- March 10, 202635 min read
Proximal Policy Optimization (PPO)
How Proximal Policy Optimization (PPO) achieves the stability of TRPO with a simple clipped objective and first-order optimization, explained step by step: the advantage function, the value model, Monte Carlo and temporal difference estimates, GAE, the probability ratio, and clipping. Part 3 of a series on policy optimization for LLMs.