Writing
Posts
Notes on machine learning, mathematics, and the occasional engineering recipe, written while learning out loud.
- October 22, 202612 min read
Contrastive Perception Policy Optimization (CPPO)
How CPPO improves perception in vision-language models during reinforcement learning: it finds the tokens whose entropy rises most when the image is hidden, and trains them with a contrastive loss to stay stable under harmless image changes and to shift when the image's content is removed.
- October 14, 202611 min read
Perception-Aware Policy Optimization (PAPO)
How PAPO teaches multimodal models to look at the image while they learn to reason: it adds a loss that rewards responses whose probability drops when the image is masked, and keeps that loss from being gamed with two entropy terms.
- October 6, 202613 min read
Visually-Perceptive Policy Optimization (VPPO)
How VPPO makes reinforcement learning for vision-language models pay attention to the image: it measures how much each token depends on the picture, scales each response's advantage by that dependency, and updates only the most image-dependent tokens.
- September 28, 202612 min read
Adversarial Defense with Preference Optimization (AdPO)
How AdPO turns adversarial training into a preference optimization problem: a vision-language model learns, with DPO, to prefer its own description of a clean image over its description of an attacked one, and to describe the attacked image as if it were clean.
- September 20, 202611 min read
Adversarial Policy Optimization (APO)
How APO makes a reinforcement learning agent robust to high-dimensional and noisy observations: a perturber network learns small changes to the observation that change the agent's actions the most, and the policy network learns to ignore them, in a max-min game on top of PPO.
- September 12, 202611 min read
Group Sequence Policy Optimization (GSPO)
How GSPO fixes the instability of GRPO by moving the importance ratio from tokens to whole responses: one length-normalized sequence ratio, sequence-level clipping, and equal weights for every token, which stabilizes large Mixture-of-Experts training and powered Qwen3.
- September 4, 202610 min read
Clipped IS-weight Policy Optimization (CISPO)
How CISPO, from MiniMax-M1, keeps every token in the gradient: instead of clipping token updates like PPO and GRPO, which silently drops rare reasoning tokens such as 'Wait' and 'However', it clips the importance sampling weight and lets every token keep learning.
- August 27, 202613 min read
Value-Model-Based Augmented PPO (VAPO)
How VAPO makes PPO with a value model work for long chain-of-thought reasoning: pretraining the value model, decoupling and adapting GAE to response length, and borrowing DAPO's fixes for sparse rewards, to beat value-free methods like GRPO and DAPO.
- August 16, 202612 min read
Dr. GRPO: Group Relative Policy Optimization Done Right
How Dr. GRPO fixes two biases in GRPO: dividing by the response length makes long wrong answers cheap, and dividing by the group's standard deviation overweights questions that are too easy or too hard. Removing both recovers the unbiased policy gradient and stops responses from growing needlessly long.
- August 7, 202613 min read
Generative Adversarial Policy Optimization (GAPO)
How GAPO teaches language models to follow constraints: instead of pairing one prompt with a good and a bad response, it pairs one response with a matching and a violating prompt, and trains the policy against an encoder-only reward model in a GAN-style adversarial loop.
- July 29, 202616 min read
Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO)
How DAPO fixes the failures of GRPO in long chain-of-thought reasoning: Clip-Higher against entropy collapse, dynamic sampling against zero-gradient prompts, a token-level loss for long responses, and overlong reward shaping for truncated answers.
- July 21, 202614 min read
Simple Preference Optimization (SimPO)
How SimPO simplifies DPO: it replaces DPO's reward with the average log-likelihood that the model uses to generate text, which removes the reference model, and adds a target reward margin to separate preferred and rejected responses.
- June 20, 202616 min read
REINFORCE Leave-One-Out (RLOO)
How RLOO replaces PPO in RLHF with the simplest policy gradient: treating the whole response as one action, using REINFORCE, and comparing each response with the average reward of the other responses to the same prompt.
- June 10, 202618 min read
Group Relative Policy Optimization (GRPO)
How Group Relative Policy Optimization (GRPO) removes PPO's value model by comparing answers within a group, and why it works so well for training reasoning models. Part 6 of a series on policy optimization for LLMs.
- May 13, 202619 min read
Direct Preference Optimization (DPO)
How Direct Preference Optimization (DPO) aligns language models with human preferences without a reward model or reinforcement learning, by turning the RLHF objective into a simple classification loss. Part 5 of a series on policy optimization for LLMs.
- March 21, 202627 min read
Reinforcement Learning from Human Feedback (RLHF)
How Reinforcement Learning from Human Feedback (RLHF) trains language models from human preferences: collecting comparisons, training a reward model, and optimizing the model with PPO under a KL penalty. Part 4 of a series on policy optimization for LLMs.
- March 10, 202635 min read
Proximal Policy Optimization (PPO)
How Proximal Policy Optimization (PPO) achieves the stability of TRPO with a simple clipped objective and first-order optimization, explained step by step: the advantage function, the value model, Monte Carlo and temporal difference estimates, GAE, the probability ratio, and clipping. Part 3 of a series on policy optimization for LLMs.
- March 1, 202638 min read
Trust Region Policy Optimization (TRPO)
How Trust Region Policy Optimization (TRPO) keeps policy updates stable by limiting how much a model's behavior can change, explained step by step with KL divergence, importance sampling, the Fisher information matrix, and the natural gradient. Part 2 of a series on policy optimization for LLMs.
- February 14, 20268 min read
Reinforcement Learning for LLMs: SFT to Policy Gradients
Why supervised fine-tuning is not enough to align large language models, how reinforcement learning maps onto an LLM, and why the vanilla policy gradient is unstable. Part 1 of a series on policy optimization for LLMs.
- September 2, 20234 min read
Diving into Group Theory: Foundations for Equivariant Neural Networks
Dive into the foundational elements of Group Theory, from understanding the basic properties of groups to exploring the intricacies of subgroups and generators.
- February 22, 202310 min read
The Poisson Distribution: Your Key to Predicting the Unforeseeable
The blog post explores the Poisson distribution, a statistical distribution commonly used in various fields, including astronomy, to model random events. It explains the properties of the distribution, its real-world applications, and provides a step-by-step guide on how to visualize the Poisson distribution using Python. The post also discusses the issue of Poisson noise in astronomical observations and presents a practical example of how to calculate and deal with it using Python.
- February 21, 202315 min read
Dockerize Your Data Science Workflow: A Step-by-Step Guide to Setting Up Jupyter Lab on Your Private Linux Machine
Learn how to set up and deploy a Dockerized Jupyter Lab on a private Linux server, providing a secure environment for data science work. This step-by-step guide covers the installation and configuration of Docker, creation of a Jupyter Lab container, and tips for managing conda environments and accessing the Jupyter Lab interface.
- February 15, 202311 min read
Exploring Gaussian Distribution: Understanding the Math Behind the Bell-Shaped Curve
This blog post delves into the concept of Gaussian Distribution, a popular topic in mathematics and statistics. The post provides a comprehensive explanation of the distribution, including its properties, applications, and examples. Get to know the secrets behind the famous bell-shaped curve and how it is used in various fields. Whether you are a student, researcher, or data analyst, this post has everything you need to know about Gaussian Distribution.