What is DPO (Direct Preference Optimization)?

DPO is a way to align a model with preferences without the full RLHF machinery. Classic RLHF trains a reward model on human comparisons, then uses reinforcement learning such as PPO to maximize that reward while a KL penalty keeps the policy close to the starting model. DPO shows that the same goal can be written as a simple loss on preference pairs.

Each training example is a prompt with a chosen and a rejected answer. The loss pushes up the log-probability of the chosen answer relative to the rejected one, with both measured against a frozen reference model, usually the SFT model. A beta setting controls the strength of that anchoring: higher beta keeps the model closer to the reference. In practice it trains like supervised fine-tuning, with one model plus a reference, which is cheaper and much easier to tune than PPO.

The quality of the pairs drives the result. I want rejected answers that are plausible but worse in a specific way, consistent labeling guidelines, and coverage of the behaviors I care about. I also watch for side effects such as longer answers, since length often correlates with preference labels.