RLHF Illustrated — PPO Theory and Code

RLHF aligns LLMs with human preferences through three stages: SFT on demonstrations, reward model training on comparisons, PPO optimization.

PPO Core

PPO maintains a policy (LLM) and value network. It generates completions, scores them with the reward model, computes advantages, and updates the policy with clipped objectives to prevent destructive updates.

Code

ratios = exp(log_probs - old_log_probs) clipped = clamp(ratios, 0.8, 1.2) * advantages policy_loss = -min(ratios * advantages, clipped).mean()

Challenges

Reward hacking, KL regularization tuning, batch size sensitivity.