Post-Training & Alignment

A pre-trained model continues text; it does not answer questions, follow instructions or refuse harmful requests. Ask it "What is the capital of France?" and it may well continue with "What is the capital of Germany?", because that is the kind of text that follows a question in its training data. Post-training turns it into an assistant. It works in stages:

1. supervised fine-tuning: train on examples of good answers;
2. learning from preferences: train on pairs of answers where people said which is better;
3. reinforcement learning: let the model try, score its answers with a reward, and push it towards higher rewards.

This page derives the loss of each stage, from SFT (supervised fine-tuning) to DPO (Direct Preference Optimization), PPO (Proximal Policy Optimization) and GRPO (Group Relative Policy Optimization), with a small numeric example for each, and ends with the real recipe used to post-train OLMo 3 (section 10).

Supervised fine-tuning

Supervised fine-tuning, or SFT, continues training on conversations: a prompt and the answer we want. The loss is the next-token cross-entropy of section 2, with one change: only the answer is learned. The prompt tokens get the target ignore_index (-100 in PyTorch), so the model is not trained to produce user messages, and padding is ignored the same way. Conversations are serialized with a chat template that marks whose turn it is with special tokens, and the same template must be used at inference time.

Example. Prompt tokens [5, 6], answer tokens [7, 8], padded to length 6 with 0: the input is [5, 6, 7, 8, 0, 0] and the labels are [-100, -100, 7, 8, -100, -100].

Position $t$ predicts token $t + 1$, so position 1 (token 6) must predict 7, and position 2 (token 7) must predict 8. If the model gave those two tokens probabilities $0.5$ and $0.25$, the loss is $(-\log 0.5 - \log 0.25)/2 = 1.04$, averaged over the two answer tokens only. The prompt and the padding contribute nothing.

SFT needs few tokens compared with pre-training (OLMo 3's 7B Think model: 45.4B tokens, two epochs), and is often done with LoRA (Low-Rank Adaptation, section 11), which trains a small low-rank update instead of every weight. Domain adaptation (law, medicine, code) follows the same tools: continued pre-training on unlabeled text from the domain (Gururangan et al., 2020), then SFT on in-domain instructions.

Rewards from human preferences

People find it easier to compare two answers than to write the perfect one. RLHF (reinforcement learning from human feedback; Christiano et al., 2017; Ouyang et al., 2022) collects pairs $(y_w, y_l)$ where an annotator preferred $y_w$ ("winner") over $y_l$ ("loser") for the same prompt $x$, and trains a reward model $r_\theta(x, y)$, a transformer with a scalar head (a final layer that outputs one number), with the Bradley-Terry model of paired comparisons (Bradley and Terry, 1952):

where $\sigma$ is the sigmoid and $P(y_w \succ y_l)$ is the probability that $y_w$ beats $y_l$. Only the difference of rewards matters, so the reward model learns a ranking, not an absolute scale.

Example. The reward model gives the preferred answer $2$ and the other $0.5$: the difference is $1.5$, the probability that the preferred answer wins is $\sigma(1.5) = 0.82$, and the loss is $-\log 0.82 = 0.20$. If it had given them $0.5$ and $2$, the difference is $-1.5$ and the loss is $1.70$: the model is punished for getting the ranking wrong. Adding 100 to both rewards changes nothing.

InstructGPT then optimizes the policy $\pi$ (the language model) with PPO to maximize the reward, with a per-token KL penalty (Kullback–Leibler: a measure of how far two distributions are) that keeps it close to the SFT model, so that it cannot exploit flaws of the reward model.

PPO, clipped

Policy gradient methods push up the log-probability of actions (here, tokens) with positive advantage $A$ (better than expected) and push down the others. Taking several gradient steps on the same samples is efficient but can move the policy too far. PPO (Schulman et al., 2017) limits each update with the probability ratio between the new and the old policy, $\rho = \pi_\theta(a) / \pi_{\text{old}}(a)$:

where $\mathbb{E}$ is the average over the sampled tokens and $\epsilon$ (epsilon) is typically 0.2. The objective is maximized. Once the ratio has moved more than $\epsilon$ in the direction the advantage favours, the clipped term is constant and its gradient is zero: no incentive to move further. The $\min$ keeps the clipping one-sided, so that a step that made things worse is always corrected.

Example. $\epsilon = 0.2$, so $\tilde\rho$ stays in $[0.8, 1.2]$. Four tokens:

• $\rho = 1.5$, $A = +1$: $\rho A = 1.5$, $\tilde\rho A = 1.2$, the minimum is $1.2$ (the clipped term): gradient 0.
• $\rho = 0.5$, $A = +1$: $\rho A = 0.5$, $\tilde\rho A = 0.8$, the minimum is $0.5$: gradient $\rho A = 0.5$.
• $\rho = 0.5$, $A = -1$: $\rho A = -0.5$, $\tilde\rho A = -0.8$, the minimum is $-0.8$ (the clipped term): gradient 0.
• $\rho = 1.5$, $A = -1$: $\rho A = -1.5$, $\tilde\rho A = -1.2$, the minimum is $-1.5$: gradient $\rho A = -1.5$.

First case: a good token (positive advantage) that has already become 50% more likely: no more push. Second: a good token that became less likely: pushed up. Third: a bad token already made less likely: no more push down. Fourth: a bad token that became more likely: pushed down, strongly.

DPO: preferences without a reward model

DPO (Direct Preference Optimization; Rafailov et al., 2023) shows that the RLHF objective, with its KL penalty, has a closed-form optimal policy, and that the reward model can be written in terms of that policy. Substituting it into the Bradley-Terry loss removes both the reward model and the RL (reinforcement learning) loop:

where $\pi_{\text{ref}}$ is the frozen SFT model, $\log \pi(y \mid x)$ is the sum of the log-probabilities of the answer's tokens, and $\beta$ (beta) is a constant. The quantity $\beta \log \frac{\pi_\theta}{\pi_{\text{ref}}}$ plays the role of an implicit reward. DPO is supervised learning on pairs: no sampling, no value function, much simpler to run than PPO.

Example. At the start of training the policy is the reference, so both $\Delta$ are 0 and the loss is $-\log \sigma(0) = \log 2 = 0.693$.

After some training, the policy has made the preferred answer more likely than the reference did ($\Delta_w = +1$) and the other less likely ($\Delta_l = -1$). With $\beta = 0.1$ the loss is $-\log \sigma(0.1 \cdot 2) = 0.598$: lower, because the policy moved in the right direction.

GRPO and verifiable rewards

For math and code the reward does not need a model: a verifier checks whether the final answer is correct or the tests pass reinforcement learning with verifiable rewards (RLVR; Lambert et al., 2024). GRPO (Shao et al., 2024) also removes PPO's value network (a second model that estimates how good a state is). For each prompt it samples a group of $G$ answers, scores them, and uses as advantage each answer's reward relative to its group:

where $r_i$ is the reward of answer $i$ and $\mathrm{std}$ is the standard deviation (how spread out the rewards are). The same advantage is given to every token of answer $i$, and the policy is updated with PPO's clipped objective. DeepSeek-R1 (DeepSeek-AI, 2025) showed that this kind of RL on verifiable rewards makes long chains of reasoning emerge, the subject of section 14.

Example. A prompt with $G = 4$ answers, rewards $1$ for a correct answer and $0$ for a wrong one.

• Rewards $(1, 0, 0, 1)$: mean $0.5$, standard deviation $0.5$, advantages $(+1, -1, -1, +1)$: the correct answers are pushed up, the wrong ones down.
• Rewards $(1, 1, 1, 1)$: every advantage is 0. The group carries no information (all correct), so it gives no gradient.
• Rewards $(1, 1, 1, 0)$: mean $0.75$, standard deviation $0.43$, advantages $(0.58, 0.58, 0.58, -1.73)$: the single wrong answer is pushed down hard.

The real recipe: OLMo 3

The OLMo 3 report (Olmo Team, 2025, sections 4 and 5, appendix A.6) documents every stage of turning the base model into Olmo 3 Think:

• SFT on Dolci Think SFT, reasoning traces with a final answer: 45.4B tokens for the 7B, two epochs, sequences of 32,768 tokens packed into 1M-token batches.
• DPO with "delta learning" (Geng et al., 2025): the chosen answer only needs to be clearly better than the rejected one, not perfect, so pairs are built by contrasting good completions with deliberately worse ones. The loss is a length-normalized DPO, each log-ratio divided by the answer's length, with $\beta = 5$.
• RL with OlmoRL, GRPO plus several recent changes (from DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization), Yu et al., 2025, and Dr GRPO, Liu et al., 2025). Groups have 8 answers. Rewards come from verifiers for math (comparison with SymPy), code (test cases) and instruction following (constraint checks), and from an LM (language model) judge, Qwen3 32B, for chat. The changes are:
  • groups with identical rewards are dropped (they carry no gradient);
  • the loss is averaged over all tokens of the batch rather than per answer, to avoid a length bias;
  • there is no KL term;
  • the clipping is asymmetric, $1 - 0.2$ to $1 + 0.272$ ("clip-higher"), to allow larger updates on promising tokens;
  • advantages are not divided by the group's standard deviation, which would overweight prompts that are almost always solved or almost never.

Each change has a small effect that you can compute:

Example.

• No standard-deviation division. For rewards $(1, 1, 1, 0)$ the plain advantages are $(0.25, 0.25, 0.25, -0.75)$; divided by the standard deviation $0.43$ they become $(0.58, 0.58, 0.58, -1.73)$. For $(1, 0, 0, 1)$ they are $\pm 0.5$ plain or $\pm 1$ divided. A prompt that is almost always solved (or almost never) has a small spread, so dividing by it inflates the advantages: that prompt would weigh more than the others, which is what the change avoids.
• Clip-higher. A token with a positive advantage whose ratio is $1.25$. With symmetric clipping at $1.2$ its gradient is 0: it has already moved enough. With $1 + 0.272 = 1.272$ it is not yet clipped, and it can still go up.
• Token-level average. Two answers, the first with 1 token whose term is $1.0$, the second with 3 tokens whose terms are $0.2$ each. Averaged over answers: $(1.0 + 0.2) / 2 = 0.6$, so each token of the short answer counts three times as much as one of the long answer. Averaged over all 4 tokens: $(1.0 + 3 \cdot 0.2)/4 = 0.4$: every token counts the same.

References

• L. Ouyang et al., Training language models to follow instructions with human feedback, arXiv, 2022. InstructGPT: reward model loss and PPO with a per-token KL penalty.
• P. Christiano et al., Deep reinforcement learning from human preferences, arXiv, 2017.
• R. A. Bradley, M. E. Terry, Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons, Biometrika 39(3/4), 324–345, 1952.
• J. Schulman et al., Proximal Policy Optimization Algorithms, arXiv, 2017.
• R. Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model, arXiv, 2023.
• Z. Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, arXiv, 2024. GRPO.
• N. Lambert et al., Tülu 3: Pushing Frontiers in Open Language Model Post-Training, arXiv, 2024. RLVR.
• DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, arXiv, 2025.
• Q. Yu et al., DAPO: An Open-Source LLM Reinforcement Learning System at Scale, arXiv, 2025.
• Z. Liu et al., Understanding R1-Zero-Like Training: A Critical Perspective, arXiv, 2025. Dr GRPO.
• S. Geng et al., The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gains, arXiv, 2025.
• S. Gururangan et al., Don't Stop Pretraining: Adapt Language Models to Domains and Tasks, ACL, 2020.
• Olmo Team, Olmo 3, arXiv, 2025. Sections 4.2–4.4, Tables 47–49.