Process vs Outcome Reward Scoring

The problem

A reasoning solution is a list of steps. An outcome reward model judges only the end result. A PRM (process reward model) judges every step: it gives each step the probability that the step is correct. To rank whole solutions, Lightman et al. (2023) score a solution as the probability that every step is correct, the product of its step probabilities. A solution with one wrong step and a lucky final answer gets a low score, as it should.

Three ways to score

For the step probabilities $p_1, \dots, p_n$ of one solution:

• "product": $p_1 \cdot p_2 \cdots p_n$, the PRM score of Lightman et al.;
• "min": the weakest step, $\min_i p_i$, a common alternative;
• "last": only the last step's probability, $p_n$: an outcome-style score, which sees only how the solution ended.

Example. Solution A has steps $(0.9, 0.9, 0.9, 0.9)$ and solution B has $(0.99, 0.99, 0.3, 0.99)$.

method	A	B	best
"product"	$0.656$	$0.291$	A
"min"	$0.9$	$0.3$	A
"last"	$0.9$	$0.99$	B

The outcome-style score prefers B, whose last step looks perfect even though one step in the middle was probably wrong. The product and the minimum see the weak step.

Picking the best solution

pick_best(solutions, method): solutions is a list of step-probability arrays; return the index of the best one under method (the first one on a tie). In the example, pick_best([A, B], "product") is 0 and pick_best([A, B], "last") is 1.

Long products

A product of many probabilities shrinks fast: 1,000 steps of $0.99$ give $4 \times 10^{-5}$, and 2,000 steps of $0.5$ give a number below the smallest a float64 can store: exactly 0. In practice such products are computed as a sum of logarithms, which stays well-behaved ($-1386$ for the second case). Both np.prod and np.exp(np.sum(np.log(p))) are fine for the tests here, but remember why log-space exists.

What to implement

• solution_score(step_probs, method): step_probs is a 1-D array, one probability per step; method is "product", "min" or "last". Raise ValueError for any other method.
• pick_best(solutions, method): the index of the best solution.