Position-Debiased LLM-as-a-Judge

The problem

For open-ended answers, a strong model is often used as a judge: it sees a question and two answers, labelled A and B, and replies "A", "B" or "tie". But judges are known to be biased towards one position: some favour the first answer whatever it says. The standard fix is to ask twice, with the two answers swapped, and trust only a verdict that survives the swap.

judge(question, answer_a, answer_b) is given (a scripted fake in the tests).

The debiased verdict

debiased_verdict(judge, question, ans1, ans2):

1. call judge(question, ans1, ans2): $A$ means ans1;
2. call judge(question, ans2, ans1): now "A" means ans2, so translate the verdict back ("A" becomes "B" and the other way round; a tie stays a tie);
3. return "1" if both calls prefer ans1, "2" if both prefer ans2, and "tie" otherwise: they disagree, or either says "tie".

Example. A judge that always says "A", whatever it sees: first call prefers ans1 (it said A), second call also says "A", which translated back means ans2. They disagree: verdict "tie". The position bias cancels.

A judge that really prefers the longer answer: for ("long answer", "short") the first call says A and the second says B, which translates back to A (ans1 again): verdict "1". For ("s", "longer one") both calls prefer ans2: verdict "2".

The win rate and the consistency

• win_rate(judge, items): items is a list of (question, ans1, ans2). Return the score of model 1: each win counts 1, each tie 0.5, each loss 0, averaged over the items.
• position_consistency(judge, items): the fraction of items where the two calls agree after translation (both prefer the same answer, or both say tie). Zheng et al. (2023) report this "consistency" for their judges: a perfectly position-biased judge scores 0.

Example. Three items: ("long answer", "short"), ("s", "longer one"), and two identical answers. With the length-preferring judge the verdicts are "1", "2" and "tie" (equal lengths: it says A both times, which disagree after translation). Win rate: $(1 + 0 + 0.5)/3 = 0.5$. Consistency: 2 of 3 items agree, $0.67$. With the always-A judge: consistency $0$, win rate $0.5$ (all ties).

What to implement

debiased_verdict, win_rate and position_consistency.

Requirements

• The judge must be called exactly twice per item.