Best-of-N and Weighted Voting

The problem

You sampled $N$ solutions to a problem. For each you have its final answer and a score from a verifier (a model or a program that judges solutions: higher is better). There are two natural ways to pick one answer, and a way to measure how useful sampling was.

• Best-of-N: the answer of the single highest-scoring solution.
• Weighted voting: add up the scores of all the solutions that reach the same answer, and return the answer with the largest total. It combines the agreement of voting with the verifier's judgement.
• Coverage: whether at least one of the $N$ answers is correct. It is the upper limit of what any picking rule could achieve.

Example. Four solutions with answers = ["12", "12", "15", "8"] and scores = [0.5, 0.6, 0.9, 0.2].

• Best-of-N: the highest score is $0.9$, so the answer is "15".
• Weighted voting: "12" totals $0.5 + 0.6 = 1.1$, "15" totals $0.9$, "8" totals $0.2$: the answer is "12".
• Coverage: if the correct answer is "8", one solution found it: True, although neither rule would pick it. If the correct answer is "9": False.

The two rules disagree here: weighted voting prefers two moderately good solutions that agree over a single confident one.

What to implement

With answers a list of strings and scores a NumPy array of the same length:

• best_of_n(answers, scores): the best-of-N answer. On a tie, the first solution wins.
• weighted_vote(answers, scores): the weighted-voting answer. On a tie, the answer that appeared first wins.
• coverage(answers, correct): True if at least one of the answers equals correct.

Requirements

• np.argmax returns the first maximum.
• Keep the order of first appearance for the tie rule (a dict does).

Hint

For the weighted vote, a dict from answer to total score, filled while you walk through the two lists, keeps the order of first appearance. max(totals, key=totals.get) returns the first key with the largest total.