The problem
A single chain of thought from a model can go wrong at any step. But if you sample several chains, the correct ones tend to end with the same answer, while the wrong ones disagree with each other. Self-consistency exploits this: sample chains, extract the final answer of each, and return the most frequent one. It needs no extra model, only a way to read the answer out of a text. You will write both pieces.
Part 1: reading the answer
extract_answer(text) returns the final answer of one chain as a string, or None if there is none. Two formats are understood:
- Boxed: the answer is written
\boxed{...}. Take the last\boxed{...}of the text and return what is inside (no nested braces needed). - GSM8K (a benchmark of grade-school math problems), where the answer follows
####. If there is no\boxed{}, take the text after the last####.
In both cases strip the spaces around the answer.
Example.
"so \boxed{ 41 } no wait \boxed{42}."returns"42": the last box, stripped."18 + 2 #### 18"returns"18"."no answer here"returnsNone.
Part 2: the vote
majority_vote(texts) extracts the answer of every text, ignores the Nones, and returns the most frequent answer, or None if no text had one. On a tie, return the answer that appeared first among the tied ones.
Example.
- Six chains ending with
\boxed{42},\boxed{42},\boxed{41},#### 42, no answer,\boxed{7}: the answers are 42, 42, 41, 42, 7, so 42 wins with 3 votes.- Four chains with 3, 5, 5, 3: a tie, 2 votes each; 3 appeared first, so the answer is 3.
Answers are compared as strings after stripping: " 42" and "42" are the same answer.
What to implement
extract_answer(text): as in part 1.majority_vote(texts): as in part 2.
Requirements
- Use only the standard library (
re,collections).
Hint
re.findall(r"\\boxed\{([^{}]*)\}", text) finds every boxed answer; take the last one. collections.Counter counts the answers, and the list of answers (in order) gives the tie rule: pick the first answer whose count is the maximum.