Running Generated Code Against Tests

The problem

A model generated several candidate implementations of a function, each as source code. To tell which are right, run them against test cases, and rank them by how many tests they pass. Along the way, learn from every failure: a wrong answer, an exception, a program that does not even load. The error messages are what a repair step would give back to the model.

One candidate

run_candidate(source, func_name, tests): tests is a list of (args, expected) pairs, where args is a tuple. Execute source with exec in a fresh namespace, get func_name from it, and call it on every test. Return a dict:

• "passed": the number of tests whose result == expected;
• "total": the number of tests;
• "errors": a list of short strings, one per failing test: "test <i>: wrong answer <result!r>" or "test <i>: <ExceptionName>: <message>";
• if the code does not load, or does not define the function, a single error "load: <ExceptionName>: <message>", with passed = 0.

Example. func_name = "add", tests add(1, 2) = 3, add(0, 0) = 0, add(-1, 1) = 0.

source	result
return a + b	passed: 3, no errors
return a - b	passed: 1; errors test 0: wrong answer -1 and test 2: wrong answer -2
missing colon after def add(a, b)	passed: 0; one error load: SyntaxError: ...
return a + b + undefined_name	passed: 0; three errors test i: NameError: name 'undefined_name' is not defined
defines plus, not add	passed: 0; one error load: KeyError: 'add'

Ranking

rank_candidates(sources, func_name, tests) returns the indices of the candidates sorted by number of tests passed, best first, ties by index. For the five candidates above the passed counts are $3, 1, 0, 0, 0$, so the order is [0, 1, 2, 3, 4].

What to implement

• run_candidate(source, func_name, tests): the dict above.
• rank_candidates(sources, func_name, tests): the indices, best first.

Requirements

• Nothing a candidate does may crash your function: catch every Exception.
• Use a new namespace dict for every candidate, so they cannot see each other's definitions.
• This is not a sandbox: exec runs the code with your program's permissions. Real systems run each candidate in an isolated process with time and memory limits (see the theory page).

Hint

A missing function is a KeyError on the namespace; turn it into the "load: ..." error. type(exc).__name__ gives the exception's name.