Research essay
Generating better answers is easier than recognising them
Published
Executive summary
Producing more candidate answers helps only if you can choose well among them. Judge that ability on the actual task: another model’s agreement is not a guarantee, and using the same model is not an automatic failure.
A reading of published experiments, with a proposed evaluation method. These are not experiments run by AlphaZeta.
Suppose ten attempts contain one correct answer. A perfect checker can rescue the task; a poor selector may confidently discard the only useful attempt. Generating candidates and recognising quality are two different capabilities.
That distinction matters whenever extra computing is spent on retries, competing drafts or multiple agents. More output creates an opportunity. It does not establish that the final answer improves.
In the 2024 paper Large Language Monkeys, Llama-3-8B-Instruct’s coverage on a 128-problem MATH sample rose from 82.9% at 100 attempts to 98.44% at 10,000. Coverage means at least one answer in the pool was correct. Over that range, the largest improvement from the studied voting and reward-model selection methods was only 40.50% to 41.41%.
The correct answer became much more likely to exist without becoming much easier for those selectors to find. This is a historical result for specified models, tasks and methods—not a universal ceiling on verification.
The paper also discusses imperfect software tests. An automatic check is useful evidence only to the extent that it tests the requirement that matters.
A second model may repeat the first model’s blind spot. The same model, asked to compare explicit alternatives or inspect evidence, may make a better decision than it did while generating a first answer. Neither setup deserves a pass or fail solely from its label.
Our earlier view treated independence too strongly as a prerequisite. The better requirement is demonstrated discrimination: does this judging procedure select more correct answers on relevant held-out cases? Independence remains a useful safeguard against shared errors, not a substitute for measurement.
OpenDeepThink reports a useful comparison before candidate evolution. Across 192 programming problems, generation-zero random selection scored 61%, its selector 72%, and an oracle able to recognise any correct candidate 83%. The selector recovered half the available 22-point gap, using the rounded table values.
On its CF-73 subset, the corresponding figures were 80%, 92% and 96%: three quarters of the gap recovered. This challenges a blanket claim that a model judge can capture only a small minority of the opportunity.
The subsets overlap with difficulty groupings and are not independent replications. Later generations change the candidate pool; comparing their final score with the initial oracle would mix candidate improvement with selection. The authors report no access to hidden tests during inference. These findings do not establish equivalent gains in open-ended investment research.
Freeze the tasks, candidate answers and scoring rule before evaluating a selector. Compare its chosen answer with a simple baseline and with the best result an ideal checker could extract from those exact candidates.
Report the absolute improvement, the remaining headroom and the cost. Use held-out tasks and uncertainty estimates where possible. A judge that improves one benchmark may still fail where the evidence is ambiguous or the answer cannot yet be resolved.
Recovered headroom = (selected accuracy − random accuracy) ÷ (oracle accuracy − random accuracy).
For a hypothetical pool with random accuracy 50%, selected accuracy 65% and oracle accuracy 80%, the selector gains 15 percentage points and recovers half the available headroom. Report both numbers: “half” alone conceals the size of the opportunity.
The ratio is undefined when the denominator is zero and unstable when headroom is tiny. If the candidate pool changes, calculate a new oracle. An oracle here is an evaluation reference built with known outcomes, not an available production service.
Ask whether a claim matches its source, whether the calculation reproduces, and what evidence would overturn the conclusion. Those checks can expose particular failures without pretending to verify an entire thesis.
Also separate three questions: did the agent follow the specified process; does that process improve outcomes; and can the evaluator judge either reliably? A neat audit trail answers none of them automatically.
For an example of a promising score meeting the harder test of actual research work, see our Jev evaluation.