How Do You Know When an AI Is Right?

Getting a correct answer and recognizing it are different skills.

A model may fail on its first attempt even when it is capable of solving the problem. Letting it try again creates more chances—but it also creates a selection problem.

AI generates:

Answer A  ✕
Answer B  ✕
Answer C  ✓
Answer D  ✕
Answer E  ✕

The correct answer exists.
The product still needs to find it.

This is the proposer–verifier model. The proposer explores possible solutions. The verifier scores evidence. A decision policy then selects an answer, asks for human review, or refuses to proceed.

A benchmark tells you how a model performs on average. A verifier helps decide whether the answer in front of you should be accepted now.

Some tasks come with a built-in answer checker.

Coding has tests. Arithmetic has known answers. Database queries can be checked against schemas and constraints. These tasks are unusually friendly to verification.

Generate code A → run tests → FAIL
Generate code B → run tests → FAIL
Generate code C → run tests → PASS
                              ↓
                         select C

No second AI needs to “feel” that solution C is better. The system executes an objective check. That is why coding, mathematics, and formal proofs are powerful environments for repeated sampling: extra attempts can be converted into real gains because correct candidates are comparatively easy to recognize.

Business questions are harder. There is no pytest strategy_memo. A customer-risk recommendation may depend on incomplete evidence, competing objectives, and judgment. That is where the four verification methods below become useful.

The cheapest trustworthy verifier is usually an external tool or rule—not another LLM.

The methods differ by what they inspect.

Do you check only the final answer? Every reasoning step? Automatically generated evidence? Or several imperfect judges at once?

1Outcome verification: check the destination

An outcome verifier looks at the completed response and asks: “Is the final answer correct?” It does not need to know where the reasoning first went wrong.

This is relatively cheap because one solution gets one label. Cobbe et al. used this pattern to rank many generated math solutions and found that verification improved performance.

The catch: a model can use invalid reasoning and still land on the right final answer. Outcome verification may reward the result while missing a fragile or misleading path.

Use outcome verification when the endpoint is objective and the path does not itself create risk.

2Process verification: check the route

A process verifier scores each intermediate step. Instead of learning only that the solution failed, it can identify the first step that made the rest of the answer unreliable.

An outcome label says only “20% is wrong.” A process label says the first error was using the wrong denominator; the correct growth rate is $50 ÷ $200 = 25%.

Lightman et al. found process supervision significantly outperformed outcome supervision in a large candidate-selection experiment. Using a PRM to rank 1,860 generated solutions per problem, the system selected a correct solution for 78.2% of a representative 500-problem MATH subset, compared with 72.4% for the ORM and 69.6% for majority voting. The team also released 800,000 step-level labels as PRM800K.

The catch: labeling every step takes much more expert time than labeling one final answer.

Use process verification when a correct-looking answer can hide unacceptable reasoning.

3Automatic process supervision: scale the labels

What if you want step-level verification but cannot pay experts to annotate every line? Math-Shepherd offers a clever answer: evaluate an intermediate step by asking whether multiple continuations from that step can still reach the known correct answer.

Intermediate step
      │
      ├── continuation A → correct
      ├── continuation B → correct
      ├── continuation C → wrong
      └── continuation D → correct

3 of 4 reach the answer → step score ≈ 0.75

The model turns final-answer checks into approximate step labels. This makes process supervision far more scalable.

The catch: “a correct answer can still be reached from here” is not identical to “this reasoning step is logically correct.” The automatic label is useful, but noisy.

Use automatic process labels when the domain has a reliable final answer and human step labels are the bottleneck.

4Verifier ensembles: combine imperfect judges

For open-ended tasks, there may be no single trusted checker. You might have an LLM judge, a reward model, a retrieval-grounding check, a policy engine, and a human reviewer. Each catches something different.

Candidate answer
      │
      ├── factuality judge     0.90
      ├── policy checker       1.00
      ├── reasoning verifier   0.55
      └── citation checker     0.80
                    ↓
          normalize + weight
                    ↓
             combined score

Weaver shows why naive majority vote is not enough. Some verifiers are more accurate than others. Some produce scores on different scales. Some are redundant. Weaver normalizes, filters, and estimates weights so that weak verifiers can be combined more effectively.

The catch: five judges trained on similar data may share the same blind spot. Agreement is not proof of independence—or truth.

Use an ensemble when no perfect checker exists, but measure diversity and calibration rather than counting votes.

Choose the verifier from the failure you cannot accept.

Do not start with “Which reward model should we train?” Start with “What would a costly false acceptance look like, and what evidence could catch it?”

A verifier can be confidently wrong too.

Trap 1: Treating an LLM judge as ground truth

An LLM judge is another prediction system. It may prefer polished writing, longer answers, or outputs that resemble its own style. Measure it against independently labeled examples.

Trap 2: Selecting the best of a bad batch

The highest-scoring candidate may still be unacceptable. Compare the winner with an acceptance threshold, not only with its competitors.

Trap 3: Mistaking agreement for correctness

Correlated verifiers can form false consensus. Track where each signal came from, what data shaped it, and which failure mode it is meant to catch.

The whole guide in one chain.

More attempts
      ↓
more chances to generate a correct answer
      ↓
but which answer is correct?
      ↓
Verifier
      ↓
check the outcome, the process, or several signals
      ↓
Selection policy
      ↓
select · escalate · abstain

If you remember only five words, remember:

Part 2 turns this mental model into an engineering system: how to train outcome and process reward models, create automatic step labels, combine weak verifiers, calibrate thresholds, and detect reward hacking.