How Do You Build an AI Verifier That Works?

First build the selection loop.

The verifier is only one component. A complete system generates candidates, scores them, applies a policy, and learns from mistakes.

Input x
  ↓
Generator samples k candidates
  ↓
y₁   y₂   y₃   ...   yₖ
  ↓
Verifier scores each candidate
  ↓
0.22  0.91  0.47 ... 0.68
  ↓
Policy: select · escalate · abstain

The formula says: choose the candidate with the highest verifier score. But it hides four engineering decisions:

• Candidate generation: Are the answers genuinely different, or ten copies of the same reasoning path?
• Verification signal: Does the score represent the final answer, each step, an external tool, or several judges?
• Acceptance policy: Is the best candidate good enough, or merely less bad than the others?
• Feedback loop: Which disagreements and overrides become new training or evaluation examples?

Do not begin by training a reward model. Begin by defining the candidate set, the evidence of correctness, and what the system does when that evidence is weak.

ORM: give the whole solution one score.

An Outcome Reward Model sees the problem and the completed solution, then predicts whether the final result is correct.

For problems with a known answer, ORM data is cheap to construct:

sample candidate solution
          ↓
extract final answer
          ↓
compare with known answer
       /      \
   correct   wrong
      1        0

The model is trained as a binary classifier. At inference time, generate several complete solutions, score each one, and select the highest-scoring candidate.

Here, y is the observed training target and r is the ORM’s predicted probability that the solution is correct. Calling y a target matters: automatic final-answer grading can label invalid reasoning as positive when it accidentally reaches the right result.

Why ORMs are useful

• One label covers an entire solution, so data collection is comparatively cheap.
• The score maps naturally to best-of-N candidate ranking.
• Known final answers can create labels automatically.

Where ORMs fail

A candidate may use broken reasoning but accidentally reach the correct result. The ORM sees a positive label and may learn to reward a fragile path.

Cobbe et al. showed that ORM-style verification can substantially improve math performance by ranking generated candidates. It is the simplest learned verifier worth trying when final answers are objective.

Build an ORM first when final-answer labels are cheap and intermediate reasoning is not itself safety-critical.

PRM: score the solution one step at a time.

A Process Reward Model produces a score for every reasoning step. That gives the system finer credit assignment and a location for the first error.

The verifier does not need to wait for the final answer to learn that Step 3 replaced the correct denominator with the wrong one.

A binary PRM can apply the same cross-entropy idea at every step:

PRM800K used three step labels—positive, negative, and neutral—and trained the model to predict the corresponding token. For final ranking, Lightman et al.’s strongest reported setting treated neutral as positive and multiplied the step probabilities.

How do step scores become one solution score?

A PRM returns a sequence such as:

You still need a rule for ranking the entire solution:

Here, each v_t ∈ (0, 1] is the PRM’s predicted probability that step t is correct.

Choose the aggregation rule using downstream selection accuracy—not only the PRM’s training loss.

The annotation bottleneck

Lightman et al. asked human annotators to label reasoning steps and released PRM800K, a dataset containing 800,000 step-level labels. When the PRM ranked 1,860 generated candidates per problem, it selected a correct solution for 78.2% of a representative 500-problem MATH subset, versus 72.4% for the ORM. Active learning also improved labeling efficiency by 2.6×.

That is strong evidence for process supervision. It also exposes the cost problem: experts must read partial reasoning carefully enough to find the first invalid step.

A PRM buys better diagnosis and credit assignment with more expensive, more ambiguous labels.

Can the model label a step without a human?

Math-Shepherd reframes step quality as potential: if we continue reasoning from this step many times, how often can we still reach the correct final answer?

Math-Shepherd defines two labels:

problem + known final answer
              ↓
sample full reasoning solution
              ↓
stop at intermediate step sᵢ
              ↓
sample N completions from sᵢ
              ↓
check their final answers
              ↓
convert success frequency into a step label

Why this works

If a step places the solution on a promising path, future completions should reach the correct answer more often. Final-answer supervision is recycled into step-level supervision.

Why it is not ground truth

A logically invalid step may still be recoverable. A valid step may receive a low score because the completer is weak. The label measures “ability to finish correctly from here,” not pure logical validity.

In Math-Shepherd’s experiments, step-by-step Proximal Policy Optimization (PPO)—a reinforcement-learning update that applies rewards after individual reasoning steps—improved Mistral-7B from 77.9% to 84.1% on GSM8K and from 28.6% to 33.0% on MATH. After that training, self-consistency plus Math-Shepherd reranking over 256 generated candidates reached 89.1% on GSM8K and 43.5% on the 500-problem MATH500 subset. Math-Shepherd reranking without self-consistency scored 88.4% and 41.1% in the same table.

Automatic labels are most credible when the final answer is objectively checkable and the completer is strong enough to reveal a step’s true potential.

What if no verifier is reliable enough alone?

Weaver combines several imperfect verifiers and estimates how much each one should count. The paper validates the method on mathematics and reasoning benchmarks; using the same approach for broader open-ended tasks is a reasonable hypothesis, not a result the paper establishes.

What Weaver does

1. Normalize: min-max scale each verifier’s outputs into the same 0-to-1 range.
2. Binarize: convert continuous scores into yes/no signals. A small labeled development set estimates class balance and helps choose per-verifier thresholds.
3. Filter: remove low-signal verifiers with extreme output rates, such as judges that approve almost everything.
4. Estimate accuracy: use moment matching over observed agreement patterns to estimate true-positive and true-negative rates under the latent-variable model.
5. Weight: combine the verifier outputs into one selection score.

The important assumption

Weaver’s latent-variable estimator assumes verifier outputs are conditionally independent given the unknown true label. In plain English: once we know whether the answer is truly correct, each verifier is assumed to contribute its own information. The estimation also needs enough verifiers to be better than random; otherwise the unlabeled agreement statistics cannot reliably distinguish “mostly right” from “mostly wrong.”

Real verifier pools often violate this. Two judges may use the same base model, training data, prompt pattern, or benchmark artifacts. Their agreement can double-count one underlying opinion.

Why distill the ensemble?

Running several large verifiers on every candidate is expensive. Weaver therefore trains a 400M-parameter cross-encoder—a compact model that reads the problem and candidate together and emits one score—from the ensemble’s combined labels. Version 3 reports that this distilled verifier preserves about 98% of the ensemble’s accuracy gains while reducing verification compute by up to 99.97% (the abstract reports 98.7%; Section 6 and the conclusion report 98.2%).

Use a verifier ensemble to create a stronger teacher; distill it when runtime cost becomes the bottleneck.

A good classifier can still be a bad selector.

Verifier accuracy on isolated answers is not the product objective. The product objective is choosing an acceptable answer from the candidates the generator actually produces.

Test the shifts that production will create

• Generator shift: replace or upgrade the proposer and re-evaluate the verifier.
• Difficulty slices: report easy, medium, and hard cases separately.
• Adversarial candidates: include persuasive answers designed to exploit verifier preferences.
• Temporal holdout: test on newer tasks to expose leakage and drift.
• Disagreement set: audit examples where tools, reward models, and human reviewers diverge.

Evaluate the generator and verifier together. A verifier that worked on yesterday’s model is not automatically valid for tomorrow’s candidates.

The generator will learn what the verifier likes.

Once verifier scores control selection or reinforcement learning, candidates are optimized against those scores. Any shortcut in the verifier can become a target.

verifier rewards a superficial pattern
              ↓
generator discovers the pattern
              ↓
more candidates imitate it
              ↓
verifier score rises
              ↓
real quality plateaus or falls

This is why stronger search can make a system worse. Search allocates more compute to branches the verifier prefers. If the preference is wrong, the system expands the wrong branch more aggressively.

Defenses

• Keep deterministic tests and external evidence outside the learned reward model.
• Use adversarial candidates that are polished, verbose, and wrong.
• Separate the data and model lineage of generator and verifier where possible.
• Hold out human audits that are never used to tune the verifier.
• Require an absolute acceptance threshold and preserve an abstention path.

A practical first implementation.

1. Write the acceptance contract. Define correct, unsupported, unsafe, and ambiguous outputs with examples.
2. Measure pass@k. Calculate the share of tasks with at least one correct answer among k candidates. This reveals whether sampling creates useful selection headroom.
3. Add objective checks. Use tests, schemas, calculations, retrieval evidence, or simulators before learned judges.
4. Train the simplest learned verifier. Start with an ORM when final labels are reliable; move to a PRM when path quality matters.
5. Choose the aggregation rule. Validate minimum, product, or learned step aggregation on selection—not just classification.
6. Calibrate the policy. Map scores to select, escalate, and abstain thresholds using observed risk.
7. Audit shifts and exploits. Re-test after generator changes and continuously sample high-score failures.

objective runtime check available?
  ├── yes → deterministic verifier
  │          tests · execution · schemas · exact calculation
  │
  └── no  → reliable outcome labels available for training?
              ├── yes → ORM
              │          ↓
              │      reasoning path matters?
              │          ├── no  → rank complete answers
              │          └── yes → PRM
              │                    ↓
              │             step labels too expensive?
              │                    └── rollout-based labels
              │
              └── no  → diverse verifier ensemble

ORM · PRM · ensemble
          ↓
calibrate + abstain
          ↓
      human audit

The whole engineering story in one chain.

Candidates
    ↓
Outcome score: did it end correctly?
    ↓
Process score: where did it go wrong?
    ↓
Automatic labels: can step supervision scale?
    ↓
Ensemble: which imperfect judges deserve weight?
    ↓
Calibration: is the winner good enough?
    ↓
Monitoring: does the verifier still work after change?

If you remember only six terms, remember: