Introduction to Modern Reinforcement Learning

Human disagreement is not merely noise. It can reveal underspecification, value pluralism, context dependence, rater heterogeneity, or candidate imbalance.

Disagreement taxonomy

Diagnose the source of disagreement

Click a source of disagreement and inspect what it means for the dataset, the rubric, or the reward model.

Likely category

Rubric repair

Rater scenario

Two raters compare a concise answer with a complete answer. One chooses the short answer because it is easier to read; the other chooses the complete answer because it explains the caveat.

What it reveals

The rubric has not said how to prioritize brevity versus completeness.

Next action

Add explicit tie-breakers, examples, and edge cases to the instructions.

The practical rule is to treat disagreement as signal first, then decide whether it is avoidable noise, meaningful pluralism, or evidence that the rubric needs repair.

Disagreement diagnostics

Read entropy and margin together

Human disagreement can reveal ambiguity, value pluralism, rater effects, or data gaps.

Choice entropy

= -

\\sum_i

Preference margin

= |-|

Choice entropy: higher => more ambiguity

Margin: small => ambiguous

Choice entropy

how spread out the label distribution is

Entropy gets larger when votes are split across choices. It diagnoses ambiguity, pluralism, unclear instructions, or cases where the rater pool genuinely does not agree.

Terms

Soft labels preserve uncertainty: train on the estimated probability of A beating B, not only the majority winner.

Uncertainty diagnostics

The diagnostic above uses both equations to flag prompts that may need more labels, better instructions, or group-specific analysis. It is also useful to separate aleatoric uncertainty, where human preferences genuinely vary, from epistemic uncertainty, where the model simply lacks enough data or coverage.

Annotator-aware reward models

If rater differences are stable, model them instead of flattening them away. A simple annotator-aware scoring model can be written as:

Equation walkthrough

Annotator-aware score

Click each term to see how the base reward model is adjusted for a specific annotator or annotator group.

=++

Feature contribution

annotator-weighted feature score

The dot product says: measure the answer's features, then weight them by this annotator's priorities.

Terms

This score is a modeling device: it helps capture stable rater differences instead of forcing every annotator into one flattened average.

After scoring each candidate this way, the preference probability compares the annotator-aware scores for A and B:

Equation walkthrough

Annotator-specific preference probability

Click each term to see how an annotator-aware reward model turns two scores into one predicted preference.

=([-])

Preference margin

the only part that decides which answer is favored

If this margin is positive, A is more likely to be chosen. If it is negative, B is more likely. If it is close to zero, the pair is ambiguous for this annotator.

Terms

The key object is the annotator-specific margin: the same answer pair can look different for raters with different sensitivities, biases, or decisiveness.

Here $r_\theta(x,y)$ is the base reward model, $f(x,y)$ is a feature vector of the output, $w_a$ captures annotator-specific feature sensitivity, $b_a$ is an annotator bias term, and $\beta_a$ is an annotator-specific inverse temperature. A strict annotator may use a different threshold than a lenient annotator; an expert may weight factuality more heavily than style.

For pure pairwise comparisons on the same prompt, a candidate-independent $b_a$ cancels from the difference. It is still useful for absolute ratings, accept/refuse thresholds, or models where the bias attaches to position, source, or rater severity. The broader lesson remains: model annotators separately when disagreement is structured.

✦

Aggregation beyond majority vote

Use soft labels such as $y^*=P(A\succ B)$ and train with $\ell=-[y^*\log p+(1-y^*)\log(1-p)]$ . Learn calibrated rater parameters such as $b_a$ and $\beta_a$ . Infer rater reliabilities and latent labels with mixture-of-raters models. Use group-aware rewards $r_{\theta,g}(x,y)$ when values legitimately differ across user segments.

Operationalizing disagreement

Use disagreement to collect smarter data. Target items with high entropy or $m\approx0$ , seek varied failure modes rather than only uncertain ones, and collect short "why A over B?" rationales to expose hidden criteria. Those rationales can refine rubrics and improve rater training.

Design can also reduce avoidable ambiguity. Tighten rubrics with explicit tie-breakers such as correctness $\succ$ safety $\succ$ brevity. Randomize candidate order, blind raters to model identity, insert gold checks, and use contrastive prompts that surface real tradeoffs such as politeness versus directness.

⚠

Do not erase pluralism too early

A simple majority vote can hide subgroup gaps. If novices and experts disagree, or if different user communities weight values differently, the disagreement may be exactly the information your alignment process needs.

Disagreement is Signal

Diagnose the source of disagreement

Read entropy and margin together

Uncertainty diagnostics

Annotator-aware reward models

Annotator-aware score

Annotator-specific preference probability

Operationalizing disagreement