When human opinion isn’t one answer, should a reward model still give a single score?

2026-09-30

Imagine two sets of ratings:

In the first, everyone gives a response a score of 3.

In the second, half the people give it a 1 and half give it a 5.

Both average out to 3.

But clearly these two 3s mean completely different things: the first says people largely agree, the second says opinion is deeply split.

For most reward models today, both cases are likely to be compressed into a similar scalar reward.

That is the starting point for Diffusion Reward Models (DRM).

Why do reward models need a distribution?

Reward models are a core component of RLHF and post-training for large models. Whether a traditional discriminative RM or one of the many generative judges, the final step usually maps a prompt-response pair to a single determinate reward.

That embeds a strong assumption:

For any given response, there is one stable, unique "true score."

Real human preference is not necessarily like that.

Annotators can reach different, even conflicting judgments on an identical response because of their values, evaluation criteria, reading of the rubric, and the trade-off they strike between helpfulness and harmlessness. Averaging those judgments into a single point not only erases uncertainty but can also wipe out a genuinely multimodal structure.

Based on this observation, a research team from Tsinghua University proposes Diffusion Reward Models (DRM):

Instead of predicting a single reward, the reward model learns the full reward distribution.

➤ Paper

🔗 https://arxiv.org/abs/2609.33803

➤ Code

🔗 https://github.com/thunlp/DRM

➤ Model

🔗 https://huggingface.co/Teburile/DRM

From a score to a distribution: what does the reward model lose?

Mainstream reward models today fall into a few broad lines:

The most common, discriminative RM: learns preference through objectives such as Bradley-Terry and outputs a scalar score.

Multi-objective RM: splits helpfulness, correctness, harmlessness and other factors into separate dimensions, but practical decisions usually still require re-aggregating them.

Distributional reward model: predicts, for example, the mean and variance of a Gaussian, or a set of quantiles.

These methods already concede that a reward is not necessarily a fully determinate point.

But the problem is not fully solved.

A Gaussian is inherently unimodal, and a fixed quantile grid fixes the way the distribution is described in advance. In other words, before the model ever sees the data, we have already decided for it what the reward distribution should look like.

If real human preference can itself have multiple peaks, such a fixed parametric assumption becomes a new bottleneck.

So the team took a different angle:

What if we simply don't presuppose the shape of the reward distribution?

That naturally leads to diffusion models.

Method: putting a diffusion model into reward space

The core change in DRM is quite direct:

Keep the LLM encoder, but replace the conventional scalar value head with a lightweight Diffusion Reward Head.

Given a prompt x and a response y, the frozen LLM encoder first extracts a semantic representation. A lightweight Diffusion Transformer (DiT) then learns to progressively recover reward samples from Gaussian noise instead of directly regressing a reward.

Repeated sampling yields an empirical reward distribution.

In other words:

Conventional RM

DRM

One response in → one score out

One response in → a set of possible rewards → a distribution

So the same response can produce a tightly concentrated distribution, a wide variance, or even several distinct peaks.

DRM training and inference pipeline

During training, the hidden states of the LLM encoder condition the Diffusion Reward Head; during inference, DDIM sampling starts from multiple Gaussian noise draws to form the empirical reward distribution.

Why freeze the encoder?

The team did not retrain the whole LLM. They froze the encoder of an existing reward model and trained only the lightweight Diffusion Reward Head.

This preserves the semantic information already learned in the LLM representation, and it lets the experiments answer one key question more cleanly:

If the backbone and training data stay the same and only the reward head changes from deterministic/parametric to diffusion, what happens?

One architecture for two kinds of reward supervision

Training data for reward models is usually not uniform. Some datasets provide scores on multiple attributes directly, such as helpfulness, correctness and coherence; others contain only pairwise preferences such as chosen/rejected.

DRM comes in two versions:

DRM-Multi-8B: trained with multi-attribute reward supervision in a unified 19-dimensional reward space

DRM-Pref-8B: trained directly on pairwise preference data

In multi-attribute data, many samples label only some attributes, so DRM uses a mask to keep unlabeled reward dimensions out of the denoising loss.

For pairwise preference, where no explicit absolute reward label exists, the paper constructs symmetric pseudo-reward targets and adds a Bradley-Terry ranking objective on top of the diffusion denoising objective, ensuring the preferred response receives a higher reward than the rejected one.

Two seemingly different reward modeling paradigms thus collapse into one framework: learning a conditional distribution in reward space.

Experiment 1: does the Diffusion Reward Head itself help?

After a new reward model is proposed, the obvious question is:

Is the improvement due to the architecture, or to differences in data, backbone or training scale?

So the paper starts with a comparison that is as controlled as possible.

DRM-Multi uses the same multi-attribute corpus as ArmoRM and inherits the encoder of FsfairX-LLaMA3-RM-v0.1. In this comparison, the change is concentrated in the reward head.

The paper evaluates on five reward modeling benchmarks — RewardBench v2, PPE, RMB, RM-Bench and JudgeBench — covering chat, instruction following, math, code, factuality and safety.

DRM results across benchmarks

The results:

Model

Average score

ArmoRM

62.3

QRM

64.1

DRM-Multi-8B

66.2

DRM-Pref-8B

65.8

DRM-Multi reaches 78.0 on RMB Pairwise, and DRM-Pref reaches 78.2.

The point is not that DRM tops every benchmark — it does not.

What matters is the controlled comparison:

With training data and backbone held as fixed as possible, changing only the reward head, the Diffusion Reward Head still delivers an overall gain over both the scalar head and the fixed-quantile head.

Experiment 2: is human preference really multimodal?

This is a critical link in the chain.

If "multimodal human preference" were only a nice motivation and real annotations merely jittered slightly around some mean, diffusion would not be necessary.

So the team first looked not at DRM but at real repeated human annotations.

The paper uses two datasets with repeated annotations:

HelpSteer2-Disagreements: multiple pointwise ratings for the same prompt-response

MultiPref: multiple preference judgments for the same response pair

Rather than only computing variance, the researchers examine three more structurally meaningful phenomena: rating range, separated clusters and polarization.

On the helpfulness dimension of HelpSteer2:

43.49% of samples span a rating range of at least 2 points

28.20% show clear separated clusters

24.34% contain both low and high ratings

On MultiPref, 45.64% of overall preference samples contain judgments pointing in opposite directions.

These experiments ask:

Human disagreement is not always small noise around a mean, like 3.1, 3.2, 3.3. It can form genuinely separate clusters of judgment.

If the data itself is multimodal, predicting only a point estimate, or only a unimodal Gaussian, necessarily loses information.

Has DRM learned a distribution, or just randomness?

There is a second question.

Diffusion is stochastic by nature. A model that samples a different reward each time has not thereby shown that it learned human disagreement.

The simplest counterexample: the model may just add a similar blob of random noise to every sample.

So the paper runs two levels of validation.

Level one: does the predicted distribution match the real human distribution?

The team compares reward samples generated by DRM on HelpSteer2-Disagreements with the empirical distribution formed by repeated human ratings of the same response.

They measure the gap with Wasserstein distance, JS divergence and L1 distance.

On helpfulness, DRM's Wasserstein distance is 0.804, below the empirical prior's 1.030, the global Gaussian's 1.032 and the pointwise baseline's 1.032; it also achieves the smallest distance on JS and L1.

DRM is not learning a global template of "how uncertain people are in general." It adjusts its reward distribution to the specific prompt-response.

🔥 The more telling experiment: the more people disagree, the more DRM splits

If DRM truly captures human disagreement, one prediction follows naturally:

The more divided human opinion is, the more likely DRM's reward distribution should be multimodal.

The paper tests it directly.

The team groups samples by the strength of disagreement in repeated human ratings and measures the multimodal ratio of DRM's output.

The result is clear:

DRM multimodality rises with human disagreement

For helpfulness:

Low human disagreement: 37.6%

Rating range ≥ 2: 55.5%

Clear low-high polarization: 63.2%

Correctness follows the same path: 37.6% → 54.7% → 62.0%.

In other words:

The stronger the human disagreement, the more pronounced DRM's multimodality.

This closes the argument:

Real human annotations contain structured disagreement → DRM's output distribution tracks the human distribution → as human disagreement grows, DRM's multimodality grows with it.

Experiment 3: can a full reward distribution do what a scalar reward cannot?

If the distribution is ultimately averaged back into a scalar reward, most of the method's value disappears.

So the paper designs two kinds of distribution-aware decision-making experiments.

Setting one: let the model abstain when it is unsure

If the system allows abstention, the natural question is:

Can the reward distribution tell us which decisions are themselves unreliable?

In Best-of-N, the paper compares not only the mean reward of the top and second candidates but uses their full reward samples to estimate the probability that the runner-up beats the current winner.

If the two distributions overlap heavily, the choice is unstable even when the means differ slightly.

The team then does something very direct:

Rank by uncertainty and reject the most uncertain samples first.

As coverage drops from 100% to 70%:

PPE Correctness improves by 2.81 percentage points on average across five subtasks

RMB Helpfulness/Harmlessness in BoN and Pairwise settings improves by 4.56–7.31 percentage points

This shows that

Variance and distribution overlap are not decorative statistics. They carry real information about whether a given judgment can be trusted.

Setting two: if abstention is not allowed, score stability into the reward

In real Best-of-N, the system often must pick one candidate and cannot say "I'm not sure about this one."

So the paper adds a full-coverage experiment.

The idea is simple: reward high-mean responses but also slightly penalize candidates whose distribution is especially unstable.

The paper uses a Lower Confidence Bound (LCB) for risk-sensitive aggregation, subtracting a fraction of the standard deviation from the reward mean.

As a result:

A response with a high mean but huge variance no longer easily beats one with a slightly lower mean but a very stable judgment.

In the PPE and RMB Best-of-N settings tested, LCB beats mean-only ranking; PPE BoN, for example, improves at every candidate count from 2 to 32.

A new scaling direction: think harder about the reward, not generate more answers

Test-time scaling is familiar by now.

Classic Best-of-N generates more responses and lets a reward model pick among them. The paper calls this response-axis scaling.

DRM opens a second, entirely different axis: reward-axis scaling.

For a fixed prompt-response, no new answers are needed; you simply draw more reward samples from the Diffusion Reward Head.

As the number of samples grows, the empirical estimate of the reward distribution stabilizes.

On RewardBench v2:

Effect of DRM reward-axis scaling

DRM-Multi rises from 56.5% with a single reward sample to 65.6% with 32 samples

DRM-Pref rises from 64.4% to 65.7%

The perspective is interesting:

Inference-time compute has usually been spent on generating more answers.

DRM offers an alternative:

Keep the answer fixed and spend more compute on what reward it should actually receive.

For the first time, the reward model itself has a natural test-time compute knob.

The last step: do benchmark gains carry over to RLHF?

One important validation is still missing.

A higher offline benchmark score does not guarantee a better policy when that reward model is used for training.

So the team put DRM into an actual RLHF pipeline.

The setup fixes:

The same Tulu3-8B-SFT initial actor

The same RLHF training setup

The same data source

Only the training-time reward model changes.

Comparing ArmoRM, FsfairX and DRM-Multi:

Model

Arena-Hard v2

MT-Bench

Tulu3-8B-SFT

1.0

59.4

+ArmoRM RLHF

1.0

71.5

+FsfairX RLHF

1.3

73.6

+DRM-Multi RLHF

2.0

74.8

This matters because it moves the work beyond a new RM benchmark method into actual policy optimization:

the gains from distributional rewards do not stop at offline scoring; they carry through to RLHF.

From scoring to describing a distribution of judgment

In hindsight, DRM is not really about the last layer of a reward model.

It challenges a very basic default in reward modeling:

Does a response really have one correct reward?

If everyone rates a response as "average," or half the people love it and half hate it, and both collapse to the same mean, the reward model throws away information that may be crucial to later decisions.

DRM's approach is to keep that information first.

Diffusion is only one tool for that goal.

The real change is reframing reward modeling from

point estimation

into

conditional density estimation.

The experiments answer three questions in turn:

1. Do multimodality and disagreement really exist in human preference? Repeated annotation says yes.

2. Did DRM actually learn this structure rather than output noise? The distance between predicted and human distributions, and the pattern that higher human disagreement brings higher DRM multimodality, provide further evidence.

3. Does this distributional information have practical value? Uncertainty-aware rejection, risk-sensitive ranking, reward-axis test-time scaling and the final RLHF experiments all show that information beyond the mean can be put to use.

The core question of this work comes down to one sentence:

A reward model should tell us not only how good a response is, but how people might judge it — and how certain that judgment really is.