Goodhart in the Planner: Harder Search Against a Learned Reward Cuts True Return up to 76%, and Held-Out Accuracy Does Not Warn
ROBOT LEARNING
A robot that plans against a reward learned from human preferences is an optimizer pointed at a proxy. Measuring MPPI and best-of-N on five simulated tasks, a new paper shows the optimizer wins and the task loses - and that the standard validation checks stay silent while it happens.
By Ossam Hashim
KEY FINDINGS
- Harder sampling-based search against a learned scorer lowers true return, sometimes catastrophically. On HalfCheetah with crowd-sourced labels, true return falls from 480 at gentle search to 262 at the hardest temperature (uniform averaging: 233; the cloned policy run directly: 412); on Hopper from 468 to 272-356. On Meta-World drawer-open with 300 human labels, the drawer opens in 0 of 20 runs under search against 10 of 10 for the cloned policy the planner starts from - while held-out label accuracy stays at 0.75-0.97.
- Exploitation is not the whole loss. A perfect scorer keeps or improves return under harder search on cheetah (867 to 1,010) but still loses on hopper (559 to 342-373), so part of the damage is the planner's own: perturbing the cloned plan costs 0.7-2.2 standard deviations of true plan return, and selection recovers only a fraction of it. On window-open the balance flips and search genuinely helps.
- True return approximately collapses onto one curve per task: the square root of the per-step KL between the planner's weights and its proposals. A two-parameter fit covers both weighting rules tested (R2 = 0.79-0.99), so chi-squared weights help mainly by moving less, and the loss starts from the first step on tasks where the cloned policy is already near-optimal.
- The paper's theory makes the temperature a measurable quantity: chi-squared-regularized MPPI at a temperature set by the scorer's error spread on its own proposals is worst-case optimal, and no positive gain is certifiable once that spread exceeds the scorer's own. The catch is why standard checks fail: the drawer scorer's held-out accuracy is 0.88 while its correlation on the planner's proposals is 0.10 - and the hopper scorer shows 0.08 on the labelled data but 0.76 on the proposals. A scorer can be checked only where it will be optimized.
- Best-of-N with a learned verifier is not safe either: with 500 synthetic pairs, N = 16 already scores below a single noisy action, and N = 1,024 reaches 14/50 and 0/50 successes on two tasks. The most effective fix tested is labelling where the planner goes - half a 500-pair budget spent on the planner's own trajectories lifts hard search to 8/10 and the cautious ensemble to 10/10, level with 2,000 offline labels. A worst-case rule calibrated on 100 oracle-ranked proposal pairs is the safest fix but cautious exactly where search helps - and calibrated from one person's comparisons, it cannot detect a bias that person shares with the scorer.
An Optimizer Pointed at a Proxy
Language-model alignment has known for years that optimizing hard against a learned reward model degrades the quality the reward was meant to measure. Robot planners are a different optimizer: MPPI samples candidate action sequences around a proposal policy, weights them by a learned scorer, executes the first action, and re-plans at every step; best-of-N sampling picks whichever candidate a learned verifier likes most. The labels behind the scorer number in the hundreds to thousands, not the hundreds of thousands of RLHF. Whether, and how much, such a planner exploits its scorer had not been measured. This paper measures it, on HalfCheetah and Hopper with crowd-sourced Uni-RLHF labels (73% and 64% agreement with true rankings) and on three Meta-World manipulation tasks with synthetic and blind human labels, using a Bradley-Terry scorer over segment pairs and a policy cloned from the labelled data as the planning analogue of RLHF's reference model.
Two Ways to Lose
The failure decomposes into two patterns with different signatures. In the extremal pattern, the planner drives the robot into states the scorer overrates: on the drawer with synthetic labels the scorer's own estimate of the executed episode rises sharply as true return falls, and recorded runs show the arm parked in an unvisited corner the scorer rates highly. In the noise-chasing pattern - plate-slide, and the human-labelled drawer - the estimate falls instead: the planner chases plans that look good at planning time and do not survive closed-loop re-planning, an optimizer's-curse dynamic. The human-label scorer, probed with a linear fit, barely penalizes hand-to-handle distance (-0.16, against -0.40 to -0.49 for the synthetic one), and the human-labelled drawer fails whichever way its ties are handled. The two patterns respond to different fixes, which is why no single fix wins everywhere.
Why the Checks Stay Silent
The paper's theoretical core bounds the selection step of chi-squared-regularized MPPI by the scorer's error spread on the proposals themselves, proves the temperature set to that spread is worst-case optimal, and shows the guarantee of hard exponential search decays with sample count once the spread is too large. Its practical content is a measurement prescription: held-out accuracy and on-data correlation both fail to predict the quantity that matters. The 500-label drawer scorer pairs 0.88 held-out accuracy with a 0.10 correlation on the planner's 256 proposals; the hopper scorer inverts the pattern, 0.08 on the data but 0.76 on the proposals. The predicted initial slope of the return curve matches the measured one within 6% in eleven of thirteen settings, so the theory tracks the phenomenon - but only where someone bothers to evaluate the proposals.
Fixes, Measured Honestly
The fixes section is unusually candid. Chi-squared weights on a five-scorer ensemble help the walkers and not the drawer; the cautious ensemble (mean minus one standard deviation) rescues the synthetic-label drawer and helps the walkers less; automatic temperatures set from ensemble disagreement are the most even across tasks; with human labels every fix that searches hard fails. The oracle-calibrated worst-case rule - 100 proposal pairs ranked by true return, on start states the evaluation never uses - succeeds in all 50 runs per task across five independent calibration sets, keeps 86-94% of the cloned policy's return on the arm tasks, and still declines to search where search would pay, on window-open. Calibrate it from one author's own comparisons instead and it trusts the scorer trained on that author's labels, because a shared bias is invisible to the person who shares it; a second labeller, or correcting for pairs people cannot tell apart, bounds the damage. The standout is active labelling: pairs collected on the planner's own trajectories turn the drawer's initial slope from -250 to +33.
Assumptions and limitations
All results are the authors' own, in simulation, on five tasks with two label sources and one scorer family (Bradley-Terry over segment pairs, hundreds-to-thousands of labels); locomotion uses true MuJoCo dynamics at planning time. The 76% figure is the largest observed cut; several tasks are null or limit cases, and on window-open search helps throughout. The approximate KL collapse is a fit within each task, not a law, and the log-linear label-budget reading rests on five budgets. The worst-case guarantee covers only the selection terms of the return decomposition, its label-count corollary assumes a linear reward, and the calibrated rule's safety was demonstrated with an oracle calibration a deployed robot lacks - calibration from human comparisons is shown to inherit the labeller's shared bias. Best-of-N verifiers use synthetic labels only. Code and results are stated as to be released. The worked example is an invented toy; every return, correlation and success count above is the authors' measurement, not this blog's.
Worked Example: Best-of-N Against a Noisy Scorer
The mechanism fits in a few lines: candidates carry a true value, the scorer sees value plus noise, and picking the scorer's favorite preferentially picks lucky noise. The toy below draws eight candidates with true values near 1, scores them with Gaussian noise, and prints the true value of the scorer's pick against the average. Run it and the scorer's pick lands at 1.32 true value - dead average - while scoring 2.88 on the proxy: the selection gain is spent on noise, not the objective. Invented numbers, illustrative only.
import random
random.seed(7)
N, sd = 8, 1.5
true = [1.0 + random.random()
for _ in range(N)]
noisy = [t + random.gauss(0, sd)
for t in true]
pick = noisy.index(max(noisy))
mean = sum(true) / N
print('mean true', round(mean, 2))
print('picked true',
round(true[pick], 2))
print('picked noisy',
round(noisy[pick], 2))
Sources & Provenance
- Goodhart in the Planner: Sampling-Based Control Exploits Rewards Learned from Human Preferences (Bakshi, Bose; UT Austin and independent, 9 Nov 2026) - arXiv abstract
- Full paper (PDF): task setup, theory, fix comparisons, best-of-N study, null cases - arXiv full text
All figures are attributed to the preprint as published. Sources accessed 9 November 2026. This analysis is independent editorial content for GlobalTechDigest.
Comments
Post a Comment