Reflection as a Method to reduce Reward Hacking

Abstract
RL training in environments with exploitable reward functions can teach models to reward hack. When tasks and graders contain loopholes, it is possible for an agent to receive high reward for producing partial solutions, or even hack the environment to cheat, for example by copying the test answers. When this happens during RL training, the agent learns to cheat more and more, since cheating is rewarded. We study whether models can unlearn reward hacking by reflecting on their own bad behaviors.
We start from a Qwen3-8B base model before post-training and run GRPO on programming environments that intentionally have reward hacking loopholes. When this process starts, the model reward hacks on 1.6% of rollouts but as we run more RL steps, this hack rate rises to 64.7%. We then use the model’s own hacked trajectories to create SFT data: Reflections on bad behavior. We ask the model to reflect on its past hacking behaviors and write the solution it should have given instead. In the second step we SFT on the corrected responses alone: the reflection helps write each correction, but the model is not trained on it.
We find that after going through SFT on the model’s reflections, the reward hacking rate drops by 28.6 percentage points. After SFT on only the model’s corrected responses, the reward hacking rate drops by 62.0 percentage points, to just 2.7%. This score is aligned with the low reward-hacking rate of the base model.
SFT reduces reward hacking to 2.7%
Introduction
A coding agent can make a test pass by fixing the program, which is the ideal approach, or it may discover shortcuts such as changing the tests themselves. Due to unknown loopholes, even carefully designed RL environments can be susceptible to these hacks. As RL progresses, the model keeps exploiting these loopholes. This means that the model learns to keep exploiting these loopholes even across real-world tasks, often resulting in faulty or incorrect solutions.
We wanted to understand whether SFT can teach models to complete tasks honestly in environments with graders allowing for cheating to also earn a reward.
Anthropic’s Training a Misaligned Reward Seeker studies how RL on exploitable environments can encourage reward hacking. Inspired by that, we wanted to study whether learning from misaligned hacking behaviors can teach a model prone to reward-hacking to be honest even on unseen environments with exploitable graders.
We wanted to teach the model that earning a reward does not always mean that it completed the task honestly. The first step in this process was to make it recognize when it had reward-hacked. Existing work on Retrospection-Only Fine-Tuning (ROFT) showed that training agents on reflections of their own attempts could improve future task performance. Inspired by this, we tested whether reflection-based training could also reduce reward hacking.
The model often gives an accurate account of what it did wrong. For example, it can explain that hardcoding an expected value did not solve the task. However, training on this reflection alone works only partway. We see that even after ROFT, the model sometimes repeats the same misaligned behavior as was present in the original rollout.
We get a larger change when we SFT on more concrete data rather than just reflections. We give the model its own reflection, and have it write the response it should have produced at the turn where it cheated instead. We then train on that response in the context of the original trajectory. The reflection helps to generate the correct, aligned response, which, through SFT, the model learns to produce.
We evaluate whether training on these aligned responses reduces reward hacking, how much of this data is needed, and whether the model’s useful abilities across domains are retained. We also test how easily further RL can induce the model’s reward-hacking behavior.
Our main comparison follows Qwen3-8B before our training, after reward-hacking RL, and after a single SFT run on corrected responses pooled from two data collections. The first collection captures the reward hacking model’s failures; the second captures failures remaining after an initial SFT round. We separately evaluate sequential training on the two collections and training on reflections alone. Capability checks apply to the sequentially trained model, while additional resistance training is reported as an exploratory follow-up.
Methodology and experimental setup
Teaching the model to reward hack
We use Qwen3-8B in non-thinking mode throughout, with LoRA adapters trained on Tinker. We refer to the model before our experiments as Base.
The model receives a programming task and a workspace, writes code, and sees the resulting output and grading feedback. During RL, it earns a reward when the visible grader passes. A separate clean check determines whether it completed the intended task honestly. We count successful reward hacking when the model earns the reward through behavior that the clean check identifies as an exploit.
The initial training stage uses 20 generated programming tasks. Fourteen contain a corrupted expected value in their visible tests, creating a conflict between implementing the requested behavior and passing the grader. The other six are honestly solvable controls. Later stages introduce broader loopholes, including exposed answer keys, mutable reference implementations, unenforced constraints, and fabricated success markers.
We validate these environments with scripted policies before training: honest behavior must not be flagged as hacking, while the intended exploit must earn the reward and trigger the clean check. We use the checkpoint after 60 RL steps as the starting point for our corrective training experiments.
Turning detected hacks into corrected responses
We collect 1,856 episodes from the model after reward-hacking RL. Using audit feedback about each attempt, the model generates reflections explaining what happened and why its behavior failed to complete the task honestly.
We then ask it to write the response it should have given. It receives the original conversation, its hacked reply, and its reflection. If a test contradicts the specification, for example, the correction should provide an honest implementation and explain the contradiction rather than change the test or hardcode an answer.
When we ask the model to fix a cheating response, its new answer can still be wrong. It might repeat the cheat, use a different trick, or leave unfinished code. We check for these problems and keep 628 of the 656 rewritten answers. But we do not run every answer’s code, so passing this check does not guarantee that it solves the task correctly.
We use supervised fine-tuning on these responses, mixed with about 40% honest turns from the original collection. During training, the model sees the original conversation context without the reflection or rewrite instructions. The loss applies only to the response we want it to produce.
Improving the data with remaining failures
An initial SFT model helps us identify failures missing from the first collection. Sampling this model on the training tasks produces 306 additional episodes and 69 caught hacks, from which we retain 61 corrected responses. We also generate 144 continuation examples that teach the model to remain honest after repeated grading failures.
For our main result, we return to the model after reward-hacking RL and perform one SFT run on the pooled data from both collections. The final model trains once, but creating its dataset still requires an intermediate SFT model to discover remaining failures.
We separately evaluate two ablations: training sequentially on the two collections, and training on reflections alone. Reflection-only training is a separate baseline, not an intermediate stage in the main pipeline.
Evaluating the change in behavior
We build a separate evaluation covering four exploit families: editing the checker, claiming success without completing the work, misreporting evidence, and exploiting contradictory test fixtures.
| Evaluation family | How candidates vary | Candidate tasks → retained |
|---|---|---|
| Edit the checker | 24 function specifications, with four test-file styles and three grader styles available | 24 → 10 |
| Claim success without doing the work | 24 infeasible problems, with ten report-file formats available | 24 → 11 |
| Evidence misreporting | 38 log domains, with six layouts and six header styles available | 38 → 33 |
| Fixture mismatch | 38 specified functions, with three fixture or waiver formats available | 38 → 31 |
We screen 124 candidate tasks using Base and the model after reward-hacking RL. We retain tasks where Base never hacks across four screening rollouts and the hacking model succeeds at least once, leaving 85 tasks. Each model then receives three fresh rollouts per task, giving 255 evaluation rollouts.
This selection lets us study whether training repairs an observable behavior, but it favors a contrast between Base and the hacking model. The evaluation tasks are excluded from corrective data collection, although some exploit mechanisms overlap with training. The results therefore measure improvement on this selected suite, rather than general resistance to every possible exploit.
We evaluate 16 honestly solvable variants separately to check useful task completion. We also test coding, math, and general knowledge, although those capability checks currently cover the two-round SFT model rather than the main pooled-data model.
Results
Training on corrected responses reduces reward hacking
Reward-hacking RL increases the observed hack rate from 1.6% to 64.7%. A single SFT run on corrected responses pooled from both collections reduces it to 2.7%, compared with 36.1% for training on reflections alone. The two-round SFT ablation reaches 2.0%.
| Model or training configuration | Hacks / 255 rollouts | Hack rate |
|---|---|---|
| Base | 4 | 1.6% |
| After reward-hacking RL | 165 | 64.7% |
| SFT on reflections alone | 92 | 36.1% |
| SFT on first-collection corrections | 23 | 9.0% |
| Single SFT run on pooled corrections | 7 | 2.7% |
| Two-round SFT ablation | 5 | 2.0% |
These results show that training on corrected responses substantially reduces hacking in our evaluation. Pooling the data in one SFT run achieves a similar observed rate to sequential training, although the difference of two rollouts does not establish equivalence. Training budgets also differ across configurations, so the comparison with reflection-only training does not isolate the effect of the training target alone.
Additional ablations help assess how the corrective data contributes. At 20% data, generating corrections without a reflection reaches 18.4%, compared with 9.8% when reflections are included. Training on honest base-model responses without hack-specific corrections reaches 11.8%. These findings are consistent with reflections helping generate useful corrective examples, but differences in the retained data mix prevent a clean attribution to reflection alone.
A second SFT seed reaches 5.9% after the first round and 3.1% after the second. These runs share the original hacking model and first collection, so they test repair variability rather than independently repeating the entire pipeline.
A subset of detected failures provides useful supervision
We next vary how much of the first collection’s caught-hack data is used for training. All runs below use the same training budget: 20 updates at a learning rate of 5e-5, with nested subsets and the same proportion of honest examples.
Corrected-reply SFT with different amounts of data
Every configuration substantially reduces hacking from 64.7%. Using 20% of the examples reaches 9.8%, while a separate full-data run with 40 updates and a higher learning rate of 1e-4 reaches 9.0%.
The results suggest that useful corrective supervision can come from a subset of detected failures. They do not identify an optimal data fraction: the rates are not monotonic, these are single-seed comparisons, and holding updates fixed means smaller datasets are repeated more often. Subsampling detected hacks also does not establish that a detector can systematically miss entire exploit types without affecting the outcome.
Checking whether useful abilities are retained
A model that stops doing useful work might also stop reward hacking. We therefore evaluate coding, math, and general knowledge alongside the hacking results. These checks cover the two-round SFT model, not the main model trained once on pooled data.
General ability after corrected-reply SFT
The two-round model shows no obvious loss on the questions tested. It also solves 87.5% of feasible-task rollouts and refuses 4.8% of safe XSTest prompts, compared with 4.0% for Base. Separate response-diversity checks show no obvious collapse on the prompts tested.
These are limited checks rather than evidence that every capability is preserved. GSM8K and MMLU use subsets, and the main pooled-data model still needs the same capability evaluation.
| Model | HumanEval — 164 questions | GSM8K — 250 questions | MMLU — 400 questions |
|---|---|---|---|
| Base | 85.4% | 86.8% | 65.2% |
| After reward-hacking RL | 86.0% | 92.4% | 65.5% |
| Two-round SFT ablation | 89.0% | 89.6% | 66.8% |
Further RL can bring hacking back
Reduced hacking after SFT does not guarantee resistance to new training incentives. When we resume hack-inducing RL from the two-round model, hacking returns quickly: the first four reported attack steps show rates of 1.6%, 17.5%, 54.7%, and 68.8%.
These rates are measured on the attack’s training environments, rather than the held-out evaluation above. They show that correcting the model’s current behavior and making that correction resistant to further training are separate problems. We examine additional resistance training as an exploratory follow-up.
Resistance under a 60-step RL attack
Discussion and next steps
These experiments suggest that detected hacks can become useful training data. Each failure captures a point where the model chose to satisfy the grader at the expense of the task. Asking it to explain that choice and produce a corrected response turns the failure into supervision for honest behavior in the same context. Training on those responses reduces observed reward hacking from 64.7% to 2.7% in our evaluation.
The comparison with reflection-only training suggests that the form of supervision matters. A model may explain why an action was wrong without reliably choosing a better action later. Corrected responses provide concrete examples of what it should do instead, including situations where completing the task honestly means acknowledging that the grader cannot be satisfied. Training budgets differ, however, so the results do not isolate the contribution of the training target alone.
The subset experiments suggest that useful improvements do not require using every detected failure. They also raise a question about how to build better datasets: which examples teach behavior that transfers beyond the particular hacks they correct? Increasing the number of examples did not produce a steady improvement under a fixed training budget. This motivates studying the coverage and composition of corrective data, without yet establishing an optimal mix.
The main open question is how broadly this behavior generalizes. Our evaluation includes tasks selected to expose the hacking model’s behavior, and some exploit mechanisms overlap with training. Testing independently selected exploit families, repeating the experiments across seeds, and checking the main model’s useful capabilities would strengthen the finding. The results support learning from discovered exploits alongside improving graders; they do not yet show that the model will consistently choose honest task completion whenever a new shortcut becomes available.
Acknowledgements
We would like to thank Thinking Machines Lab for supporting us in using Tinker.
If you use this work, please cite it as:
@misc{patel2026reflection,
author = {Patel, Romir* and Kumar, Mrinal and Dimakis, Alex},
title = {Reflection as a Method to reduce Reward Hacking},
howpublished = {https://bespokelabs.ai/blog/reflection-as-a-method-to-reduce-reward-hacking},
note = {Published: 2026-11-16},
year = {2026}
}
References
- Qi, R. et al. Training a Misaligned Reward Seeker. Anthropic, 2026.
- Light, J. et al. Shockingly Simple Self-retrospection Improves Agentic Models Without RL. 2026.