Only correct signal teaches
We trained a 1.5B model by reinforcement against a code-running checker, next to three controls that differed only in the reward. The checked reward taught the model a skill it did not have. Coin-flip reward and failure-only reward, run identically, did not teach it; they broke the model.
The question
A model that teaches itself needs a training signal, and the cheapest signal is the model's own attempts, checked. Before building a loop on that idea we asked the narrow question it rests on: does reward from an code-running checker teach a model something it cannot already do, and is it the correctness of the reward that teaches, or just the act of training?
The second half matters. Reinforcement learning can raise scores with rewards that carry no information at all, when the data the model is tested on resembles what it already saw in pretraining. The way to tell the two apart is to run the same training with a reward that is deliberately meaningless, on tasks the model cannot have memorised.
Setup
The tasks are compositions of small program operations, generated at run time from a seed, so no particular task can be in any pretraining set. Difficulty is the depth of the composition, a dial with no ceiling. A checker grades every answer by running it on 12 hidden tests. Before the run we measured how often near-miss wrong programs get past those tests: between 0.7% and 4% at depths 1 to 8.
Correction, 25 October 2026. An earlier version of this page said the checker was certified at depths 1 to 12 and could not be passed by a wrong program. Both were wrong: that certification belongs to a later, fixed version of the task generator, and the one this run used was certified at depths 1 to 8, with the small error rate above. That version also has three defects we found afterwards: 2 of 120 sampled tasks had a wrong answer key, so a correct program fails; some steps change nothing, so a task's real depth can be lower than its stated depth, mostly for text tasks; and a few tasks return their input unchanged. Every arm saw exactly the same tasks, so the comparison between arms stands; the depths are nominal and the absolute accuracies are understated. Our next experiment uses the fixed generator.
We first measured where the model stops. Qwen2.5-1.5B-Instruct solved 0.33 of depth-1 and depth-2 compositions, 0.17 at depth 3, 0.04 at depths 4 and 5, and nothing beyond. Its frontier is depth 2. We trained on depths 2 and 3 and tested on depth 4: one step past anything it was trained on.
Four arms, identical in model, optimizer (GRPO with a rank-64 LoRA on all seven linear modules), 60 steps of 8 prompts with 8 samples each, prompts and seed. Only the reward differs:
| Arm | Reward | Role |
|---|---|---|
| Checked | 1 if the program passes all 12 hidden tests, else 0 | the treatment |
| Coin-flip | 1 or 0 at random, independent of the answer | the spurious-reward control |
| Failure-only | −1 if the program fails, 0 if it passes | learns only from mistakes |
| Untrained | no training | the baseline |
The rule, written before any code
The verdict was fixed in our ledger before the notebook existed. The run counts as only correct signal teaches if, at a test depth, the checked arm beats the untrained copy with an interval that excludes zero, the coin-flip arm does not, and the checked arm also beats the coin-flip arm. If coin-flip reward had moved the score too, the verdict would have been random teaches, and we would have reported that instead. With 100 test tasks the run was sized to detect an effect of about 0.13 reliably; the verdict itself is read from the intervals, not from that number.
Results
| Comparison | Depth 4 (trained 2–3) | Depth 3 | GSM8K (outside) |
|---|---|---|---|
| Checked − untrained | +0.100 [+0.040, +0.170] | −0.033 [−0.117, +0.050] | −0.050 [−0.111, +0.017] |
| Checked − coin-flip | +0.150 [+0.090, +0.220] | – | – |
| Coin-flip − untrained | −0.050 [−0.100, −0.010] | −0.167 [−0.267, −0.067] | 0.217 vs 0.70 |
| Failure-only − untrained | −0.050 [−0.100, −0.010] | −0.167 [−0.267, −0.067] | 0.156 vs 0.70 |
95% intervals. Depth 4: 100 tasks; depth 3: 60 tasks; GSM8K: 180 problems, where the untrained model scores 0.70 and the checked arm 0.65. The untrained copy scored identically before and after the run (drift +0.000), so the evaluation itself is not the source of any difference.
Why each arm failed
A difference between arms means little until you know how each one failed. We classified every final answer at depth 4:
| Arm, per 100 tasks | Passes | Partly right | Wrong | Syntax error |
|---|---|---|---|---|
| Checked | 15 | 5 | 72 | 8 |
| Coin-flip | 0 | 0 | 16 | 84 |
| Failure-only | 0 | 0 | 0 | 100 |
The final evaluation was repeated when a free GPU session resumed, so raw tallies are multiples of 100; decoding is greedy, each repeat gives the same answers, and the counts above are divided by the number of repeats.
Both controls destroyed the model's ability to write code, and with it most of its math: GSM8K fell from 0.70 to 0.22 under coin-flip reward and to 0.16 under failure-only reward. That changes how the result should be read. The gap between the checked arm and the coin-flip arm (+0.150) is partly "random reward breaks the model". The clean claim is the comparison with the untrained copy: +0.100 [+0.040, +0.170], a skill gained at no measurable cost.
What this does not show
- One seed. A replicate is needed before the size of the gain means anything precise.
- One depth. The gain appears at depth 4 and not at depth 3 (−0.033 [−0.117, +0.050]). We do not have an explanation yet.
- No transfer. Nothing moved on GSM8K. The skill is narrow.
- Not a new mechanism. It reproduces, under matched controls and on uncontaminated tasks, the finding that on clean data only correct rewards move a model (Wu et al., arXiv 2507.10532).
- Learning only from failures did not work here. The failure-only arm is a published idea; at this scale and with this reward it collapsed.
Why it matters, and what is next
A model can only teach itself past the point where human data runs out if checked experience is enough to learn from. This run says it is, at least one step past the frontier, and that the checking is what does the work: the same training with an uninformative reward did harm, not good. Earlier, at the same scale, fine-tuning on the model's own verified successes produced no measurable gain. Reinforcement against the checker did.
The next experiment lets the model choose its own level on an unbounded ladder and measures the slope: levels climbed per thousand verified attempts, against the same recipe with coin-flip reward, a fixed curriculum and an untrained copy. It is built and registered, with its decision rule written first, and its first run is pending.
Details
- Model: Qwen2.5-1.5B-Instruct. Training: GRPO, LoRA rank 64 on all seven linear modules, group size 8, 60 steps × 8 prompts = 3,840 rollouts per trained arm.
- Hardware: one free Colab T4, across several sessions with checkpoint and resume.
- Run: completed 18 October 2026. Ledger entry
rl_depth_loop_v1. - Every number on this page was printed by the run and checked a second way: per-arm accuracies decoded from the run's output reproduce each printed difference.