The Workshop
A 4B model, fine-tuned to write pytest suites that expose bugs, measured against the same model prompted, on 315 functions it could not have seen. In order of what moved the metric: the harness, then a paragraph of prompt, then twelve thousand teacher examples that added nothing on top of it, then four rounds of self-training that added nothing at all.
Experiments
- 001The metric came firstBefore any training I built the thing that would judge it: a sandbox that runs a generated suite against the correct function and against dozens of deliberately broken copies, a pool of real-world functions written after the model's training cutoff, and paired statistics. The pilot said the metric could tell a good suite from a bad one. The baselines said validity, not bug-finding, was the headroom.
- 002I gave the model the right answers and it got worseThe first fine-tune trained on the model's own suites with the wrong expected values corrected by execution. Validity fell 4 points against zero-shot prompting, on an interval that includes zero. The mechanism was in the data: I had taught the model to state exact values with confidence, and at inference it still could not compute them.
- 003Preferences, traces, and the one skill I could not installTwo more signals aimed at the same bottleneck. Preference pairs built from the model's own passing and failing suites: no measurable change, with an interval that would have missed a gain below five points, and the assertion mix returned to the base's. Execution traces rendered above each assert as a scratchpad: a validity gain that turned out to be brevity, and derivations that were fluent and invented. Three signals, two bases, and the per-assert accuracy never moved.
- 004So I stopped asking it to predict outputsIf the model cannot compute expected values, let the harness compute them, in every arm, base included. Filling values by execution was worth 28 validity points on its own. A fine-tune on top of it, on the model's own pairs, was worth +0.024 with an interval that included zero. I pre-registered the bar before the run, and the run failed it.
- 005Four thousand examples from a stronger teacherThree research passes said the recipe was fine and the data was the problem: every working small-model recipe uses a stronger teacher at a scale I had refused on purpose. Four thousand execution-verified examples from a public GPT-4o dataset, curated through my own harness, cleared the pre-registered bar. Twelve thousand repeated the gain and added nothing to it. Then a control that three reviewers asked for, the base prompted for the teacher's style with no training, scored higher than both adapters.