Four thousand examples from a stronger teacher

After 004 I asked myself a question I should have asked after 002: is this a ceiling of the model, or something I am doing wrong? I do not trust my own answer to that, so I had three research passes done, with the instruction to be sceptical and to cite.

What the practice review said

The first pass read how the groups that fine-tune small models for real actually do it, from their own reports. Qwen2.5-Coder: 200,000 synthetic instruction pairs. Llama 3: 2.7 million synthetic code examples, filtered by a reward model trained on human labels. Tülu 3: 939,000 prompts. Nemotron Nano 2: about 11 million supervised samples. In every one of them the data comes from a stronger model or a human-trained judge, and at two to three orders of magnitude more than my 388 to 645 examples. No source showed a self-generated loop working at small scale.

The second pass looked for the self-improvement literature directly. The paper that mattered was "Mind the Gap" at ICLR 2025: self-improvement is bounded by the gap between what a model can generate and what it can verify, and that gap scales with pretraining compute and sits near zero for models in my size class. ReST-EM's gains scale with model size. STaR could not bootstrap GPT-2. My four nulls were a flat dose-response from 388 to 645 self-generated examples, which is what that literature predicts.

The third pass audited my recipe line by line against the mlx-lm source and the current LoRA guidance. The adapter scale semantics, the choice of all linear projections, the learning-rate schedule and the optimizer-step accounting, completion-only loss masking, the chat template at training and inference, the checkpoint selection: nothing was broken. Two small things: the DPO rank of 8 sits below the default band, and DPO with no supervised warm start and mostly coarse pairs is a known way to get the "training fits, held-out flat" pattern I had seen. Neither would turn +0.024 into a resolved gain.

So: not a model ceiling, a data ceiling. And the thing I had ruled out in entry 002 on purpose, a stronger teacher, was the only lever with published evidence behind it.

The ladder

I wrote down three stages, cheapest first, each pre-registered on the same test split with the same success rule as 004, and a stop at the first resolved gain. One: an existing dataset of function and test pairs, curated through my harness. Two: a local 30B-class coder as teacher, sampling suites for my training functions. Three: online reinforcement learning with the harness as the reward. Same base model throughout, because switching base would have confounded the comparison with four earlier iterations.

Stage one survey. Two more research passes over Hugging Face, papers, GitHub and Kaggle. Most of what calls itself a unit-test dataset is Java, or repository-level, or an I/O judge for competition problems. Three survived: KodCode-V1, 487,000 synthetic problems each with a GPT-4o solution and a pytest module verified by execution, licensed CC BY-NC 4.0; NVIDIA's OpenCodeInstruct, five million rows with assert lists, CC BY 4.0; AceCode-87K, MIT, assert lists. KodCode's tests import their function with from solution import name, which is my inference format exactly. Every one of the three is model-generated, so stage one was teacher distillation with a frontier teacher at scale, whatever I called it. If that worked, stage two was redundant; if it did not, a weaker local teacher would not do better.

I took KodCode and accepted the non-commercial licence. The adapters inherit it and the notice says so.

Curation

I did not want to train on KodCode as shipped. I wanted training examples that would have passed my own held-out harvest, so the dataset went through the same filters as my evaluation pool.

Stage A, syntax only, over 169,000 rows from the five subsets that read like utility functions rather than puzzles: the solution must be imports plus exactly one top-level function that passes the same purity filter as the held-out pool, no I/O, no globals, no third-party imports; the test must import only the solution, pytest and the standard library; the test is cut to its first eight functions; the function must yield at least eight live mutants. 44,527 rows survived.

Stage B, execution, on a seeded random sample: the suite must pass on its own solution and kill at least one mutant of a training category. 24,000 sampled, 21,085 passed. 88 percent, which tells you the dataset's own verification is good.

Stage C: n-gram and AST decontamination against both held-out splits, 14 removed; exact-AST duplicates dropped, 7,025; fit within 1,024 tokens, 13,429; then the top 4,000 by mutation score and then by fewer tests. Mean mutation score of the kept set 0.991, 5.7 tests and 644 tokens per example, 89 percent of asserts exact-value literals. Held-out is code written after June 2026 and KodCode is from March 2025, so direct leakage is impossible, and I ran the check anyway.

Why 4,000 and not more. The hybrid base trains at about 35 tokens per second, and I had checked that no newer mlx-lm release had added the missing kernel. Four thousand examples was one epoch in about ten hours. I would rather have one clean point than a run that ends on Wednesday.

The result

Same recipe as entry 002, same base, one epoch, a checkpoint every 400 iterations, selection on dev-171 by grounded score. Four checkpoints scored 0.601, 0.589, 0.623 and 0.630 against the base's 0.621; the last was chosen.

Test, 315 functions, grounded harness in both arms:

The lower bound clears zero. Against few-shot, +0.064 on [+0.016, +0.114]. Grounded validity +0.086, McNemar p = 0.002, with 51 functions valid only for the fine-tune and 24 only for the base. Mutation score on the 202 both-valid functions −0.023, [−0.048, +0.001]. Unaided validity +0.022, unresolved. Arithmetic probe: 258 kills against 206, no drift.

The bar was met. Then I ran the second point. The ladder's stop rule was for its stages, and stage one had resolved, so stages two and three were off. The second point was a scale question inside stage one, registered before it ran: the yield table said the same pipeline had 12,000 rows at the same quality bar, and the first objection to a 4,000-example result is that it is small.

Twelve thousand

Same pipeline, 12,000 more executed rows, a 12,000-example cut whose mean mutation score was 0.937 rather than 0.991, because the cut reaches further down the ranking. One epoch, about thirty hours. Dev curve 0.646, 0.596, 0.633, 0.596; checkpoint 2400 chosen.

The gain repeated on an independent curation sample and training run, scored on the same 315 test functions. That replicates the data cut and the training, not the evaluation: the test split is shared, so the two intervals are not independent draws. Three times the data added nothing measurable. Either this kind of data saturates at a few thousand examples, or the lower selection quality of the larger cut offsets its size; the two are confounded in this design and I have not separated them. The 12,000 model does lean less on the harness, 520 literals rewritten against 798, and its unaided validity is 0.467, still unresolved against the base.

The control, and what it did to the headline

Three reviewers read the draft of this entry and asked the same question in three voices: the adapters write short suites of exact-value asserts, that is exactly what the harness repairs, and you never prompted the base for that style. So I registered the control before running it, with its reading rule: the base model, no training, a system prompt asking for one short test per behaviour with the exact return value and at most five tests, and two KodCode training examples as exemplars. If its gain over zero-shot cleared zero and the adapters' gain over it did not, the headline would be that the style is worth the points and prompting gets them.

The control clears zero by a wider margin than either adapter. The 12,000 adapter against the control: −0.014, [−0.051, +0.022]. The 4,000 adapter: −0.017, [−0.057, +0.022], and its grounded validity is lower by 0.057 with p = 0.02. The control also moved the number no fine-tune had moved: unaided validity 0.571 against 0.438, +0.133 with an interval from +0.073 to +0.190. It writes 4.65 tests per suite, 82 percent exact-value asserts, and the harness had to rewrite 247 literals for it against 520 for the adapter, because a suite with fewer asserts has fewer to get wrong.

One more row, because the question follows immediately: the best adapter with the best prompt. The 12k adapter under the style prompt scores 0.667, the same number it scores under the plain prompt, and −0.014 against the control with an interval that includes zero. The prompt raises the adapter's grounded validity by four points and lowers its kills per valid suite by four points, and the two cancel. The adapter and the prompt teach the same thing; stacking them buys nothing.

So the reading rule fires the way the reviewers predicted. The six points were the style. Asking for the style gets all of them and more, and the two adapters match the prompt without beating it. That is the headline of this entry now, and the one I had written before the control was wrong.

What the model learned

Style. The adapters write 4.7 to 4.9 tests per suite where the base writes 7.7, and 85 to 89 percent of their asserts are exact-value literals where the base's are 72 percent. Fewer membership asserts, fewer truncated suites. Under a harness that fills literals, that is the optimal style, and the harness rewrote about twice as many values per suite as it did for the base. What remained invalid after filling fell from 89 suites to 62 and 56.

Unaided validity did not resolve. The model did not learn to predict outputs. It learned to write suites the harness can complete, and to choose inputs that exercise more of the code. The loss in mutation score is the price of shorter suites: on the suites each arm gets valid, the adapters kill 0.826 and 0.811 of the live mutants where the base kills 0.842, and on the 202 functions both arms get valid the paired difference is −0.023. The net over all functions is the six points, and it is all validity.

Correction, 14 September 2026. The first version of this table listed 0.868 and 0.843 in the mutation-score column for the two adapters. Those were the mutation scores of the suites valid as written, copied from the wrong field of the run summary. The grounded column now shows the values on grounded-valid suites, which the grounded score divides into exactly. Two reviewers caught it by checking that grounded validity times mutation score equals the grounded score, which it now does on every row.

What this is and is not: it is a repeated, pre-registered, six-point gain in mutants caught per function, under the harness, over the same model prompted zero-shot, and it is matched by a system prompt. It is not evidence that a 4B model learned execution prediction, and it is not evidence that fine-tuning was needed.

What this does not say

Not that data scale is the lever in general. Scale beyond a few thousand did nothing here. The lever was the source: examples whose expected values were correct by construction, written by a model that can compute them, filtered by execution.

Not that stages two and three were unnecessary in principle. They were unnecessary for this claim. Reinforcement learning with the harness as reward is the one untried angle with a published small-model result behind it, and it is still on the list.

Threats to validity

Two runs, one checkpoint selection each on 171 functions, one test evaluation each. The intervals are wide; the true effect could be half or twice the point estimate. The chosen 4,000 checkpoint led the base by 0.009 on dev and by 0.059 on test, and two of its four checkpoints sat below the base on dev, so checkpoint selection is drawing from a curve as wide as the effect. Across the whole series the same 315 functions have been scored against the same base arm eight times with no multiplicity correction; under a Bonferroni-style correction the 4,000 lower bound would not clear zero and the 12,000 one barely would. Selection on mutation score favours simpler functions and the training distribution is synthetic GPT-4o style, which is not the distribution of the held-out pool. Layer-2 embedding decontamination was skipped before training, on the argument that 2025 synthetic data cannot contain post-2026 GitHub. A reviewer pointed out that this answers leakage and not paraphrase, so I ran it afterwards: 2 of 11,756 rows exceed the 0.90 flag, both GPT-4o rgb_to_hex solutions against one held-out rgb_to_hex, and 9 sit in the review band, canonical utilities like format_size and slugify. Four held-out functions of 486 are touched. The adapters were trained with those rows in; the report is committed with the data. The adapters are non-commercial. The gain exists only under the grounded harness.

The 12,000 point confounds scale with selection quality, as above.

How to check this instead of trusting me

git clone https://github.com/Catafal/llm-test-generation
make setup && make sync-models && make models-pull KEYS="4b-bf16"
make adapters-pull
make demo ID="NanmiCoder/open-image-prompts:retrieval/engine.py::weighted_tag_similarity" FROM_RUNS=1

The adapters are on the Hub under jorcagra/qwen3.5-4b-testgen-lora-ext12k and -ext4k, with the model card as their README. The eight test-split generation files behind the tables are committed, so the demo replays the exact suites the numbers came from without a model. Generation itself needs Apple Silicon; everything else runs anywhere.

What I take from this

Five iterations and a control. The recipe was never the problem: the audit found nothing broken, and the run that moved used the same adapter, the same layers, the same learning rate as the first one that did not. What changed was who wrote the training examples, and what the examples taught was a style that a system prompt teaches for free.

I refused a teacher in entry 002 because I wanted the interesting story. The interesting story turned out to be smaller and sharper than either version I had in mind: a 4B model cannot lift itself by its own outputs on a task that needs a skill it lacks; it can be taught a style by four thousand examples; and it can be asked for the same style in a paragraph. The control that showed the last part was the cheapest run in the series, and I ran it last because three strangers told me to.