10/22/2026
If your fine-tuning comparison rests on one training run, your confidence interval is answering the wrong question.
That's the main finding from Part 2 of my Laya field report, now on Medium.
I fine-tuned a self-hosted Laya model on 11,010 human-reviewed record-linkage pairs. On a second client's real data, it reached a hosted model's precision band. Then I pre-registered a retrain that added six domain rules to the training question. At one seed, recall fell 21.9 points. At a second seed of the identical recipe, it rose 4.9. Both intervals cleared zero in opposite directions.
Both were valid intervals. They just measure label sampling at a fixed checkpoint, not variation across retraining runs. In my runs, changing only the seed moved out-of-sample recall by 9 to 18 points, which is as large as most of the training effects I had been reporting.
What holds up: fine-tuning made the local model competitive on this client's data.
What doesn't: any single-seed claim about the recipe, including the rules.
Next up are seed budgets, calibration, and routing at matched review load.
https://medium.com//i-fine-tuned-a-local-model-to-match-a-hosted-model-35a4a8a650db
Fine-tuned on the same 11,010 human-reviewed pairs my production auditors learned from, a self-hosted Laya model reached the hosted model’s…