At 2:14 in the morning, my fine-tuned model beat the baseline on every metric I had, and I sat there feeling nothing at all.
Accuracy up. F1 up. The eval dashboard was a wall of green. I should have been happy. Instead I had the specific, sinking feeling you get when a number agrees with you a little too eagerly — the way a friend agrees with you when they have already decided not to argue.
I shipped it anyway. Of course I did. Three days later the people actually using it told me, politely, that the new model was worse.
They were right. It took me a week to understand why, and understanding why rearranged how I think about this entire craft. So let me save you the week.
The first uncomfortable truth: your eval set is usually the same lie, told twice
Here is what I had done, without noticing I was doing it. I exported a pile of historical tickets. I split them into train and test. I trained on one half, evaluated on the other, and celebrated when the numbers went up.
But both halves came from the same export. The same six-month-old snapshot. The same handful of tired humans who had labelled them, at the end of long shifts, using a taxonomy that had quietly changed twice since.
My model didn’t learn the task. It learned the annotators. And then my eval, drawn from the same well, cheerfully confirmed that it had learned the annotators very well indeed.
A held-out split is not an independent opinion. If your test set was born in the same room as your training set, it will inherit its parents’ blind spots — and then congratulate you for having them too.
This is the part nobody puts in the tutorial, because the tutorial has to end with the loss curve going down.
What fine-tuning actually does — and what everyone thinks it does
Ask ten engineers why they want to fine-tune and eight will say some version of: “so the model knows our stuff.” Our docs. Our pricing. Our product names. Our 400-page policy PDF.
That is almost exactly what fine-tuning is worst at.
The LIMA paper took a 65B LLaMa and fine-tuned it on 1,000 carefully curated examples — no RLHF, no preference model, nothing fancy — and got startlingly strong instruction-following. The authors’ explanation is the part worth tattooing somewhere: alignment may be a process of teaching the model the style and format of interaction, so that it can expose knowledge it already learned during pretraining. They called it the superficial alignment hypothesis. “Superficial” is doing a lot of quiet work in that phrase.
And when researchers tested knowledge injection head-to-head — Ovadia et al., “Fine-Tuning or Retrieval?” — retrieval consistently beat unsupervised fine-tuning. Not just for brand-new facts. For facts the model had already seen in training. Fine-tuning struggled to absorb new information; RAG just handed it over at inference time and moved on with its life.
So here is the sentence I now say in every kickoff meeting, usually to visible disappointment:
Fine-tuning teaches your model manners, not facts. You don’t send an employee to elocution lessons to make them memorise the price list.
If your complaint is “it doesn’t know our current pricing” — that is a retrieval problem, and no amount of gradient descent will make it a fine-tuning problem. If your complaint is “it knows the answer but says it like a LinkedIn post,” now we can talk.
The truth nobody says out loud: you can only inherit an accent, not an education
The most popular fine-tuning recipe on the internet is: get a big model to generate a few thousand answers, train your small model on those answers, watch your small model become shockingly good.
It does become shockingly good. That’s the trap.
“The False Promise of Imitating Proprietary LLMs” is the most quietly devastating paper I’ve read on this. The researchers trained imitation models, and crowdworkers rated the outputs as competitive with ChatGPT. Then they ran targeted automatic evaluations and found the imitation models closed little to none of the actual capability gap on tasks that weren’t well represented in the imitation data.
Read that twice. The humans said it was as good. The benchmarks said it wasn’t. The imitation model had learned the confident tone, the tidy bullet points, the helpful preamble — and none of the underlying reasoning.
Which means your human eval, the one you trust most, the one where you ask five colleagues “which output do you prefer,” is the single most style-biased instrument in your toolbox. People grade fluency. They grade formatting. They grade confidence. They do not, in a 90-second side-by-side, grade whether the answer is quietly, elegantly wrong.
Fine-tune on a bigger model’s outputs and you buy its accent, not its education. And then your evaluation process — staffed by humans, who are impressed by accents — will hand you a rosette for it.
The money nobody counts
Let’s do the arithmetic honestly, because the industry keeps quoting the wrong half of it.
Compute is a rounding error. QLoRA demonstrated fine-tuning a 65B model on a single 48GB GPU, and their Guanaco family reached 99.3% of ChatGPT’s performance level on the Vicuna benchmark with 24 hours of fine-tuning on one GPU. For the 7B-class LoRA runs most teams actually do, you are looking at a few hours on a rented A100. Call it the price of a decent dinner.
Data is the entire bill. Say you want 5,000 genuinely good examples. Not scraped — good. Written or verified by someone who understands the task. Budget four minutes each, which is optimistic the moment anything requires judgement. That’s 333 hours. Two months of one person’s undivided attention, and nobody’s attention is undivided.
So the real shape of a fine-tuning project looks like this: tens of dollars of compute, tens of thousands of dollars of human attention. Roughly three orders of magnitude apart. Every conversation I’ve been in budgets carefully for the first number and hand-waves the second.
And then the part that isn’t in the proposal at all
Here’s the truth I have never once seen on a vendor slide:
A fine-tune is a subscription, not a purchase.
Every fine-tuned model is a frozen bet that your task distribution will hold still. It won’t. You’ll add a product line. Support will rewrite the category taxonomy. Marketing will change the tone of voice. A regulation will land. Some upstream team will start sending you a new field.
None of these break your model loudly. That’s the cruel part. It doesn’t crash. It doesn’t throw. It just gets a little more wrong every month, in the exact places where nobody is looking, and your dashboard — still evaluating against that snapshot from last spring — keeps showing green.
Model rot is silent, and it is the only failure mode I have never seen anyone budget for.
The thing you should actually be scared of
If you take one link from this post, take this one.
Qi et al. (2023) stripped the safety guardrails off GPT-3.5 Turbo by fine-tuning it on 10 adversarially designed examples, at a cost of less than $0.20 through the official API. Ten examples. Twenty cents.
And the finding that should genuinely change your behaviour is the one in the title: even when users do not intend to. Their work shows safety alignment degrading through fine-tuning on benign datasets. Not attacks. Ordinary, well-meaning, boring training data.
So: fine-tuning can silently uninstall safety behaviours you didn’t know you were depending on. The base model came with a lot of expensive alignment work baked in, and you are now, gently, sanding it off. The moment you ship a fine-tune to real users, the alignment is yours. You own it. Nobody hands you a warning.
LoRA is not “full fine-tuning, but cheaper”
This one cost me real time to internalise. “LoRA Learns Less and Forgets Less” (Biderman et al., TMLR 2024) is exactly what the title says. LoRA underperforms full fine-tuning on the target domain — they measured this on code and math — but it forgets less of everything the model knew before. The mechanistic reason is lovely: full fine-tuning produces high-rank weight updates, on the order of 10–100× the rank a typical LoRA config even has available. LoRA physically cannot make that large a change. So it doesn’t.
Most people read “learns less” and hear worse. Wrong frame.
LoRA’s weakness is the feature. The low-rank constraint is a leash, and the leash is what stops your model from forgetting how to be a model.
The paper found LoRA mitigates forgetting better than weight decay or dropout, and keeps generations more diverse. So the decision isn’t “can I afford full fine-tuning?” It’s: do I want a generalist who has picked up a new habit, or a specialist who has forgotten how to do anything else? Most products want the first one and reflexively reach for the second.
If you do go LoRA, their practical recipe held up for me: set alpha to 2× the rank, target all transformer modules rather than just attention, and actually sweep your learning rate across roughly 1e-5 to 5e-4 instead of copying whatever was in the blog post you found.
The three questions I ask before touching a GPU
1. Is this a knowledge gap? The model doesn’t know something. Then it is retrieval, and it was always retrieval. (Ovadia et al. above.) Fine-tuning here is an expensive way to memorise something you could have looked up.
2. Is this a behaviour gap? The model knows the answer and delivers it badly — wrong format, wrong tone, too long, ignores your schema, won’t shut up. This is what fine-tuning is genuinely, brilliantly for. It’s worth remembering that in the InstructGPT paper, outputs from the 1.3B model were preferred by human evaluators over the 175B GPT-3 — a model 100× larger. Not because the small one knew more. Because it had been taught how to behave.
3. Is this a reasoning gap? The model genuinely cannot do the thinking. Then no amount of fine-tuning saves you. Get a better base model, or break the problem into steps a smaller model can actually hold. This is precisely where imitation models fail, and precisely where teams burn a quarter finding out.
And the silent fourth question, the one that decides everything: can I measure this? If you cannot describe, today, the specific eval that would tell you the fine-tune failed — you are not doing engineering. You are gambling with extra steps and a nicer dashboard.
What actually worked
Back to my 2 a.m. model. Here’s what fixed it, and I want to be honest that none of it is glamorous.
I wrote the eval first, and I wrote it from last week. Not a random split of the old export — a fresh sample of recent tickets, labelled by the person who actually owns the outcome, who could say “no, that’s not what that category means anymore.” The eval set became the most valuable artifact of the entire project. It outlived the model.
Then I deleted most of my training data. I had two people independently label a slice of it, and I threw away every single example where they disagreed. It felt insane. I watched the dataset shrink by well over half and had to physically stop myself from putting it back.
But the logic is airtight, and it’s the thing I most wish someone had told me three years earlier:
If two competent humans can’t agree on the label, there is no signal in that example. There is only noise wearing a label’s clothing. Train on it and you are teaching your model to imitate a coin flip — with total confidence.
The training run itself took under an hour. It always does. The month of work was upstream of it, in a spreadsheet, arguing with a colleague about what “billing issue” actually means. That argument was the project. The GPU was a formality.
(A note on this section: swap in your own numbers here — accuracy before and after, latency, cost per request. Real figures from your own run will land far harder than anything I could describe in the abstract.)
The short version, if you skipped to the end
- Fine-tuning teaches form, not facts. Knowledge gaps are retrieval’s job.
- Your held-out split is not an independent judge if it shares parents with your training data.
- Human preference evals reward style. Imitation models are extremely good at style. Do the math.
- Compute costs tens of dollars. Data costs tens of thousands. Budget accordingly — almost nobody does.
- Fine-tuning can degrade safety alignment even with completely benign data. Ten examples, twenty cents, guardrails gone.
- LoRA learns less and forgets less. Often the second half is the half you wanted.
- A fine-tune is a subscription. The rot is silent. Plan the re-training before you plan the training.
- Delete the examples your annotators disagree on. All of them. Yes, really.
I still fine-tune models. I fine-tune them often, and when it’s the right instrument it is a genuinely beautiful one — a small model, taught to behave, quietly outperforming something a hundred times its size for a fraction of the latency and cost. That is a real and repeatable win.
I just no longer start there. I start with the eval, and the eval usually tells me I didn’t need a GPU at all.
That, more than anything, is the part nobody says.
References
- LIMA: Less Is More for Alignment — Zhou et al., 2023. 1,000 curated examples; the superficial alignment hypothesis.
- Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs — Ovadia et al., 2023. RAG consistently beats unsupervised fine-tuning for knowledge.
- The False Promise of Imitating Proprietary LLMs — Gudibande et al., 2023. Imitation closes the style gap, not the capability gap.
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! — Qi et al., 2023. 10 examples, <$0.20, guardrails removed; benign data degrades safety too.
- LoRA Learns Less and Forgets Less — Biderman et al., TMLR 2024. The low-rank constraint as an implicit regulariser.
- QLoRA: Efficient Finetuning of Quantized LLMs — Dettmers et al., 2023. 65B on a single 48GB GPU; Guanaco at 99.3% of ChatGPT on the Vicuna benchmark.
- Training Language Models to Follow Instructions with Human Feedback — Ouyang et al., 2022. The 1.3B model humans preferred over the 175B one.
If you’ve shipped a fine-tune that quietly went wrong — or one that quietly went right — I genuinely want to hear about it. The failure modes are where the real curriculum lives, and almost nobody writes them down. Find me on LinkedIn or GitHub — I read everything.