The classifier prompt I rewrote four times (and why)

Vamshi Krishna V.

Patient work, woven in public.

This is Thread 08 of Loomwork. New here? Thread 01 explains why this newsletter exists — 2 minutes.

I rewrote the same prompt four times. Here's the score it got each round:

0% → 29% → 0% → 65%

Look at the third number. My smartest version — cleanest category definitions, most careful rules — scored zero. Worse than the simpler prompt before it.

That's the moment this stopped being about the prompt. The model wasn't getting dumber between versions. I was measuring the wrong thing the whole time.

So here's the whole story: four real versions of one classifier, the exact failure mode each one shipped with, and the single sentence that finally fixed it. Nothing reconstructed — these are the versions I actually ran.

First, the job. Foundry's GitHub agent reads a pull request — title, body, diff — and labels it one of six things: feature, fix, refactor, docs, test, or noise. That label decides whether the PR surfaces in the next morning's briefing. A wrong label means the briefing lies to me. So it has to be right — and it has to be cheap, because it's a Haiku call (last week was the cost math on why).

(One caveat I'll put up front and stay honest about: this is a small, cheap eval — 17 PRs, and the "ground truth" is just each PR's commit prefix. Directional, not a benchmark. I get specific about exactly what that number does and doesn't mean at the end — and that turns out to be the most useful part of the whole exercise.)

To make the failure modes concrete before showing the prompts: here is what each version actually returned for the same PR — feat(chat): Phase D — Chat tab with Vercel AI SDK streaming. Ground truth: feature.

Same model. Same PR. Same temperature. Four prompts, four different output shapes. The semantic meaning was right in V1, V2, and V3 — the model knew the answer. The problem was the shape of the response, not the reasoning behind it.

v1 — the obvious first prompt.

Eval against 17 PRs: 0% correct. The failure mode was not what I expected. The model wasn't inventing categories — it was returning markdown-formatted explanations instead of a bare label word. Every response looked like **feature** or **feature**\n\n this PR introduces... Because my eval script checks for exact match against a set of six words, **feature** is not feature, and every answer scored wrong.

The wrong lesson: "the model can't classify." That's almost never the actual lesson.

The right lesson: the prompt had no output-format instruction. The model was in instruct-tuned mode and defaulted to explaining its answer in markdown. I gave it a classification task but forgot to tell it how to format the response.

v2 — add a fallback.

Eval: 29% correct. The markdown-formatting problem improved (the EXACTLY ONE of instruction shifted the model toward tighter output in some cases), but it still produced **label**\nreasoning... for most PRs. The five it got right were all the same shape — a bare feature on an unambiguous feat() PR (feat(catalog), feat(seo), feat(admin), two feat(auth) PRs). The moment the body gave the model something to explain, it explained — and the trailing text broke the exact-match check.

The Do not invent new labels instruction was addressing a problem that hadn't actually appeared yet — the model wasn't inventing categories, it was formatting. I was fixing the wrong problem.

v3 — separate user-visible from internal in the definition.

Eval: 0% correct. The user-visible distinction was the right semantic improvement — but the bullet-list format made things worse, not better. The model saw markdown bullets and switched fully into reasoning mode. Every single response was multi-line — the label wrapped in a markdown explanation, a bold **feature** trailed by a **reasoning:** block walking through the PR description.

I didn't see this until I wrote a small eval script that printed every misclassification with the raw model output side by side. The pattern was immediately obvious: 100% of V3 responses were multi-line, every one wrapping the label in a markdown reasoning or rationale block. Not one returned a bare word. Until I built that script, every prompt rewrite was a guess. After the script existed, the problem was obvious.

The real lesson of this iteration wasn't about the prompt. It was about the eval.

Prompt engineering is small experiments. The discipline is the eval, not the wording.

v4 — the working version.

Eval: 65% correct on 17 PRs — 11 right, 6 wrong. The 6 are worth pulling apart, because they aren't all the same thing. Some are the model overruling my ground truth and winning: a fix(agent) PR that was actually a full cloud rewrite — it called it feature, the better label than the fix prefix I'd assigned; a chore PR that only added a changelog — it called it docs, more honest than the noise my rule stamped on. Others are the model genuinely wrong: a feat PR that added a real new subsystem, which it under-called as refactor. So part of that 35% is the model being right while my labels were wrong, and part is the model actually missing. Telling those two apart by hand is the only reason the number means anything — and it's the work the accuracy figure alone hides.

The same model, the same temperature, the same eval set. Four prompts, and the only one that returns a bare word every time is the one that explicitly says Return: a single word, lowercase. The format instruction was the fix, not the semantic refinements.

The actual discipline.

Looking back at the four iterations, the thing that mattered most wasn't any of the four prompt rewrites — it was the eval script I wrote while debugging V3's surprising 0% that finally revealed V1 and V2's actual failure mode too. Without the script, I would have kept refining the semantic content of the prompt (better category definitions, clearer rules) while missing the fact that the model wasn't reading the categories at all — it was generating explanations.

Once I could see that every V1 and V3 response wrapped its label in markdown — a bold **feature**, often trailed by a **reasoning:** block — the fix was a single sentence: Return: a single word, lowercase, from the list above.

This is what prompt engineering actually is, in production. It is small experiments with measurable outcomes, run against a stable eval set you own, iterated until the failure modes become rare enough to ship. It is not sitting with a prompt for an hour and trying different wordings.

If someone tells you they "do prompt engineering" and they don't have an eval set, they're doing prompt typing. The eval is what separates the two practices.

(So what does "65%" actually mean? Less than it looks — and I'd rather say so than dress it up. The answer is half in the input, the sample is 17 PRs (out of 32 closed; the rest had no clean prefix to label from), and part of the "misses" are my own prefix-labels being wrong rather than the model. I'm not publishing this as a benchmark. I'm publishing it because a cheap eval I own taught me the real failure mode — output format — and then showed me my own labels disagree with my best prompt. That's the return on an eval, even a noisy one. The number is the least interesting thing in it.)

One real question before I go, and I mean it as a real one: if you put an LLM in front of users, what does your eval set look like? Not your prompt — your eval. The set of cases you run before you ship. I think it's the most under-discussed part of this entire job, and I'd genuinely like to know how you do it. I'll read and reply to every answer.

Next Sunday is the one I've been building toward: First agent, end to end. No mockups — the actual output. A real morning briefing the agent wrote, annotated line by line. The pipeline visualized. What a week of running it cost. What it got right, and the things it missed.

See you then.

— Vamshi

Loomwork is by Vamshi Krishna Veggalam. Building Foundry at mindloomhq.com. Find me on LinkedIn.