A retrieval assistant degrades silently: the answers stay fluent while they stop being correct. Write fifty real questions with client-approved answers before building, run them on every change, and you get a number that moves instead of a feeling that something is off.
An eval is a fixed set of real questions with approved answers, run automatically on every change to the system. It exists because a retrieval assistant fails silently — the prose stays confident while the facts stop being right — and without a number, nobody finds out until a customer does.
The failure mode that costs the most is not the assistant refusing to answer. It is the assistant answering beautifully and being wrong, six weeks after launch, to a customer who believed it.
#Why retrieval quality degrades without anyone touching it
Nothing has to break for a retrieval system to get worse. The document set grows and the relevant passage now ranks fourth instead of first. Somebody uploads a superseded policy PDF without deleting the old one. The model provider ships a new version of the same model name. Each is invisible, and each moves the answer.
Traditional monitoring cannot see any of this. Every request returned a 200. Latency is fine. The logs are clean. The only symptom is that the answers are quietly less true than they were last month.
#What actually goes in the test set
Fifty questions, pulled from the client\'s real support queue or sales inbox. Not generated. Real questions carry the phrasing, the typos, the local terminology and the half-stated assumptions that synthetic ones never do, and those are precisely the things retrieval struggles with.
Each gets an approved answer written by whoever currently answers that question for a living. That person is not us. We do not know which of two plausible policies applies to a customer in Sharjah, and pretending otherwise is how an eval ends up validating our own misunderstanding.
#Letting the client grade it
This is the part that gets pushback, and it is the part that matters. We send the client a scoring sheet: the question, the assistant\'s answer, and three buttons — correct, incomplete, wrong. No scores, no jargon, no confidence intervals.
Two things happen. The client discovers what the system genuinely does, rather than what a demo suggested. And we get a number owned by the person who will be embarrassed if it drops. That second thing is worth more than the first.
#When to run it
On every prompt change, every model upgrade and every index rebuild. Those three events cover almost every real regression we have seen, and all three are moments where someone is already waiting for a result, so the eval costs no extra calendar time.
- A wrong answer from an LLM reads exactly like a right one. Fluency is not a quality signal.
- Fifty real questions from the client's own support queue beats five hundred synthetic ones.
- The client grades the answer key, not the engineer. They are the only ones who know what "correct" means here.
- Run the eval on every prompt change, model upgrade and index rebuild — those are exactly the moments quality moves.
