Why AI projects fail (and the one afternoon that prevents it)

Most AI projects don't fail on the model. They fail because nobody wrote down what a correct answer looks like before building. Twenty real questions with verified answers, collected in an afternoon, prevents most of it.

Every failed AI project I have been brought in to rescue has the same shape. The demo worked. Everyone saw it work. Six weeks later nobody uses the thing.

The demo is the problem, not the proof

A prototype gets tested on five clean documents and three questions whose answers everyone already knows. It performs. Budget gets approved on the strength of that.

Then it meets the real corpus. Scanned PDFs with no text layer. Three versions of the same policy, none marked current. A spreadsheet where the actual answer lives in a comment on cell F14. Someone asks a question phrased the way people genuinely phrase things, and the answer comes back confident and wrong.

Confident and wrong is worse than nothing. It costs trust that you don't get back.

The gap is measurable, and almost nobody measures it

Here is the uncomfortable part. Most teams have no way to answer the question "is this better than last week?" They rely on someone senior trying it for a few minutes and saying it feels sharper. That is not a signal. It is a mood.

Without measurement, every change is a guess. You tune a prompt, retrieval quietly degrades on a category of question nobody tested, and you find out from a customer three weeks later.

The fix takes an afternoon

Before writing any code, do this:

1. Get twenty real questions from the people who will actually use the system. Not questions you invented. Questions they asked last week, in the words they used.
2. Answer all twenty yourself, correctly, and write the answers down.
3. That is your test set. Every change from then on gets measured against it.

It sounds too simple to matter. It changes the entire project.

What you learn immediately

Usually that three or four of your twenty questions cannot be answered from your documents at all. The information isn't there, or it contradicts itself, or it lives in someone's head.

That is a content problem. No model fixes it. Finding it in week one costs an afternoon; finding it in week six costs the project.

Why this step gets skipped

Because it doesn't look like progress. There is no demo at the end of it. Someone senior wants to see something working by Friday, and writing down questions doesn't photograph well in a status update.

So teams skip it, spend six weeks tuning something they cannot evaluate, and ship because the deadline arrived rather than because it worked.

What to ask before you sign anything

If you are hiring someone to build this, ask them one question: how will we know it's working?

If the answer involves evaluations, a test set, and a number that goes up or down, they have done this before. If the answer is about model quality or a framework name, they haven't.