Five Questions Before Your Next AI Pilot

You bought the tool. You ran the pilot. Six weeks later, nothing changed except your budget. The vendor called it a success.

Between 70% and 95% of AI pilots fail to reach production or deliver measurable business value, according to research from RAND, MIT Sloan, Gartner, McKinsey, and BCG. That range is wide, but the direction is not ambiguous. Most pilots die. The question worth asking is where they die — because the answer changes what you should do before you sign the next contract.

The failure happens before week one

The instinct is to blame execution. The team didn't adopt it. The onboarding was poor. The budget ran out before the rollout finished. These are real causes, and they kill real pilots. Execution failures are worth taking seriously.

But OECD and GPAI research surfaces a different problem upstream: 21.4% of AI adopters cannot identify a suitable problem for AI before they attempt adoption. Not during the pilot. Before it. These founders are not failing at execution — they have not reached execution yet. They selected a tool without knowing what the tool was supposed to fix.

That 21.4% figure matters because no amount of change management reaches it. You cannot onboard your way out of a pilot built around a problem you never clearly defined. The failure was in the selection decision, not in what followed.

What the five questions actually do

The diagnostic is not a checklist. Each question maps to a documented failure mode.

What data do you have? AI tools do not manufacture signal from noise. A customer churn model trained on incomplete CRM records produces confident wrong answers. If your data is incomplete, inconsistent, or siloed across systems that do not talk to each other, the tool fails regardless of its benchmark scores.

Who will use it? This is the question founders skip most often, and skipping it is why change management failures look like execution problems. If the people who are supposed to use the tool were not involved in the decision to select it, resistance is not a surprise — it was baked in at selection.

What is your biggest pain point? Vendors sell horizontal tools. Your problem is specific. A general-purpose AI writing assistant does not fix a broken sales qualification process. Naming the pain point precisely forces a match between the tool's actual function and the problem you are paying to solve.

Can you measure impact? If you cannot state what success looks like in numbers before the pilot starts, you cannot tell whether the pilot worked. Pilots without defined metrics do not fail — they drift. Drift is harder to stop than failure.

Do you trust the vendor? I find most AI vendor due diligence embarrassingly thin. Founders who would spend three weeks negotiating a SaaS contract sign AI pilots after a forty-minute demo. The vendor's data handling practices, their model update policies, and their actual customer retention rate all affect whether the tool works in six months the way it works in the demo today.

The counterargument deserves a fair hearing

The strongest objection to this framing is that execution failures dominate the 70–95% failure rate, not pre-selection failures. McKinsey and BCG, two of the sources behind that figure, document skills gaps, change management shortfalls, and inadequate organizational buy-in as failure drivers — all of which occur after a tool has been chosen. Under that reading, a pre-selection diagnostic screens for the wrong problem.

This objection holds for a portion of the failure population. Pilots do die during execution. The five questions do not prevent that. A founder who confirms data quality, user buy-in, a specific pain point, a measurable outcome, and vendor accountability before selecting a tool can still watch the pilot collapse if the team reverts to the old process in week three.

What the objection cannot explain is the 21.4% figure. That group never reached execution. And the user adoption question in the diagnostic exists precisely because user resistance is predictable when you select a tool without confirming the people who will use it agreed to change. The change management failure that follows is a consequence of the selection decision, not a separate event.

Where this leaves you

Run the five questions before your next pilot. Not as a formality — as a filter. If you cannot answer "What data do you have?" with a specific source, format, and quality assessment, you are not ready to select a tool. If you cannot name who will use it and confirm they want to, you are selecting a tool for a team that has not agreed to change.

The 70–95% failure rate has not moved despite better tools, better vendors, and better onboarding programs. Pre-selection decisions are where the math is broken.