Why Your AI Pilot Never Ships

Your AI pilot worked. The model ran, the integration held, the demo impressed the team. Six months later it still hasn't shipped. Nobody killed it. Nobody approved it. It just lives in a Notion doc under "things to revisit."

This is what the AI deployment literature calls pilot purgatory, and the research is specific about the cause: pilots stall after technical success because no one agreed in advance what "done" or "successful" looks like.

The failure isn't the idea

Katila and colleagues reviewed 177 empirical studies on organizational experimentation and found that experiments matter for performance because they enable learning under uncertainty. The operative word is learning, not testing. An experiment without a decision rule is half-finished. It generates activity, not evidence.

The distinction sounds minor until you watch a founder spend four months iterating on an AI email tool with no defined target, no baseline measurement, and no agreed point at which the team commits or walks away. The tool might be working. It might not be. No one knows, because no one specified what "working" means.

What the exit rule actually does

The Experimentation Growth Model drawn from Microsoft, Booking.com, Skyscanner, and Intuit shows that scaling experimentation depends on codified processes and standardized metrics, not enthusiasm for testing. Every experiment in those organizations carries three elements: a clear goal, a fixed overall evaluation criterion, and an agreed implementation decision. The implementation decision follows threshold logic: if the primary metric improves and guardrails hold, ship it. If not, stop.

The exit rule is not about discipline for its own sake. It forces the team to treat the pilot as a real decision before the test begins. "If we don't reduce average email response time by 30% in 7 days, we shut it down" is not a constraint on ambition. It is the only mechanism that converts a test into a commitment.

When the exit rule punishes the wrong question

The strongest objection to this approach is worth taking seriously rather than dismissing.

Pre-defined KPIs draw their authority from the online controlled experiment literature, which was built on high-volume consumer products where the outcome is unambiguous and observable within days. A founder running an AI experiment on email response time faces a different condition: they often don't know which metric matters most before the pilot begins. Forcing a single KPI before understanding the system's behavior selects for the most measurable outcome, not the most important one. The research names this failure mode directly: strict success rules risk pushing teams toward incremental local gains rather than deeper product innovation.

This objection has real weight. A fixed KPI is only as good as the founder's prior understanding of the process being changed. Setting a 30% response time target without first measuring baseline response time is not a disciplined hypothesis. It's a guess formatted as a rule.

The rebuttal is not that the objection is wrong. It is that the alternative is worse. The AI pilot literature documents what unstructured exploration actually produces: experiments stuck indefinitely because no one agreed what success looks like. Katila et al. also show that structured experimentation supports exploration of new domains, not just incremental optimization. The design does not restrict what question you ask. It restricts only the absence of an answer.

Before you set the KPI, measure the baseline

The practical implication the research points toward is a step founders routinely skip. Before writing the exit rule, spend two days measuring what currently happens. If you don't know your average email response time, you cannot set a credible 30% improvement target. The baseline observation is what makes the threshold meaningful rather than arbitrary.

Once you have the baseline, the seven-day structure follows directly. One metric. One threshold. One binary outcome. The Katila et al. review shows that experiments framed as structured causal tests, with a specific change tied to a measurable outcome, produce better results than loosely defined trials. The seven-day window forces a decision before the team normalizes the ambiguity.

Founders who skip this step don't run experiments. They run extended demos with no closing date, and the research is consistent about where those end up.