Your AI Pilot Isn't Failing Because the Tool Is Wrong

Forty-two percent of organisations abandoned most of their AI initiatives in 2025. A year earlier, that number was 17 percent. The jump is not evidence of worse tools. It is evidence of founders finally noticing they had no way to know whether the tools were working.

The measurement problem hiding inside every pilot

MIT's Project NANDA reviewed more than 300 AI implementations and found 95 percent produced zero measurable profit-and-loss impact. Not marginal impact. Zero. The same research identified why: pilots ran "off the books," meaning no one attached a number to what success would look like before the experiment started. RAND Corporation's 2025 analysis found that more than a quarter of AI project failures delivered no value because teams never defined what value they expected to see or how to measure it.

The failure mode is not a bad tool. It is a missing question asked too late: what would have to be true, numerically, for this to be worth keeping?

When that question gets asked after the data arrives, founders rationalise. A response time that dropped 8 percent feels like progress if the team is excited about the tool and a disappointment if they are tired of the workarounds. The number means whatever the mood says it means. Pre-committed thresholds exist to prevent exactly that.

What you are actually designing when you design a pilot

A 90-day pilot for lead response is not a technology test. It is a decision machine with one output: go or no-go. Every design choice either sharpens that output or muddies it.

Start with the primary metric. For lead response, the most direct candidate is time-to-first-response, measured from the moment a lead submits a form or sends a message to the moment they receive a substantive reply. Pick one number. Not two, not a composite score. One. The reason is not methodological purity. It is that two metrics let you declare victory on whichever one moved.

Alongside the primary metric, set guardrail metrics. These are not success criteria. They are stop conditions. If AI-handled leads show a 20 percent drop in reply quality scores from customers, that triggers a review regardless of what the speed number says. Guardrail metrics protect against the case where the primary metric improves because the AI is doing something customers find annoying at a rate you have not noticed yet.

Then set the exit threshold before day one. A specific number. Not "meaningful improvement" or "positive trend." Something like: if time-to-first-response does not drop by 15 percent in the AI arm relative to the control arm by day 90, the tool does not scale to the full workflow. Write it down. Date it. Share it with whoever will see the day-90 data.

The control group problem small businesses skip

The hardest part of this design is the control group, and it is the part most founders skip. Without one, you are not running a pilot. You are running a before-and-after, and before-and-after comparisons are contaminated by everything else that changed during the 90 days: seasonality, a new salesperson, a pricing change, a competitor going quiet.

A control group for lead response does not require sophisticated tooling. Route incoming leads by alternating assignment or by channel. Leads from one source go through the AI-assisted workflow. Leads from another go through your current process. Track both. The control group's job is to absorb all the ambient change so the difference you observe is attributable to the AI workflow, not to October being a slower month than July.

The objection worth taking seriously here is volume. A small business processing 30 to 80 inbound leads per month ends up with 45 to 120 leads per arm over 90 days. That is thin. A 15 percent difference on 60 leads sits inside normal variance. The threshold you set will not carry the statistical weight a large technology firm's A/B test would carry.

This is a genuine constraint, and the research acknowledges it directly. Small businesses lack the data volume to run the kind of controlled experiments that produce clean p-values. But the alternative the research documents is not careful iterative evaluation. It is no evaluation at all. MIT found that pilots without defined value targets produced zero measurable P&L impact in 95 percent of cases. The imperfect threshold on thin data still outperforms the anecdotal excitement on no data.

The threshold's function is not statistical significance. It forces you to name what success looks like before you see the results. That one act breaks the rationalisation loop RAND and MIT both document.

What day 90 looks like when the design holds

On day 90, you have a primary metric reading from both arms, guardrail metric readings from both arms, and a pre-committed threshold sitting in a document you wrote before the pilot started. The decision is not a judgment call. It is a comparison.

If the AI arm hit the threshold and the guardrails held, you scale the workflow. If the AI arm missed the threshold, you stop the tool in that workflow and document what the actual number was. That documentation matters. S&P Global's data shows abandonment rates climbing sharply, which suggests organisations are now recognising failure faster. Recognising failure faster is only useful if you record what the failure looked like, so the next pilot starts with a better-calibrated threshold.

The 90-day window will not capture every effect worth knowing. Customer trust effects and relationship quality changes take longer than 90 days to surface. Set those as separate measurements to track after the go decision, not as criteria inside the pilot window. Mixing long-horizon effects into a 90-day kill criterion is how pilots become permanent experiments with no endpoint.

One thing worth naming plainly: I find the "run AI everywhere fast" argument, which the research attributes to advocates of rapid organisation-wide deployment, genuinely counterproductive for small businesses. The average sunk cost of a failed AI pilot runs between $25,000 and $100,000 in staff time, vendor fees, and integration work that never pays back. Moving fast without a control group and a pre-committed threshold is how you accumulate that cost without noticing.

Set the threshold. Write it down before day one. The decision at day 90 will be uncomfortable either way, but at least it will be a decision.