Measuring AI ROI Without a Data Team

A Kenyan randomized controlled trial gave 640 small-business entrepreneurs access to a GPT-4 assistant via WhatsApp and measured what happened to their revenues and profits. The average treatment effect was minus 0.09 standard deviations, statistically significant at p = 0.007. AI access, on average, made performance worse. That finding deserves to sit at the front of any conversation about measuring AI ROI for small businesses, because most of those conversations start somewhere else entirely.

The number your investor will ask for next

Before you pick a metric, you need to understand what the Kenyan RCT actually found, because it changes the measurement problem. The negative average effect masked a 0.27 standard deviation split between high and low performers. Entrepreneurs above the median baseline saw roughly a 15 percent performance increase from the AI assistant. Those below the median lost roughly 8 percent relative to the control group. The difference did not come from the quality of the AI advice or the questions entrepreneurs asked. It came from how they selected and applied that advice.

This means a founder who tracks one metric before and after AI implementation will get accurate numbers. Whether those numbers help or hurt the investment case depends on something the measurement itself does not control: implementation quality. The measurement works. The outcome it captures is a function of what you actually did with the tool.

Why one metric is still the right starting point

Italian researchers studying 210 innovative startups found that AI assimilation into internal activities produced measurable improvements in technological performance, operational efficiency, and economic outcomes. The metrics that responded first and most clearly were operational ones: response times, error rates, throughput. These are not sophisticated. They are the numbers small businesses already track informally, and they move quickly when AI changes a workflow.

A Saudi Arabia study using the Technology-Organization-Environment framework documented the same pattern in SMEs: operational performance improved through better forecasting and automation of routine tasks, and the gains showed up in metrics like order processing time and unit throughput before they showed up anywhere else. If you need to demonstrate AI value to an investor who has twelve minutes for your numbers, one clean operational metric with a matched before/after comparison is more persuasive than a five-page report full of partially related data.

The trap is treating that metric as self-explanatory. It is not.

What matched periods and documented confounders actually do

A Lebanese study of 417 SMEs found that AI assimilation into workflows, as opposed to isolated pilots, produced significant positive effects on firm performance across financial and non-financial indicators. The firms that failed to show those effects were running AI tools alongside their existing processes without integrating them. Their metrics did not move because the implementation did not change anything.

This is the confounding problem in its most common form. You add an AI tool to your customer response workflow in March. Response time drops in April. Your investor asks whether the drop came from the AI or from the two support staff you hired in late February. If you did not document the hiring, you cannot answer that question. The metric is accurate. The attribution is not.

Matched measurement periods address a different problem. Your business has seasonal patterns. If your baseline covers October through December and your post-implementation window covers January through March, you are comparing different demand environments. The metric will move for reasons that have nothing to do with AI. Matching the periods, same months year-over-year or same weeks in the same quarter, removes that noise without requiring any statistical sophistication.

The part most founders skip

The Lebanese and Italian studies both identify absorptive capacity and dynamic capabilities as mediators of AI returns. Strip the academic language: firms that already knew how to integrate new tools into their workflows got more from AI than firms that did not. The Saudi Arabia study frames this as compatibility with existing processes, which is an adoption factor founders address during implementation, not a fixed precondition.

What this means for measurement is specific. You need to record what you changed, when you changed it, and how completely the change was adopted by the people using it. Not as a formality. As data. If you implemented an AI response tool and three of your five support staff reverted to manual handling within two weeks, your metric will reflect partial implementation. An investor looking at a flat response time line will conclude the AI did not work. The accurate conclusion is that the AI was not fully used. These are different problems with different fixes, and your documentation is the only thing that tells them apart.

I find the whole category of "AI readiness assessment" tools borderline useless for this purpose. They produce scores, not records. What you need is a log: date of change, scope of change, adoption rate among users, and any other business changes in the same window. A spreadsheet works. A vendor dashboard does not.

When the metric moves the wrong way

The Kenyan RCT's low-performer result is not a verdict. It is a diagnostic. Entrepreneurs who lost 8 percent relative to the control group were not permanently incapable of benefiting from AI. They selected and applied advice in ways that produced negative results in that trial. A founder who documents implementation quality alongside the operational metric has the information needed to identify whether a negative result came from poor capability fit, a mismatched measurement window, or a confounding change that was not recorded.

An investor reviewing that documentation can distinguish between two different conclusions: the AI tool does not work for this business, or the AI tool was implemented at partial depth and here is the evidence of where the adoption broke down. The first conclusion ends the conversation. The second one opens a specific remediation path. The measurement method does not guarantee a positive outcome. It guarantees the outcome is legible, and legibility is what investor-credible evidence requires.

Pick one metric your business already tracks. Record it for eight to twelve weeks before implementation. Document every other operational change in that window. Implement the AI tool. Record the same metric for the same duration. Note adoption rate and any deviations from the planned implementation. Hand that to your investor.

If the number went up and the documentation is clean, you have a case. If the number went down and the documentation is clean, you have a diagnosis. Either way, you are not guessing, and neither is the person reading it.