Measuring AI ROI Without a Data Team

GitHub told Accenture that developers using Copilot coded up to 55% faster. Noy and Zhang ran a randomized experiment at MIT and found ChatGPT cut professional writing time by 40% while raising evaluator-rated output quality by 18%. These are real numbers from real studies. They are also completely useless for your next board meeting, because they describe someone else's team, not yours.

The benchmark trap

Vendor benchmarks and consulting projections serve a communication function. They give founders language that matches how investors already think about technology returns. McKinsey estimates generative AI adds between $2.6 trillion and $4.4 trillion annually across 63 use cases. PwC projected in 2017 that AI could raise global GDP by 14% by 2030. These figures land in pitch decks and board updates because they sound authoritative. The problem is they carry no information about your firm specifically.

The OECD's 2024 cross-country firm analysis found that AI adopters show higher labor productivity than non-adopters within the same size and sector categories. Then the same report found that advantage weakens substantially once researchers control for digital maturity, ICT skills, and existing infrastructure. Read that carefully: a meaningful share of what looks like AI's productivity contribution is actually the firm's pre-existing capability. A founder citing the McKinsey figure has no way to know which side of that distribution they occupy.

What role-level tracking actually tells you

Noy and Zhang's experiment produced one finding that rarely appears in the summary statistics: the productivity distribution compressed. Low-ability workers gained more from ChatGPT access than high-ability workers. The aggregate 40% time reduction masks a distributional story. If your team skews toward strong performers, your actual time savings from AI writing tools will sit below that benchmark. If it skews toward newer hires handling first drafts, it will sit above. You cannot know without tracking at the role level.

The GitHub/Accenture study worked because it collected DevOps telemetry from that firm's own developers on that firm's own projects. Coding speed, pull request throughput, and AI suggestion acceptance rates were all firm-specific. The 55% faster coding figure is credible not because GitHub published it, but because Accenture measured it internally. That distinction is the entire argument.

The steelman case for doing nothing

The OECD 2025 SME paper documents persistent gaps between smaller and larger firms in the enablers that make AI gains materialize: connectivity, skills, finance, and AI-enabling infrastructure. For a resource-constrained founder, building and maintaining a role-level tracking system is not a trivial cost. If the underlying complementary assets are missing, the dashboard shows flat or negative numbers, and the founder now has a worse problem — a visible record of non-performance with no way to separate tool failure from measurement failure from capability gap.

This objection is worth sitting with. It does not resolve cleanly.

What it does not do is make aggregate benchmarks more useful. CFO survey data cited in the research shows that AI budgets in finance functions rose faster than observed outcomes. Those CFOs were tracking at the budget level, not the task level, and the result was exactly the mismatch between spend and flat results the OECD paper describes. Aggregate measurement did not protect them from false comfort. It produced it.

The Google Sheets baseline

The mechanics are straightforward. Before AI adoption, log time per role per task category for two weeks. After adoption, log the same. McKinsey's function-level use-case data points to where the signal is strongest: customer operations, marketing and sales, software engineering, and research and development. Start tracking in one of those four areas, not across all operations simultaneously.

Map the time savings to revenue-generating tasks specifically. An hour saved on internal reporting does not carry the same weight as an hour saved on a sales proposal or a product build. The correlation between time freed in revenue-adjacent tasks and pipeline changes is not causal evidence, but it is observable and specific to your firm, which puts it ahead of any benchmark in a stakeholder conversation.

The OECD 2025 SME paper recommends better evidence gathering as a policy priority, framing the measurement gap as structural rather than incidental. A two-tab Google Sheet with a pre-AI baseline and a weekly log is not a research instrument. It is the minimum required to say something true about your own firm rather than borrowing someone else's number and hoping it applies.

BCG's GPT-4 experiment found that roughly 90% of participants improved on creative tasks with AI assistance, but also identified value destruction when users relied uncritically on suggestions. If your tracking shows time savings alongside rising error rates or revision cycles, that is information a vendor benchmark will never surface.