How to Know if AI Actually Saves You Time

The GitHub Copilot trial gave half a group of professional developers access to an AI pair-programming tool and asked both groups to implement an HTTP server in JavaScript. The AI-assisted group finished 55.8% faster. Completion time dropped from roughly 160.9 minutes to 71.2 minutes. That is a real, large, statistically significant result. If you are a founder who codes, that number should matter to you. The problem is what you do with it next.

Where the math breaks down

Most founders who claim twenty-plus hours of monthly AI savings are not lying. They ran the mental calculation: "I spend a lot of time on writing and coding, AI cuts those tasks roughly in half, so I must be saving twenty hours a month." The direction is right. The number is not.

The Noy and Zhang study on ChatGPT-assisted writing found a 40% speed gain on bounded writing tasks under controlled conditions, with independent graders rating output quality 18% higher at the same time. Workers who used ChatGPT in the experiment were twice as likely to report using it in their real jobs two weeks later. That is the mechanism worth paying attention to: after an early win, AI use spreads rapidly across tasks. Your subjective sense of efficiency compounds. The actual logged hours do not.

Self-reported time savings drift from reality when no baseline exists and recall replaces logging. This is documented in IT productivity and measurement error research, not as a theory about founders specifically, but as a general finding about how humans estimate their own time. You remember the wins. You forget the rework, the prompt iteration, the output you checked and discarded.

When the experimental effect sizes make logs feel unnecessary

A reasonable critic looks at the Copilot trial's 55.8% speed reduction and the Noy and Zhang 40% writing gain and says: these effect sizes are large enough that even a rough monthly estimate built from them lands in a defensible range. If you spend half your week on writing and coding, and AI cuts those tasks by 40-55%, the math gets you somewhere near twenty hours. The direction is correct. The order of magnitude is plausible. Demanding a spreadsheet log before accepting this estimate treats a reasonable inference from strong experimental data as if it were speculation.

This objection is not wrong about the direction. It is wrong about the number. The controlled trials produced those effect sizes on single, bounded tasks with explicit performance incentives and random assignment. When AI use spreads across a real month of mixed, uncontrolled work, you are no longer applying a percentage to a stable set of tasks. You are applying it mentally to an expanding, shifting set of tasks, many of which do not resemble the experimental conditions. The "roughly right" estimate requires knowing which hours were AI-assisted, which were not, and what your baseline was before AI. Without a log, you are working from a feeling.

The only number worth citing to an investor

Track one workflow. Pick the one where you believe AI saves the most time. Log how long it takes before AI, for two weeks. Log how long it takes after AI, for two weeks. Use a shared spreadsheet with a date column, a task column, a start time, and an end time. That is the entire system.

The Noy and Zhang findings and the Copilot trial are both built on exactly this structure: a before group, an after group, a single bounded task, logged completion times. You do not need a control group. You need a before. Without a before, you have no claim.

The weekly log matters because it catches the rework the timer misses. A writing task completed in 40% less time and then revised three times is not a 40% gain. A coding task finished faster and then debugged for two hours is not a 55.8% gain. The log catches this. Memory does not.

Two weeks of pre-AI logs and two weeks of post-AI logs on a single workflow will not prove your twenty-hour monthly claim. They will tell you whether that claim is worth making. If the data supports it, you have something an investor or lender can audit. If it does not, you know before they ask.