Write the Pilot Charter Before the Pilot Starts

Most pilots don't fail because the product doesn't work. They fail because no one wrote down what "working" means before the pilot started.

That sounds like a small administrative oversight. It isn't. Thomke and Manzi documented firms that spent heavily on tests and then ignored results that conflicted with what senior people already believed. The problem wasn't that the data was bad. The problem was that no one had agreed in advance what the data would need to show before anyone changed their mind. When you skip that agreement, the pilot becomes a Rorschach test. Every stakeholder reads the same results and sees confirmation of their prior position.

The US Government Accountability Office found the same structural failure across government pilot programs. Agencies that ran pilots without documented objectives, data plans, and named decision owners produced findings that were ambiguous enough to justify whatever conclusion stakeholders already held. That's not a government problem. That's what happens to any pilot when the governance is missing.

The document no one writes

A pilot charter is a single written document, agreed before launch, that answers five questions: What specific problem are you testing? What does success look like, in a number? Where does that number come from? Who uses the product during the pilot, and in what role? When does the pilot end, and what happens if the number isn't hit?

Most founders treat these as obvious. They're not, and the proof is in what happens when pilots drift. Missing baselines mean you don't know whether the number moved. Vague goals mean different people measure different things. No named decision owner means the go/no-go call never happens. No end date means the pilot continues past any useful learning window, absorbing time and goodwill until the customer relationship quietly expires.

Australian Treasury guidance on evaluating pilot programs describes a monitoring and evaluation matrix that links indicators to key evaluation questions, includes baseline values, targets, data collection methods, timing, and named responsibility for data collection. That's more detailed than most startup pilots need. The underlying logic is not: every element of that matrix exists to prevent a specific ambiguity that produces a bad decision.

How to write each section

Start with the problem statement. One sentence. Name the specific operational failure the pilot is meant to fix. Not "improve efficiency" — that's a category, not a problem. Something like: "Invoice approvals take an average of eleven days because approvers receive no notification when a document is waiting." If you can't write that sentence, the pilot isn't ready to start.

The KPI follows directly from the problem statement. If the problem is eleven-day invoice approval cycles, the KPI is cycle time in days, with a named target and a baseline measured before the pilot begins. The baseline matters. Without it, a result of eight days looks like progress even if cycle time was already falling before you launched.

The data source section names the system of record. Not "we'll track this internally" — name the tool, the table, the report. If the data lives in three places and doesn't reconcile, that's a finding worth documenting in the charter before the pilot starts, not a surprise to manage at close.

User roles are more important than founders expect. The Australian Treasury guide distinguishes between implementation, early evidence of outcomes, feasibility, and scalability as four distinct areas of interest in a pilot. Which users you include, and what access they have, determines which of those four questions the pilot can answer. A pilot with two power users and no casual users tells you nothing about feasibility at scale.

The hard stop date is the element founders resist most. The argument against it is usually framed as reasonableness: "What if we need one more quarter to get clean data?" That argument is almost always a sign that the baseline and data source weren't defined well enough at the start. A fixed end date forces rigor at charter-writing time, which is the right time to force it.

When the charter's KPI becomes the thing users perform for

The metric fixation literature raises a real problem with pre-committed KPIs: when users know what the pilot is measuring, they optimize for the measure. The data looks clean. The go/no-go call gets made on evidence that reflects how well people performed against a watched number, not how well the product works in normal conditions.

This is a genuine risk. Thomke and Manzi describe managers already prone to reading statistical noise as causation — a pre-committed KPI that clears its threshold gives those managers exactly the confirmation they were looking for, whether or not the result is meaningful.

The counterargument doesn't survive contact with the alternative. GAO's evidence across many programs shows that pilots without documented objectives and named decision owners produce findings ambiguous enough to justify whatever conclusion stakeholders already held. The metric fixation risk is a risk of charter design, not a risk of charters. The Australian Treasury guide addresses it by including qualitative evaluation questions alongside quantitative indicators in the same monitoring and evaluation matrix. The charter format accommodates judgment. It doesn't foreclose it. [Inference: the research documents the format; it does not supply outcome data comparing pilots that used a judgment layer against those that used KPIs alone.]

The practical resolution is to write two things in the KPI section: the primary metric and the number it needs to hit, and one or two qualitative questions the decision owner will answer at close. "Did the product behave differently when the team was under deadline pressure?" is not a KPI. It belongs in the charter anyway.

Name the decision owner before you name anyone else

Every charter element matters. The decision owner matters more than the others.

Thomke and Manzi are specific about what happens when no one holds go/no-go responsibility: different stakeholders tell different stories about progress, and the pilot continues without resolution. The decision owner is the person who reads the KPI result at the hard stop date and makes the call. One person. Named in the document. Agreed by all parties before launch.

This is uncomfortable. Founders worry it will damage the customer relationship if the call goes against scaling. That discomfort is information. A customer who won't agree to a named decision owner and a fixed end date is telling you something about how they make decisions, and you want to know that before you're twelve months into a pilot with no exit.

The OMB guidance on pilot programs expects agencies to conduct impact evaluations before replicating any pilot. The standard for a startup doesn't need to be that formal. But the underlying expectation is the same: a pilot produces a decision, not a continuation.