Route Before You Regret It

Air Canada's chatbot told a grieving customer he qualified for a bereavement fare discount. He did not. The airline argued in court that its chatbot was a "separate legal entity" responsible for its own statements. The court rejected that argument, held Air Canada liable, and ordered it to honor the discount. The case is Moffatt v. Air Canada, and it is the clearest recent illustration of what deferred AI oversight costs.

Small businesses read that story and assume it applies to airlines, not to them. That assumption is worth examining.

What confidence scoring actually does

Large language models produce factually wrong or inconsistent output at rates academic researchers describe as non-trivial. The models do not flag these errors the way a spell-checker flags a typo. A confidence score is the model's internal estimate of how certain it is about a given reply. When that score falls below a threshold you set, the reply routes to a human before it reaches the customer. The human reviews it, corrects it if needed, and sends it. The customer gets an answer. You get a record of how often the AI needed intervention.

This is not a complete solution. The research is explicit about the failure mode routing cannot catch: a model scores high confidence on a hallucinated answer. It has no mechanism to distinguish between what it knows and what it has generated plausibly. Routing intercepts uncertainty. It does not intercept confident wrongness.

The error you are not measuring yet

If you have deployed a customer-facing AI without tracking error rates, you do not know your baseline. You cannot set a defensible threshold without one. You cannot tell whether a threshold change improved performance or made it worse. The research frames this precisely: tracking error rates with the same discipline applied to other quality metrics is what separates responsible deployment from speculative deployment.

The measurement approach the research describes is not complicated. Before full rollout, run the AI on a sample of real customer messages and score its outputs for accuracy. During rollout, log every instance where a human reviewer corrects an AI reply. Track the correction rate by message category. Tighten the confidence threshold in categories where the correction rate stays high. Widen it where the AI performs reliably. The threshold is not a setting you pick once — it is a number you move based on evidence.

Where routing still leaves you exposed

A business owner could reasonably argue that routing creates a false sense of control. The errors most likely to produce a complaint or a court filing are exactly the ones confident enough to clear the threshold. The research does not dispute this. It names the problem directly.

The rebuttal is not that routing eliminates this risk. It is that a business running AI with no routing and no error measurement has no interception at any confidence level and no data on either failure mode. Routing plus measurement is a partial control. No controls is not a more honest posture — it is a worse one.

The EU AI Act adds a regulatory dimension the research names explicitly. Compliance expectations for AI systems in customer-facing roles are tightening. Human-in-the-loop design is named in that context not as a quality preference but as an alignment requirement.

What to build before you widen the threshold

The research describes a layered structure: define which issues AI handles autonomously, use confidence scores and escalation signals to determine when a person steps in, and measure hallucinations and other errors before and during rollout. The sequencing matters. You do not widen the AI's autonomous scope until the correction rate in the current scope gives you a reason to.

Framing this as "AI instead of humans" is the setup for the Air Canada problem. The research frames it differently: AI handles predictable work, people retain authority over ambiguous or high-risk interactions, and the error rate data tells you which category a given message type belongs in. That classification is not static. A message type the AI handles well in month one may perform worse as your product changes and the training data ages.

The businesses most exposed right now are not the ones that refused AI. They are the ones that deployed it, assumed it was working because no one complained yet, and skipped the measurement step that would tell them otherwise.