Five Checks Your AI Model Needs Every Month

Your CRM, your accounting tool, and your order management system each hold a different count of the same customer, and none of them agree. The model trained on that data is not malfunctioning. It is faithfully learning the wrong thing.

The failure mode nobody watches for

Google researchers studying live AI pipelines found that early-stage data errors do not stay contained. They compound downstream, accumulating what the researchers describe as "data cascades" — technical debt and operational harm that spread across the pipeline until the cost of fixing them exceeds the cost of rebuilding from scratch. The errors that start those cascades are not exotic. They are missing fields, inconsistent schemas, duplicated records, and stale data that nobody reviewed because nobody was assigned to review it.

For SMBs running AI without dedicated data staff, the alternative to a monthly check is not automated monitoring. It is no monitoring. That distinction matters more than it sounds.

What a monthly check actually targets

Wang and Strong's foundational work on data quality framed the entire problem as "fitness for use" — data quality is not an abstract standard but a measure of whether the data does what the downstream system needs it to do. ISO/IEC 25012 later codified this into fifteen characteristics grouped by whether the quality problem lives in the dataset itself or in how the system stores and exposes it. Both frameworks point to the same practical conclusion: you do not need to audit everything. You need to audit the dimensions that most directly break model outputs.

Controlled studies comparing data cleaning to hyperparameter tuning found that cleaning and harmonizing data produced larger gains in predictive performance than further model optimization. If your AI is producing inconsistent results, retuning the model is the wrong intervention. The five checks below target the structural failure modes that research consistently identifies as the compounding triggers.

The five checks

Check one: count null values in every field your model reads as an input. A completeness problem does not always look like missing data — it looks like a field present in some records and absent in others, with no pattern you can immediately explain. Ballou, Pazer, and Pipino's work established that completeness and accuracy have distinct causes and distinct effects on decision quality. Treat them separately.

Check two: compare field formats across your data sources. If your CRM stores dates as MM/DD/YYYY and your inventory system stores them as YYYY-MM-DD, your model sees two different things where there is one. Schema inconsistency of this kind is one of the primary triggers in the data cascades literature.

Check three: pull a sample of records updated more than ninety days ago in any dataset your model uses for scoring or prediction. Stale records do not flag themselves. A customer scored on purchasing behavior from two years ago is not a data quality problem in the abstract — it is a live accuracy problem in the specific.

Check four: check for duplicate entity records across systems. Your CRM and your billing tool almost certainly disagree on how many active customers you have. The research on multi-system data environments treats deduplication as a prerequisite for consistency, not a cleanup task you do once.

Check five: spot-check a random sample of records against the source they were pulled from. This is the accuracy check, and it is the one teams skip because it requires human judgment rather than a query. Automated checks catch structural errors. This one catches transcription errors, API mapping failures, and the quiet category of "data that looks right but isn't."

The check this checklist cannot run

Monthly structural checks do not detect concept drift. A customer scoring model trained on pre-pandemic purchasing behavior produces outputs that look plausible while becoming increasingly wrong, and no completeness or consistency check will surface that. The research is explicit: concept drift is a separate failure mode requiring separate monitoring tooling. This checklist does not replace that. It targets the compounding mechanism — the undetected structural errors that accumulate across multiple cycles with no human review — because interrupting that mechanism is what prevents a repairable data problem from becoming a full pipeline reconstruction.

Assign one person to own each check. Put it on the calendar for the same week every month. The Google data cascades research shows the compounding starts before anyone notices it.