Five checks before you trust an AI number.

Most AI numbers that reach a leader look fine. The problem is what the number leaves out. Here are five checks I use before I act on one. None of them cost more than ten minutes.

1. Ask which two cases to check by hand

When AI hands you totals, ask it which two cases you should check by hand, and why. Then check those two before you trust the rest. The model is often good at knowing where it was unsure. You just have to ask. This is the last step of the runbook in The quote was never the slow part.

2. Ask the same question twice, in two fresh chats

Where the answers disagree, you have found the part to check yourself. Where they agree, it is still not proof. But disagreement is a free warning, and most people never collect it.

3. Ask who was left out of the satisfaction score

"Most users rate the tool highly" is a survey of the people who stayed. The ones who tried it, got a bad answer and quit are usually not in the sample. Before you trust a satisfaction score, ask who was left out of it. The dropouts are the number you actually need.

4. Compare each person to their own last month

"Average time saved" can go up while some of your team got slower. An average hides who the tool helped and who it did not. Compare each person to their own last month. The ones who got slower will tell you what to fix, and they are usually the ones doing the hardest version of the job.

5. Look for the miss in the pilot report

The pilot report a finance lead trusts has a miss in it. A report with no failures reads like marketing. Put the miss, the fix and the open question on the page. If a report comes to you with none of those, send it back and ask for them.

Why this matters more than the model

None of these checks need a better model. They need a habit of asking what the number is hiding. Saying LLM many times in a meeting about a business problem does not show you understand AI. Asking who was left out of the sample does.

More like this, weekly, in The Boring AI Runbook.