Never let AI produce a number

There’s one thing I say more forcefully than anything else when people ask me about bringing AI into their business.

Don’t let AI produce a number.

Quote totals, invoice sums, stock counts, currency conversions. Never have AI “work out” a figure like that. Drafting prose is fine. This is different.

The reason is simple: the failure behaves differently.

You can spot a wrong sentence. You cannot spot a wrong number.

When AI writes a bad sentence, people notice. The phrasing is off, the logic doesn’t connect, something feels wrong on the surface.

Numbers don’t work that way. A plausible number is indistinguishable from a correct one.

Say a quote reads “Subtotal A$4,280”. Whether that’s right can only be established by going back to the unit prices and quantities and recalculating. If it were out by a factor of ten you might catch it — but if it’s out by 5%, it looks completely normal.

And a wrong number goes straight to money. A typo is embarrassing. A wrong quote is either a loss or a credibility problem.

“Can calculate” and “should calculate” are different questions

Modern models can do arithmetic. That’s exactly why this is easy to get wrong.

The problem isn’t accuracy. It’s that the provenance disappears.

A number the model produced carries no record of which rate card it read, which rule it applied, or how it built the total. When a client asks why the figure is what it is, you can’t answer. Not to them, and not to yourself.

A number pulled from your own rate card and placed into a document, by contrast, is always traceable. The same “A$4,280” from those two routes is not the same thing at all.

So where is the line?

Safe to hand over — assembly and inspection

• Pulling the relevant lines out of a fixed rate card and laying them out
• Formatting the document, writing the surrounding explanation
• Checking the finished thing

Not safe — generating the value itself

• Filling in a rate with roughly what it usually is
• Producing a total as if from mental arithmetic
• Writing an exchange rate or a tax rate from memory

The dividing line is whether the number has a solid external source. If it does, it only needs placing. If it doesn’t, it’s a manufactured number.

Inspection is where it actually pays

This may sound backwards, but around figures the most useful place for AI is not producing them — checking them.

Once the quote exists, put these questions to it:

• Does the arithmetic add up?
• Are both inclusions and exclusions stated?
• Is there any line item still missing a price?
• For this kind of job, is the total outside the usual range?

That last one earns its keep. A quote that quietly lost a transfer, or picked up an off-season rate, looks fine if you only read the total. It doesn’t look fine against “the normal range for jobs like this”.

In practice, the number of times you catch something before it goes out is worth more than the speed at which you produced it.

What the human keeps

The human keeps the “is this OK to send” decision. Where money is involved, add one more.

Rates, margin and the final figure are set by a person.

Don’t hand that over. A system can assemble known values correctly and verify that nothing is wrong. What the price should be is a judgement belonging to the person running the business.

In short

When people bring AI into their operations, they usually start by asking how much can be automated. Fair enough — but the more useful question to settle first is what shouldn’t be.

Numbers are top of that list. Keep the step that generates money in your own hands.

---

Guardrail Ops — Hirotoshi Yamaguchi, Gold Coast QLD. Get in touch