17/06/2026
We had a client's AI tool send a tenant a letter confirming their boiler repair was "scheduled and confirmed for Thursday."
It wasn't scheduled. A contractor had submitted a quote. Nobody had accepted it.
The AI found a gap in the data and filled it with the most plausible-sounding answer. The letter read confidently, professionally, exactly like every other letter that had gone out before it.
Reviewer checked the tone. Sounded fine. Approved. Sent.
Tenant took the day off work. Nobody showed up.
Here's the bit that should worry people more than it does.
A wrong answer and a right answer look identical on the page. Same confidence.
Same formatting. Same tone. The only way to tell the difference is to check the thing the AI is talking about, not the way it's saying it.
By the fortieth letter, nobody was reading closely anymore. That's not a tech failure. That's what trust through repetition does to a review process.
This isn't a reason to avoid AI. It's a reason to build the checking in from day one.
Validate data against the source system before anything goes out. Flag anything referencing a date, status, or commitment for closer review. Review for facts, not just tone.
We build that in from the start. Because finding out you need it after a tenant loses a day off work is the expensive way to learn it.