AI has a particular flaw that makes it dangerous when it’s handed too much. It doesn’t know what it doesn’t know, and it produces a confident, coherent answer anyway. There’s no moment where it pauses and says you might want to check this. It just delivers.
Which means some decisions should never be fully handed to it. The ones that need context, domain knowledge, and the judgment to ask whether the technically correct answer is actually the right one. The trouble is that most firms haven’t decided which decisions those are, so the line gets drawn by default, usually by whoever’s moving fastest.
The 2020 exam grades
There’s a clean example of this. In 2020, an algorithm was used to standardise A-level grades against historical school performance. It did exactly what it was designed to do. Any human would have thought to ask whether it was fair to judge students against past cohorts during a pandemic that had upended their education. The algorithm had no mechanism to raise that question. Around 40% of students got grades lower than their teachers predicted, and the effect fell hardest on large state schools. The decision was reversed under public pressure.
The point isn’t the algorithm’s accuracy. It was accurate. The point is that accuracy was the wrong test, and there was no human in the loop positioned to notice.
The U-turn isn’t available to you
A government can reverse a policy under pressure. A firm that has already hired, fired, priced, advised, or restructured on the back of a flawed AI output usually can’t. The decision has consequences that don’t un-happen. Having someone in the process who’d have known to ask the question first, or who’s accountable for skipping the check, is what stops those situations arising at all.
Where the weakness shows
The clearest sign a firm hasn’t thought about this is the absence of a problem statement. AI gets adopted as a way of working rather than as a solution to a defined need, which means nobody asked which problems it’s actually solving or whether they needed AI at all. Governance and oversight get treated as a later consideration. The line between what AI should assist and what it should decide never gets drawn, so it gets drawn by default.
The longer-term sign is what leaves when people do. Redundancies made on the assumption that AI can absorb the work take domain knowledge, process understanding, and years of relationships out the door with them. The institutional memory that would have caught a bad output goes with the people who held it. Firms rarely notice the gap until they need to question something the system is telling them and find nobody left who can.
The questions worth answering
Which decisions in your firm genuinely need human judgment, and does everyone involved know which ones those are? Where is AI already influencing decisions in ways that would surprise the people affected? What would have to be true for you to trust that a human reviewing an AI output was actually reviewing it, rather than rubber-stamping something leadership had already celebrated?
For professional services firms this sits closest to the core, because judgment is the product. The whole value of the firm is that a person with expertise made a call. Handing that call to a system that can’t tell when it’s wrong isn’t efficiency. It’s giving away the thing clients are paying for, and not noticing until one of them does.