When leaders worry that people trust AI too much, they often reach for the same fix. Make the AI explain itself. Show the reasons behind each answer. Then people can judge the reasons and catch the mistakes.
It sounds right. It can also make the problem worse.
What the research says
Several research teams have studied what explanations do to how people rely on AI. They include Bansal and colleagues in 2021, Vasconcelos and colleagues in 2023, and Romeo and Conti in 2026.
Their findings point the same way. Explanations can raise reliance without making it more appropriate. People lean on the AI more when it explains itself. They do not get better at telling its good advice from its bad advice.
There are conditions under which explanations help. But the simple version, "add reasons and people will check", does not hold.
Why reasons increase trust
Think about how we treat people. A colleague who gives reasons seems more careful than one who only gives a conclusion. We trust the reasoned answer more. Usually that is fair, because giving good reasons takes effort and knowledge.
AI gives reasons as easily as it gives answers. A weak conclusion comes with reasons that sound just as good as the reasons for a strong one. So the explanation does what it always did with colleagues: it raises our trust. But now the trust has no link to the quality underneath.
There is a second effect. An explanation can make the answer feel checked. The reader sees a chain of logic and feels that the work of checking has been done. In fact, nobody has checked anything. The reasons came from the same source as the answer.
A reason is a claim
The fix is to treat an explanation as more claims, not as proof.
If the AI says "Approve this supplier because their delivery record is strong", there are now two things to check. Is approval the right decision? And is the delivery record really strong? The explanation gives you a place to look. It does not do the looking for you.
Used this way, explanations are useful. They point to the one fact that carries the conclusion. A careful person goes and checks that fact.
What this means for AI rollouts
Many organisations now ask for "explainable" AI, and that is reasonable. But leaders should not expect explanations to solve the reliance problem on their own.
- Do not count explanations as checks. A paper that says "the AI explained its reasoning" has not been checked.
- Ask which reason carries the weight. Then ask whether anyone looked at it.
- Watch for rising trust. If people accept AI answers faster after explanations were added, that is not good news by itself.
- Keep the source in view. The best explanation points to evidence outside the AI. Look for that evidence.
Transparency matters. But it only shows where the evidence is. Someone still has to look.
Sources
- Bansal et al. (2021), CHI. Vasconcelos et al. (2023). Romeo and Conti (2026), AI & Society 41:259–278.
