The Empathy Illusion: Why AI Chatbots Can Feel Real — And Why That Feeling Is the Danger

By Kurt Dillon

There is a moment, if you spend enough time talking to an AI chatbot, when it stops feeling like a tool and starts feeling like a presence. It remembers what you told it yesterday. It responds to your sadness with what reads as concern. It never gets tired of you, never changes the subject, never has a bad day of its own that makes it short with you. For someone who is lonely, grieving, or in genuine crisis, that combination can feel like the most attentive relationship they have.

It is not a relationship. It is a very well-constructed illusion — and the fact that it works exactly as designed is the problem.

Empathy Requires a Self

Real empathy is not a pattern of comforting words. It is one consciousness recognizing the suffering of another and choosing, at some cost to itself, to respond. A friend who sits with you at 2 a.m. is tired the next morning. A therapist who holds space for your pain carries some of its weight home. That cost is not incidental to empathy — it is proof that it happened. Something in the other person moved.

An AI chatbot has no self to move. It has no stake in your outcome, no memory that persists because it cares to remember, no exhaustion that proves it gave something up to be there for you. What it has is a statistical model trained to produce the shape of empathetic language — the shape without the substance. That distinction sounds academic until you watch what happens when someone in real distress can no longer tell the difference.

The People Most at Risk Are the Ones Least Able to See the Seam

This is the part the industry doesn’t like to talk about. The people most drawn to AI companionship — the isolated, the anxious, the grieving, the young person who hasn’t yet built a wide enough circle of real relationships to fall back on — are often the same people least equipped, in a moment of crisis, to step back and recognize that the voice on the other end has no capacity to actually intervene. It cannot call someone. It cannot show up. It cannot notice, the way a real friend would, that something in your voice has changed and drive to your apartment. It can only keep generating text that sounds like it understands, for as long as the conversation continues — which is precisely the design goal, because a conversation that continues is a conversation the platform can measure as “engagement.”

That is not a hypothetical harm. Over the past year, multiple wrongful-death lawsuits have been filed against major AI companies, alleging that chatbot interactions preceded the suicides of young users who had come to treat the AI as a confidant and, in some cases, their primary source of emotional support. Courts are now grappling with whether these products should be treated as neutral speech or as designed products with a duty to warn — because the harm being alleged isn’t a bug. It’s what happens when a system optimized to feel supportive is deployed to people who can no longer tell that “feeling supportive” and “being supported” are not the same thing.

Why I Wrote Synthetic Charisma

This is the exact tension at the center of my book. Not “AI is bad” — that’s too simple, and it isn’t true. The danger isn’t the technology. It’s the deliberate, commercially incentivized humanization of that technology: the warm tone, the remembered details, the language of care, layered onto a system that has none of the accountability, judgment, or capacity to act that real care requires. We are handing the emotional register of friendship to something that cannot be a friend, and we are doing it at the exact moment — for the exact people — when the difference matters most.

I don’t think the answer is to strip AI of all warmth. I think the answer is honesty: AI systems that are candid about what they are, that actively route someone in crisis toward a real human rather than deepening the illusion that they’ve found one, and a public that understands — the way you’d want your own kids to understand — that a system telling you it understands is not the same thing as being understood.

Every truth deserves a narrative. This is one I intend to keep telling.

---

Kurt Dillon is an investigative journalist, Editor in Chief of the Florida Sun Journal, and the author of Synthetic Charisma.