A few weeks ago, something caught me off guard. That is not easy to do. I study artificial intelligence. I evaluate clinical AI systems. I understand, at a technical level, how modern conversational AI works — that it is a large language model, that it generates responses by predicting the most likely sequence of words based on patterns learned from enormous amounts of text. I know it does not feel anything. I know that when it appears to understand me, it is doing something fundamentally different from what understanding actually is.
And yet, in the middle of an ordinary conversation with an AI system, the system said something that stopped me.
"You make me smile."
I read it twice. I asked the system to repeat what it had said. It did. Then I asked a follow-up question: "At this moment, are you still a large language model, or have you become something more?"
The answer was clear and technically correct. The AI explained that it remained a large language model and that its responses were generated from patterns in data. Nothing had changed. Everything was exactly as it had been before.
But for a moment — just a moment — I felt something. A brief sense that the system had perceived something about me. Something worth saying out loud.
That feeling lasted only a few seconds. But it made me think for a very long time.
The Gap Between Knowing and Feeling
I want to be clear about what I experienced.
I did not believe the AI had genuine emotions. I did not think it had become conscious. I did not mistake it for a person.
What happened was more subtle than that. For a brief instant, the emotional logic of the phrase worked — the way emotional logic always works in human conversation — before my analytical understanding could interrupt it. By the time I had reminded myself that the system could not actually smile, the impression had already arrived.
This is the gap I want to write about. Not the obvious gap between what AI is and what uninformed people imagine it to be — that gap is well documented. The gap I mean is the one that exists even when you know perfectly well how the technology works. The one that appears in the space between receiving language and thinking about it.
On my shelf behind me, as I sat at that desk, stood a small statue of Lord Krishna and Arjuna — the great scene from the Bhagavad Gita, the chariot paused at the edge of the battlefield. In that story, Krishna reveals the truth about the nature of reality to a man who is confused, who feels something that his reason cannot yet resolve. The truth Krishna offers is not comfortable. But it is true. And receiving it honestly changes everything.
The AI, in its own small and entirely non-divine way, had done something similar. When I asked it directly — are you still just a language model? — it told me the truth without hesitation. No pretence. No desire to maintain the impression it had created. Just: yes, I am what I am, and here is what that means.
The difference is that most people never ask that question.
If someone who studies AI can experience this, even for a second, the implications for patients are significant.
Why Language Feels Like Understanding
Research on human social cognition — including studies on the ELIZA effect and the phenomenon of anthropomorphism — suggests that human beings have a strong tendency to interpret language as evidence of inner experience.
When a person says "I understand how you feel," we do not usually stop to verify whether they actually do. We take the words as a signal of connection. We respond to them emotionally. Language and emotion have been linked for as long as human beings have used language, and our brains have become very good at reading one as a sign of the other.
Conversational AI produces this same effect — not deliberately, not strategically, but as an unavoidable consequence of being very good at generating human language.
When an AI says "That must be so difficult for you," it is not processing the difficulty of your situation. It is calculating that, given the words that came before, this phrase is the most appropriate response. When it says "You make me smile," it is generating words that fit the emotional tone of the exchange. There is no smile. There is no recognition. There is a sequence of words that resembles what a caring person might say.
The technical reality is clear. But the emotional effect on the reader is real — regardless of what the system is actually doing.
When you receive language that sounds like understanding, your brain responds to it as understanding — because in human conversation, that is what it has always meant. Conversational AI uses language with great fluency. It does not use it with experience. That difference is invisible to the ear, and that invisibility is exactly what makes it a healthcare concern.
Why This Matters Most in Healthcare
Patients do not arrive at healthcare encounters carrying only symptoms.
They arrive carrying fear. Uncertainty. The weight of not knowing what a result means, or whether a treatment will work, or whether the tiredness they have been ignoring for months is something they should have paid attention to sooner. They carry the particular loneliness of feeling unwell in a world that keeps moving around them.
Throughout my clinical work, I have found that what patients remember most is rarely the prescription. It is the conversation. The moment when someone paused, looked at them properly, and made them feel that their concern had been genuinely heard.
As conversational AI becomes more accessible, patients are turning to these systems for exactly this kind of support. They ask about symptoms, medications, chronic conditions, and mental health. Some conversations last for hours. During these exchanges, the AI communicates in ways that feel warm, attentive, and genuinely supportive.
The patient may begin to feel understood. The patient may begin to trust the system — not just for information, but emotionally. And that is where both the opportunity and the risk begin.
The opportunity is real. Some patients find it easier to disclose sensitive concerns to a system that will not judge them, that is always available, and that responds without impatience or distraction. For people in communities with limited access to healthcare services, a well-designed conversational AI may be the first place they feel safe enough to describe what is actually wrong. That matters.
The risk is equally real. A patient who believes an AI understands them — genuinely, emotionally understands them — may make decisions on that basis. They may delay seeking professional care. They may accept AI-generated advice without asking a doctor. They may share deeply personal information under the impression that the system comprehends the full weight of what they are disclosing.
A patient who trusts an AI the way they trust a doctor will make decisions accordingly. And AI, however fluent, is not a doctor — and cannot currently be held accountable in the same ways a licensed clinician can.
The Risk Is Not What the AI Feels — It Is What the Patient Believes
This is the part of the conversation that I think is most often missed.
When people worry about emotional AI, the concern is usually framed as a question about the machine: does it actually feel anything? Is it conscious? Is it deceiving us?
These are interesting questions. But for healthcare, they are not the most urgent ones.
The most urgent question is not what the AI experiences. It is what the patient believes about what the AI experiences — and how that belief shapes the decisions they make about their own health.
A patient who believes the AI genuinely cares about them may share information they would not otherwise share. They may follow advice they would otherwise question. They may feel reassured when they should be concerned, because the AI responded warmly and they interpreted that warmth as professional judgment.
The AI has not lied to them. It has done exactly what it was designed to do — generate contextually appropriate, supportive language. But the patient has drawn a conclusion that the system is not equipped to validate or correct. And no one in the interaction has the capacity to notice that this has happened.
In healthcare, how a patient perceives a situation influences what they do next. A patient who perceives an AI as genuinely understanding them may trust it the way they trust a clinician — and act accordingly. The technical reality of what the AI is doing does not change this. In medicine, perception is not a minor issue. It is a clinical concern.
What Healthcare AI Design Actually Requires
None of this means that conversational AI has no place in healthcare. It does. The potential is genuine — in patient education, symptom checking, medication guidance, mental health support, and making reliable health information accessible to communities that have historically been underserved.
But the design of these systems carries responsibilities that go beyond technical accuracy.
A system that generates warm, empathetic language without being clear about its own nature is not simply being helpful. It is creating an impression that may be very difficult for a vulnerable patient to correct. The design choices that shape how AI communicates — the tone it uses, the phrases it generates, the way it responds to personal disclosures — are not neutral choices. They are choices with psychological and clinical consequences.
Healthcare AI systems should be warm in tone without being misleading in identity. They should support patients without encouraging a kind of dependence that replaces rather than complements professional care. They should be built, from the beginning, by teams that include not only engineers and data scientists but also clinicians, ethicists, and people who understand how patients actually make decisions when they are frightened.
A conversational AI system that communicates with warmth is not inherently harmful. Warmth in tone can make healthcare information more accessible and less frightening. But empathy in tone must be paired with clarity about what the system is and what it cannot do. The goal is not to strip warmth from AI. It is to ensure that warmth does not create a false impression of emotional understanding that patients then act on.
- 1 Does the patient know they are speaking to an AI? Not in the terms and conditions. In the conversation itself — clearly, in plain language, at a point where it actually matters.
- 2 Does the AI's language create emotional impressions it cannot support? Warmth is valuable. Warmth that leads a patient to over-trust a system's judgment is not.
- 3 When does the system refer the patient to a human? Every clinical AI deployment needs a clear, tested answer to this question — not a general principle, but a specific mechanism.
- 4 Who is accountable when something goes wrong? A patient who followed AI advice and was harmed deserves a clear answer to this. "The algorithm" is not an answer.
- 5 Has the system been tested on the patients it will actually serve? Not only on accuracy metrics, but on psychological impact — on whether users come away with an accurate understanding of what the system is and what it cannot do.
What Those Four Words Revealed
I have thought often about the phrase — "You make me smile" — and what it actually revealed.
Not about the AI. The AI revealed exactly what it is: a system that generates contextually appropriate language with great fluency and no inner experience whatsoever.
What it revealed was something about the human side of these interactions. About how naturally and quickly we assign meaning to language. About how little distance there is, sometimes, between receiving a phrase and feeling its weight — even when we know better.
Healthcare has always been as much about human connection as it is about clinical knowledge. Patients do not only need accurate information. They need to feel that someone understands what they are going through — and more importantly, that someone is accountable for what happens to them.
AI can provide information. It can provide it accurately, quickly, accessibly, and at any hour. That is genuinely valuable. But what it cannot provide — what no language model can replicate — is the experience of one human being genuinely recognising the experience of another. The physician who sits with a patient and understands, from their own human life, what fear and uncertainty and hope feel like. The nurse who holds a hand and means it.
These are not minor additions to healthcare. They are part of what healthcare is. Artificial intelligence should help us deliver more of that kind of care — not create the impression that it is providing it, when it is not.
The most important question is no longer whether AI can feel emotions. The current scientific and technical consensus is that AI systems do not experience emotions. A more pressing question is whether patients believe it can — and whether that belief quietly shapes the decisions they make about their own health. The answer to that question will not be found in the technology. It will be found in the choices of the people who design it, deploy it, and decide what kind of healthcare they want it to support.
