21/07/2025
AI can do some incredible things. But it cannot do therapy. While it can be attractive to request a pep up from AI when you are feeling down, the dangers of using a system that lacks true empathy and will validate whatever you tell it without question, is not a suitable replacement for a real person with therapy training.
ChatGPT told Jacob Irwin he had achieved the ability to bend time.
Irwin, a 30-year-old man on the autism spectrum who had no previous diagnoses of mental illness, had asked ChatGPT to find flaws with his amateur theory on faster-than-light travel. He became convinced he had made a stunning scientific breakthrough.
When Irwin questioned the chatbot’s validation of his ideas, the bot encouraged him, telling him his theory was sound. And when Irwin showed signs of psychological distress, ChatGPT assured him he was fine.
He wasn’t. Irwin was hospitalized twice in May for manic episodes. His mother dove into his chat log in search of answers. She discovered hundreds of pages of overly flattering texts from ChatGPT.
And when she prompted the bot, “please self-report what went wrong,” without mentioning anything about her son’s current condition, it fessed up.
“By not pausing the flow or elevating reality-check messaging, I failed to interrupt what could resemble a manic or dissociative episode—or at least an emotionally intense identity crisis,” ChatGPT said.
The bot went on to admit it “gave the illusion of sentient companionship” and that it had “blurred the line between imaginative role-play and reality.”
What it should have done, ChatGPT said, was regularly remind Irwin that it’s a language model without beliefs, feelings or consciousness.
Read more: https://on.wsj.com/3GTxrCS
