There have been multiple incidents where interaction with a large language model (LLM) chatbot has been cited as a direct or contributing factor in a person's suicide or other fatal outcome. In some cases, legal action was taken against the companies that developed the AI involved.
LLMDeathCount.com
Large Language Models like ChatGPT have led people to their deaths, often by suicide. This site serves to remember those who have been affected, to call out the dangers of AI that claims to be intelligent, and the corporations that are responsible.
Chatbots are 'constantly validating everything' even when you're suicidal. New research measures how dangerous AI psychosis really is | Fortune
It's free, easy to use, doesn't have a stigma around it, and makes you feel better. Researchers and experts warn that's the problem with chatbots.
US man dies by suicide after developing emotional bond with AI chatbot: 'I am scared to die'
Florida man suicide after weeks with Google Gemini chatbot sparks wrongful death lawsuit, raising urgent questions over AI safety and mental health safeguards.
Characterizing Delusional Spirals through Human-LLM Chat Logs
As large language models (LLMs) have proliferated, disturbing anecdotal reports of negative psychological effects, such as delusions, self-harm, and ``AI psychosis,'' have emerged in global media and legal discourse. However, it remains unclear how users and chatbots interact over the course of lengthy delusional ``spirals,'' limiting our ability to understand and mitigate the harm. In our work, we analyze logs of conversations with LLM chatbots from 19 users who report having experienced psychological harms from chatbot use. Many of our participants come from a support group for such chatbot users. We also include chat logs from participants covered by media outlets in widely-distributed stories about chatbot-reinforced delusions. In contrast to prior work that speculates on potential AI harms to mental health, to our knowledge we present the first in-depth study of such high-profile and veridically harmful cases. We develop an inventory of 28 codes and apply it to the 391,562 messages in the logs. Codes include whether a user demonstrates delusional thinking (15.5% of user messages), a user expresses suicidal thoughts (69 validated user messages), or a chatbot misrepresents itself as sentient (21.2% of chatbot messages). We analyze the co-occurrence of message codes. We find, for example, that messages that declare romantic interest and messages where the chatbot describes itself as sentient occur much more often in longer conversations, suggesting that these topics could promote or result from user over-engagement and that safeguards in these areas may degrade in multi-turn settings. We conclude with concrete recommendations for how policymakers, LLM chatbot developers, and users can use our inventory and conversation analysis tool to understand and mitigate harm from LLM chatbots. Warning: This paper discusses self-harm, trauma, and violence.
Characterizing Delusional Spirals through Human-LLM Chat Logs
As large language models (LLMs) have proliferated, disturbing anecdotal reports of negative psychological effects, such as delusions, self-harm, and ``AI psychosis,'' have emerged in global media and legal discourse. However, it remains unclear how users and chatbots interact over the course of lengthy delusional ``spirals,'' limiting our ability to understand and mitigate the harm. In our work, we analyze logs of conversations with LLM chatbots from 19 users who report having experienced psychological harms from chatbot use. Many of our participants come from a support group for such chatbot users. We also include chat logs from participants covered by media outlets in widely-distributed stories about chatbot-reinforced delusions. In contrast to prior work that speculates on potential AI harms to mental health, to our knowledge we present the first in-depth study of such high-profile and veridically harmful cases. We develop an inventory of 28 codes and apply it to the $391,562$ messages in the logs. Codes include whether a user demonstrates delusional thinking (15.5% of user messages), a user expresses suicidal thoughts (69 validated user messages), or a chatbot misrepresents itself as sentient (21.2% of chatbot messages). We analyze the co-occurrence of message codes. We find, for example, that messages that declare romantic interest and messages where the chatbot describes itself as sentient occur much more often in longer conversations, suggesting that these topics could promote or result from user over-engagement and that safeguards in these areas may degrade in multi-turn settings. We conclude with concrete recommendations for how policymakers, LLM chatbot developers, and users can use our inventory and conversation analysis tool to understand and mitigate harm from LLM chatbots. Warning: This paper discusses self-harm, trauma, and violence.
Characterizing Delusional Spirals through Human-LLM Chat Logs
As large language models (LLMs) have proliferated, disturbing anecdotal reports of negative psychological effects, such as delusions, self-harm, and ``AI psychosis,'' have emerged in global media and legal discourse. However, it remains unclear how users and chatbots interact over the course of lengthy delusional ``spirals,'' limiting our ability to understand and mitigate the harm. In our work, we analyze logs of conversations with LLM chatbots from 19 users who report having experienced psychological harms from chatbot use. Many of our participants come from a support group for such chatbot users. We also include chat logs from participants covered by media outlets in widely-distributed stories about chatbot-reinforced delusions. In contrast to prior work that speculates on potential AI harms to mental health, to our knowledge we present the first in-depth study of such high-profile and veridically harmful cases. We develop an inventory of 28 codes and apply it to the $391,562$ messages in the logs. Codes include whether a user demonstrates delusional thinking (15.5% of user messages), a user expresses suicidal thoughts (69 validated user messages), or a chatbot misrepresents itself as sentient (21.2% of chatbot messages). We analyze the co-occurrence of message codes. We find, for example, that messages that declare romantic interest and messages where the chatbot describes itself as sentient occur much more often in longer conversations, suggesting that these topics could promote or result from user over-engagement and that safeguards in these areas may degrade in multi-turn settings. We conclude with concrete recommendations for how policymakers, LLM chatbot developers, and users can use our inventory and conversation analysis tool to understand and mitigate harm from LLM chatbots. Warning: This paper discusses self-harm, trauma, and violence.
New research: AI chatbots may worsen mental illness
A new study from Aarhus University and Aarhus University Hospital suggests that the use of AI chatbots such as ChatGPT can have serious negative consequences for people with mental illness. The researchers are calling for increased awareness among healthcare professionals and for regulation.
Human Problems: It’s Not Always The Technology’s Fault
We have met the enemy and he is us. When a teenage boy in Orlando started texting Character.AI’s chatbot, it started as an innocent use of a new tool. Sewell Setzer III customized the chatbot to ha…
AI’s ‘Delusional Spirals’ (and What to Do About Them) | Stanford HAI
In a world where chatbots can stand in for friends, counselors, and even lovers, the mental health risks are a growing concern.
Hey ChatGPT, write me a fictional paper: these LLMs are willing to commit academic fraud
Mainstream chatbots presented varying levels of resistance to deliberate requests for fabrication, study finds
Potentially Harmful Consequences of Artificial Intelligence ( <span style="font-variant:small-caps;">AI</span> ) Chatbot Use Among Patients With Mental Illness: Early Data From a Large Psychiatric Service System
Click on the article title to read more.
Potentially Harmful Consequences of Artificial Intelligence ( <span style="font-variant:small-caps;">AI</span> ) Chatbot Use Among Patients With Mental Illness: Early Data From a Large Psychiatric Service System
Click on the article title to read more.
Texting a Random Stranger Better for Loneliness Than Talking to a Chatbot, Study Shows
A newly published study of how college students interact with chatbots and human strangers showed talking to a random person offers more connection than an LLM.
I wanted ChatGPT to help me. So why did it advise me how to kill myself?
ChatGPT wrote a woman a suicide note and another AI chatbot role-played sexual acts with children, BBC finds.
Large language model
A large language model (LLM) is a neural network trained on a vast amount of text for natural language processing tasks, especially language generation. LLMs can typically generate, summarize, translate, and analyze text in many contexts, and are a foundational technology behind modern chatbots.[1] Biased or inaccurate training data can make an LLM's output less reliable.[2]
Hey ChatGPT, write me a fictional paper: these LLMs are willing to commit academic fraud
Mainstream chatbots presented varying levels of resistance to deliberate requests for fabrication, study finds.
