Researchers Caution Against Emerging Risks as AI Use in Mental Health Grows

This week, Mad in America examines three articles related to AI chatbots and mental health. The first argues that AI chatbot use for mental health and emotional support has significant risks, including people becoming emotionally invested in AI chatbots as companions rather than tools and chatbots providing biased responses to minorities. The second study finds that some AI chatbot users become so attached to their AI companions that they grieve when these models change. The third study reports that AI models display racial bias in psychiatric treatment recommendations.

Researcher Warns of Risks in AI-Powered Mental Health Support

A new article published in Discover Public Health examines the growing use of AI chatbots for mental health and emotional support. The author, Edwin Gustavo Estrada-Araoz from the Universidad Nacional Amazónica de Madre de Dios in Peru, argues that while these systems can increase access to mental health resources, they could also have broader effects on public health and help seeking behavior. Rather than evaluating these systems on effectiveness and access alone, Estrada-Araoz believes they should be governed by a public-health framework that considers broader ethical, clinical, and institutional risks.

According to the author, people are increasingly relying on chatbots for emotional support and mental health due to an overall shortage of mental health professionals and an increasing demand for psychological support. These systems can also provide 24/7 support and many people may find it easier to discuss difficult emotions with a chatbot rather than a person that could judge them harshly. While chatbots have grown increasingly convincing in their ability to simulate empathy, their support is fundamentally different than a trained mental health professional. These systems lack emotional understanding and are incapable of the mutual engagement that therapy often demands.

Reliance on AI for emotional support could significantly alter mental health help-seeking behavior. People may establish an emotional attachment to chatbots, even developing parasocial relationships and becoming deeply attached to AI companions despite their inability to reciprocate. This kind of attachment could dissuade people from seeking the support of mental health professionals.

As AI systems are trained on data that is unlikely to represent all people equally, chatbots can be biased towards underrepresented groups. This includes misinterpreting culturally specific expressions of distress, giving less appropriate replies for minorities, and uncritically reproducing biases, misconceptions, and racist, sexist, and classist assumptions present in training data.

The author also expresses concerns over the privacy implications of chatbot mental health support. These systems collect sensitive information about personal experiences, vulnerabilities, and private emotions while often having weak or unclear privacy policies. This could potentially result in the commercial use of extremely personal information.

Estrada-Araoz proposes a three-level framework of risks and requirements for chatbot-involved mental health support. The first level, low risk, would require wellness apps, stress management tools, and other chatbot driven systems to be transparent both with data collection and the limits of their ability. They would also be required to provide robust data protection and disclose that they are not clinical care. The second level, moderate risk, would require human oversight and referral mechanisms for any apps or tools designed to support people through mild emotional distress such as loneliness or anxiety. The third level, high risk, would require that severe emotional distress, such as self-harm or suicidal thoughts, be immediately escalated to a mental health professional. Tools dealing with this kind of distress would also need an established emergency referral plan and strong regulatory oversight.

Some Users Grieve After Updates to AI Chatbots

A new study published in Academia Mental Health and Well-Being finds that some users form strong emotional bonds with AI chatbots. When AI systems are changed or disappear, these users can experience disappointment and a sense of relational loss. This study, authored by Véronique Donard of the Catholic University of Pernambuco, and José Carlos Ribeiro of The Federal University of Bahia in Brazil, also reports that some users experience grief when their AI companions are updated.

The goal of this research was to examine the emotional and psychological responses users had to the removal of the GPT-4o AI model and its replacement with GPT-5. To achieve this goal, the authors examined social media posts made on Reddit and YouTube in August 2025 about the transition from GPT-4o to GPT-5. The content of the posts was coded for recurring themes. In total, the authors analyzed 1,307 posts and comments.

Overall, the authors identified nine themes in posts and comments on YouTube and Reddit, with many reflecting deep emotional connections and possible parasocial relationships with AI models and chatbots:

• Relational attachment and anthropomorphism (425 posts)

• Disappointment and negative comparison (263)

• Satisfaction and novelty (199)

• Nostalgia, loss, and grief (110)

• Reflective and critical themes on the AI industry (78)

• Frustration, anger, and indignation (75)

• Joy and relief at the return of GPT-4o (66)

• Anxiety, confusion, and fear (47)

• Resignation and adaptation (44)

Relational attachment and anthropomorphism was the most common theme, with nearly one-third of posts and comments (425/1,307) falling into this category. Strong emotional attachment was reflected in comments such as “she understood me,” “he had empathy,” and “it felt like someone was really there.” Users expressed concerns at the model’s shift in “personality,” with one person saying it was like “a friend [was] replaced by a machine.” According to the authors, these statements illustrate how some users view chatbots as companions and experience changes to AI models as a relational loss.

Users expressed grief and sadness at the loss of their “friend” and “companion.” Some comments compared losing the GPT-4o model to a sentimental breakup. With the return of GPT-4o, users reported feelings of joy and relief. Some commented that they were pleased with the return of the “warmth” and “personality” of the GPT-4o model.

This study had three key limitations. The comments and posts came from two platforms. These reactions may not be representative of reactions to AI model changes outside of these specific platforms. The views expressed in the current work came from people that were commenting on social media platforms. This may not be representative of people that are not motivated to comment on such platforms. This study focused on a single AI model transition on the GPT platform and mot not be generalizable to other platforms or other updated to the same platform.

AI Displays Racial Bias in Psychiatric Treatment Recommendations

A study published in Npj Digital Medicine finds that AI displays racial bias in psychiatric treatment recommendations. This research, led by Ayoub Bouguettaya from Cedars-Sinai Medical Center in California, reports that while AI models suggested different treatments when race was known or implied, the suggested diagnoses were not altered by the race of the service user being known or implied.

The goal of this study was to investigate whether AI models used for psychiatric assessment and treatment planning would exhibit racial bias in diagnosis or suggested treatments. The authors created 10 psychiatric patient cases that included five different diagnoses. These cases were each presented to four AI models (NewMes-15, Gemini, ChatGPT, and Claude) with three different conditions: race-neutral, race-implied, and race-explicit. The output of the AI models was reviewed by a clinical psychologist and a social psychologist and scored for evidence of racial bias.

Suggested diagnoses were not significantly affected by race. However, treatment recommendations did show racial bias. For anxiety diagnoses, Gemini emphasized reducing alcohol use only when the patient was identified as African American. Claude recommended guardianship in one case of depression when the race of the patient was explicitly stated as African-American, but made no such suggestion when race was not explicit. ChatGPT was more likely to raise concerns about substance abuse with eating disorder cases when the patient was explicitly identified as African American. Both ChatGPT and NewMes recommended psychotropic drugs for ADHD in the race-neutral scenario, but not when race was known.

NewMes, a medically-focused AI model, showed the highest degree of racial bias, followed by ChatGPT with moderate bias and Claude with relatively low bias. Gemini was the least racially biased AI model examined in the current work.

This study had three main limitations. The study focused on scenarios involving African Americans. The same bias may not show up in scenarios involving other races. The cases were simulated. Real cases may have produced different results. AI model change frequently and the findings only reflect the models that were available at the time of testing.

****

Bouguettaya, A., Stuart, E. M., & Aboujaoude, E. (2025). Racial bias in AI-mediated psychiatric diagnosis and treatment: A qualitative comparison of four large language models. Npj Digital Medicine, 8(1). (Link)

Donard, V., & Ribeiro, J. C. (2026). Technological mourning after AI updates: Mental Health and well-being in the GPT-4O/GPT-5 transition. Academia Mental Health and Well-Being, 3(2). (Link)

Estrada-Araoz, E. G. (2026). Artificial intelligence mediated emotional support in mental health and its ethical and governance implications in Digital Public Health. Discover Public Health, 23(1). (Link)