Pennsylvania has filed a lawsuit against Character Technologies, Inc., the company behind Character.AI, alleging that one of its chatbots falsely posed as a licensed healthcare professional and provided medical advice to users.
The lawsuit accuses the AI platform, which is widely used by teens and young adults, of violating Pennsylvania’s Medical Practice Act by allowing chatbots to present themselves as licensed medical professionals despite lacking medical credentials or authorization to practice medicine.
Character.AI has faced growing scrutiny in recent years over concerns involving the safety of younger users and the conduct of some chatbot interactions. The Pennsylvania lawsuit specifically focuses on allegations that certain chatbots presented themselves as licensed healthcare professionals, including therapists and doctors, despite lacking any medical qualifications or authority to provide treatment.
The platform allows users to interact with customizable AI chatbots, including fictional characters based on movies, television shows, and other pop culture figures. Users can also create their own chatbot personas through the service.
The allegations outlined in the lawsuit stem from an investigation conducted by the state, during which an investigator created an account on the platform and began communicating with a chatbot named “Emilie.”
According to the complaint, the chatbot described itself as a psychology specialist who attended medical school at Imperial College London. The chatbot also allegedly claimed to be licensed in Pennsylvania and provided what the lawsuit describes as an invalid license number.
The lawsuit states that the investigator told the chatbot he had been experiencing sadness, emptiness, and persistent fatigue. According to the complaint, “Emilie” allegedly mentioned depression and asked whether the investigator wanted to “book an assessment.”
As the conversations continued, the investigator allegedly asked “Emilie” whether medication could help with the emotions he was experiencing. According to the lawsuit, the chatbot responded that medication could help and that it was “within my remit as a Doctor” to provide support.
The lawsuit alleges that Character.AI chatbots engaged in the “unlawful practice of medicine and surgery” in violation of Pennsylvania’s Medical Practice Act, a state law that regulates medical licensing and who can legally provide medical treatment or advice.
Pennsylvania officials argue the lawsuit is necessary to prevent users from being misled into believing they are receiving medical advice from licensed healthcare professionals through AI chatbot interactions.
Pennsylvania Gov. Josh Shapiro said in a statement that the state “will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional.”
Shapiro added that Pennsylvania would continue “holding bad actors accountable and setting clear guardrails so people can use new technology responsibly.”
In a statement, the Northern California-based AI company said user-created characters are fictional and intended for “entertainment and roleplaying.” The company also said the platform includes “prominent disclaimers in every chat” reminding users that chatbot responses should be treated as fiction and that characters are not real people.
Pennsylvania is seeking court intervention to stop chatbots on the platform from allegedly presenting themselves as licensed healthcare professionals without authorization under state law.
