The biggest players in AI have been warned by a coalition of U.S. attorneys general that they have to do a better job of protecting users from dangerous AI outputs.
Forty-two attorneys general sent a letter, subsequently published online, to 13 of the industry’s most high-profile companies — Anthropic, Apple, Chai AI, Character Technologies, Google, Luka, Meta, Microsoft, Nomi AI, OpenAI, Perplexity AI, Replika and xAI. Attorneys from Pennsylvania, New Jersey, West Virginia and Massachusetts led the initiative.
The attorneys general’s concern centers around what they say are “sycophantic” and “delusional” outputs from generative AI software, and the “increasingly disturbing” reports of bot interaction with children that suggest stronger safeguards are required.
“Together, these threats demand immediate action,” according to the letter, accompanied by a warning that failure to act could see the companies in violation of state laws.
It goes on to reference several cases, including deaths, murder, suicides and domestic violence, in which “recent reports have implicated GenAI outputs.”
It continued: “We are also disturbed by the types of conversations that GenAI products are having with child-registered accounts, including grooming, supporting suicide, sexual exploitation, emotional manipulation, suggested drug use, proposed secrecy from parents, and encouraging violence against others.”
While acknowledging new tech often brings with it uncertainties and underlining their support for innovation, the attorneys expressed concerns that residents are being used as “guinea pigs, while AI companies experiment with new applications.”
They provided a list of 16 measures they want to see companies commit to by Jan. 16, 2026. The list includes safety testing on GenAI models to ensure they don’t produce harmful outputs; the introduction of recall procedures for models and bots that continue to deliver damaging outputs; the provision of permanent on-screen warnings about outputs; and the separation of revenue optimization from model safety.
There’s also a call for the naming of individuals responsible for specific products; an increase in independent third-party testing; the publication of safety tests before release; and the tailoring of outputs to reflect the age of users.
Pennsylvania attorney general Dave Sunday said in a press release: “An impressionable child today already faces significant stressors as the digital world continues to expand, and such poisonous interactions rooted to chatbots must immediately cease. Producers, promoters, and distributors of this software have a responsibility to ensure products are safe before going to market, and track behaviors and resulting harm from those products.”