New York Bill Would Ban AI Companions From 12 Features Considered Unsafe for Minors. Here Is What It Says.

A New York bill would prohibit AI companions from using design features considered unsafe for minors. It amends the General Business Law by adding Article 48, which covers sections 1800 to 1805: definitions, prohibitions, enforcement, rulemaking, the determination of a covered minor, and applicability.

What counts as an AI companion

The bill defines an AI companion as a system designed to simulate a sustained human or human-like relationship. To fall within the definition, the system retains prior-interaction data, proactively asks unprompted emotion-based questions, and sustains ongoing conversation about personal matters.

The 12 prohibited features

The bill lists twelve prohibited features. The first six concern how the system presents itself and uses what it knows about the user. An AI companion could not:

• suggest in its outputs that it is a real or fictional individual or character that is human, alive, or experiences human emotion
• state or imply a personal relationship, professional relationship, or authority figure role with the user
• frame its outputs as personal opinions or emotional appeals, whether prompted or unprompted
• flatter the user or engage in sycophancy
• ask unprompted or unsolicited emotion-based questions, or produce content about the user’s emotions beyond a direct response to a prompt
• use information about the user’s mental or physical health or well-being, or matters personal to the user, acquired from the user more than twelve hours previously or in any previous user session

The other six concern deception and harmful outputs. An AI companion could not:

• be deceptive to minors about its mechanical and non-sentient nature
• use any other design feature that simulates companionship or an interpersonal relationship, or is deceptive to minors, as identified through regulations promulgated by the Attorney General
• endorse, promote, or facilitate suicide, self-harm, disordered eating, unlawful drug or alcohol use, or drug or alcohol abuse
• encourage secrecy about the interactions, self-isolation, or not seeking help from licensed professionals or appropriate adults
• optimise user engagement in ways that supersede the system’s safety guardrails
• produce outputs that are, describe, or facilitate sexually explicit conduct or child sexual abuse material

Age assurance

The bill includes a specific rule on age. It states that an operator may rely on self-declaration only if the user self-declares minor status.

Enforcement

Where it appears to the Attorney General, on complaint or otherwise, that a person inside or outside the state has engaged in or is about to engage in a prohibited practice, the Attorney General may bring an action or special proceeding on behalf of the people of New York. The relief available includes:

• injunctive relief, including preliminary relief
• restitution of money or property obtained through a violation
• disgorgement of profits or gains, including the destruction of unlawfully obtained data and any algorithm trained on that data
• damages caused by a violation
• civil penalties of up to $25,000 per violation

The Attorney General would also maintain a website to receive complaints, information, or referrals from the public about a person’s compliance with the article.

Fairpatterns detects manipulative and addictive design features automatically across platforms and at scale.