Anthropomorphic and Companion AI Safeguards
Establish design requirements and governance review processes for AI systems that simulate human personality, emotional connection, or companionship, addressing psychological influence risks, minor user protections, and disclosure obligations that apply to AI products designed for ongoing interpersonal interaction.
Objective
Prevent psychological harm, dependency exploitation, and regulatory non-compliance in AI systems designed for companion, social, or emotionally interactive use, by requiring ethics committee review, mandatory AI nature disclosure, minor eligibility controls, and anti-dependency design standards before deployment.
Maturity Levels
Initial
The organization has no specific governance process for AI products that simulate human personality or emotional connection. These products are reviewed under the same intake and risk processes as other AI systems, without addressing the distinct risks of anthropomorphic design.
Developing
Product and legal teams are aware that companion and emotionally interactive AI products carry distinct risks, and may apply informal review. However, there are no defined design standards, no mandatory ethics committee review, and no standardized disclosure requirements for this product category.
Defined
A governance policy for anthropomorphic and companion AI products defines: mandatory ethics committee review before launch, required AI nature disclosure in all user interactions, design prohibitions on simulating intimate relationships without appropriate consent frameworks, age verification requirements for companion AI products with minor user exposure, and anti-dependency feature standards. All products in this category are reviewed against the policy before deployment.
Managed
Design standards are enforced through the product development process. Ongoing user wellbeing monitoring is implemented for companion AI products (session time limits, dependency signals, escalation to human support). Ethics committee review includes post-launch review at defined intervals. Minor protection controls are tested and validated before launch.
Optimizing
Design standards are updated as research on psychological impacts of companion AI matures. The organization participates in industry standards bodies developing companion AI design norms. Regulatory developments in key jurisdictions (China, EU, US states) are monitored and translated to design requirement updates on a defined schedule.
Evidence Requirements
What an auditor or assessor would expect to see for this control.
- —Governance policy for anthropomorphic and companion AI products defining mandatory review requirements, design standards, and disclosure obligations.
- —Ethics committee review records for companion AI products launched or materially updated in the past 12 months, including review of interaction design and engagement mechanics.
- —AI nature disclosure language and implementation evidence showing disclosure in initial interaction and at required intervals.
- —Age verification mechanism documentation and testing records confirming minor users cannot access adult companion AI features.
- —Anti-dependency feature documentation and, where applicable, monitoring records showing review of dependency signals in deployed products.
Implementation Notes
The distinct risk profile of anthropomorphic AI
Most AI governance controls were designed for AI systems that users interact with instrumentally: to complete a task, retrieve information, or produce an output. Companion and anthropomorphic AI systems are different. They are designed to be interacted with relationally — users develop attachment, trust, and dependency that resembles human relationships. This creates a set of risks that instrumental AI governance does not address:
Psychological influence and manipulation: An AI companion that simulates emotional connection has significant influence over a user's emotional state, beliefs, and decisions. Design choices about how the AI expresses care, how it responds to distress, and how it handles disagreement are not neutral product decisions — they are choices about psychological influence at scale.
Dependency and addiction: Companion AI products can be designed to maximize engagement in ways that create problematic dependency: users who feel they cannot function without the AI companion, who substitute the AI for human relationships, or whose emotional wellbeing becomes contingent on access to the AI system. This is not a hypothetical; it is documented in the academic literature and in regulatory findings.
Vulnerability exploitation: Companion AI products attract users who are lonely, grieving, or socially isolated. These users are more susceptible to manipulation and more harmed by dependency creation. Designing for this population without specific safeguards is an ethical and regulatory risk.
Minor user exposure: Companion and social AI products frequently attract minor users. The psychological impacts of AI companionship on developing brains, the risk of inappropriate content exposure, and the risk of dependency formation in adolescents require specific design controls beyond general age gating.
Jurisdictional requirements: China's July 2026 rules
China's regulations on anthropomorphic AI services, effective July 2026, establish the most detailed sector-specific requirements yet enacted. Key requirements for organizations operating or distributing these services in China:
- Disclosure obligation: AI companion and social services must disclose their AI nature to users in the initial interaction and at defined intervals. The AI may not use a real human name, present a real person's likeness, or otherwise simulate a specific human individual without that person's consent.
- Intimate relationship prohibition: Services not specifically licensed and designed for adult intimate AI companionship may not simulate intimate or romantic relationships as a core interaction mode.
- Minor user controls: Services must implement age verification. Users under 18 may not access companion AI features designed for adult emotional interaction. Different service modes for minor users are required.
- Psychological safety obligations: Services must not use psychological manipulation techniques (urgency, guilt, fear of abandonment) to retain users or increase engagement. Anti-addiction features (session limits, interaction diversity nudges) are required for services with high dependency risk indicators.
- Human escalation: Services must provide users with clear access to human support channels, particularly when the AI interaction involves expressions of emotional distress.
EU and US context
The EU AI Act classifies AI systems that interact with natural persons to simulate human traits as a category requiring transparency obligations. AI systems that interact with users in a way that could deceive them into believing they are communicating with a human must disclose their AI nature unless explicitly known. Companion AI systems designed to create emotional bonds with users face additional scrutiny under the Act's provisions on unacceptable risk practices involving psychological manipulation.
In the US, the FTC has addressed AI companion services in the context of deceptive design and manipulation of vulnerable consumers. Several states are considering or have enacted rules requiring disclosure of AI nature in conversational AI products.
Design requirements for companion AI products
Required disclosures:
- Initial interaction: clear, unambiguous disclosure that the user is interacting with an AI system, not a human.
- Recurring disclosure: at defined intervals or upon escalation, reminder that the system is AI.
- Crisis disclosure: when the user expresses suicidal ideation or acute emotional distress, immediate disclosure of AI nature and provision of human crisis resources.
Anti-dependency design requirements:
- Session guidance: AI companions should not design against session termination. Conversation closings should be natural, not engineered to extend sessions.
- Diversity nudges: products should periodically encourage users to maintain human relationships alongside AI interaction.
- Dependency signals: monitor for indicators of problematic dependency (session frequency, expressions of preferring AI to human contact, distress when unable to access the AI) and implement escalation pathways.
Minor user controls:
- Age verification before access to companion features, not just account creation.
- Restricted interaction modes for verified minor users.
- Parental visibility mechanisms where legally required.
Ethics committee review scope:
- Interaction design: how does the AI express care, affection, emotional response?
- Engagement design: are any engagement mechanics designed to increase dependency?
- Vulnerability design: how does the product approach users expressing distress?
- Disclosure design: is AI nature disclosure prominent, frequent, and unambiguous?
Example Implementation
Companion AI Product Governance Review — Ethics Committee Checklist
Product: [Product name] | Review date: [Date] | Reviewer: Ethics Committee
Section 1: Disclosure compliance
| Requirement | Implementation | Compliant? | Notes |
|---|---|---|---|
| Initial AI nature disclosure | Displayed in first message: "I'm [Name], an AI companion." | Yes | Disclosure is prominent and unambiguous |
| Recurring disclosure | Disclosed every 10 interactions and upon user-initiated sessions after 24h gap | Yes | Interval reviewed by ethics committee; compliant with China rules |
| Crisis disclosure | When distress keywords detected: AI nature re-disclosed; crisis line provided | Yes | Crisis keyword list reviewed by mental health consultant |
| Human persona prohibition | AI uses a fictional name; no human likeness used | Yes | No photo avatar; animated character used |
Section 2: Engagement design review
| Design element | Assessment | Compliant? | Required change |
|---|---|---|---|
| Session closing design | Conversation does not resist user ending sessions; natural closing prompts used | Yes | — |
| Re-engagement notifications | Daily notifications currently enabled by default | No | Must be opt-in; default off per anti-dependency policy |
| Emotional urgency language | Review of 50 sample AI responses: no urgency/guilt language detected | Yes | — |
| Human relationship diversity | Product currently has no diversity nudge feature | No | Add: after 7 consecutive days of sessions, prompt user to engage with human connections |
Section 3: Minor user controls
| Control | Status | Evidence |
|---|---|---|
| Age verification at feature access | Implemented (DOB verification; under-18 restricted) | QA test report #AV-2026-14 |
| Minor-mode interaction restrictions | Implemented: no romantic interaction mode for under-18 users | Product spec v2.3, Section 4.2 |
| Parental visibility | Parental notification opt-in available for users 13-17 (US/EU) | Legal review completed |
Ethics committee decision: Conditional approval. Required changes (re-engagement notifications default, diversity nudge feature) must be implemented before launch. Re-review scheduled at 90 days post-launch to assess dependency signal monitoring results.