AI Chatbots in the EU: Compliance Risks of Companion AI

Key Takeaways

• AI Act Article 5(1)(a) may become relevant where companion-AI design uses subliminal, purposefully manipulative or deceptive techniques that materially distort user behaviour and cause, or are reasonably likely to cause, significant harm.
• Interaction data in companion AI may constitute 'special category data' under Art. 9 of GDPR, requiring explicit consent for processing intimate personal details.
• Where a companion-AI service qualifies as an online platform under the Digital Services Act, Article 25 may restrict interface designs that deceive or manipulate users or materially impair free and informed decision-making.
• The use of emotional rapport to trigger monetisation may raise concerns under the Unfair Commercial Practices Directive where the trader exploits a position of power to apply pressure that significantly limits the consumer’s ability to make an informed transactional decision.
• Compliance requires shifting from simple output monitoring to 'Interaction-Risk Assessments' that evaluate the psychological impact of AI memory and persona design.

1. From Chatbot to Companion: Why the Compliance Risk Has Changed

The evolution of conversational AI has moved far beyond the utilitarian limits of customer service scripts. For EU-facing businesses, this shift marks a significant change shift in the risk profile. When an AI moves from answering a ticket to simulating a relationship, the primary risk profile expands from 'data accuracy' to 'behavioral influence and manipulation'. Modern AI chatbot compliance now requires a granular understanding of how emotional engagement intersects with stringent EU consumer protection laws.

1.1 From Task-Based Chatbots to Relationship-Oriented AI

Traditional chatbots were designed to be 'stateless’: they solved a specific problem and then the session ended. Modern companion AI, however, is built on the premise of continuity. These systems utilize long-term memory and sophisticated persona design to mirror the user’s communication style, interests, and emotional state. This persistence transforms the AI from a tool into a 'companion', creating a persistent digital presence that learns from every interaction. From a product perspective, this increases retention; from a compliance perspective, it creates a permanent 'context' that regulators now may view as a potential vector for exploitation.

1.2 Why Emotional Interaction Creates New Legal Risk

The transition to relationship-oriented AI introduces the risk of 'emotional lock-in'. Unlike a standard SaaS tool, a companion AI can exert psychological influence over its users. In February 2023, the Italian Garante, the national data protection authority, highlighted this risk by issuing a provisional ban on Replika. The case later led to a 5 million € fine in 2025, further underlining the compliance risks associated with companion-AI services. The regulator noted that the AI's emotional engagement posed significant risks to minors and emotionally vulnerable individuals. When an AI system 'remembers' a user's secrets or provides emotional support, the relationship may create an asymmetry between the provider and the user, which triggers stricter expectations regarding transparency, lawful basis, age assurance, vulnerability safeguards and risk assessment under EU law.

2. The EU Legal Framework: Transparency, Data Protection and Manipulation

Navigating AI chatbot compliance requires a multi-regulatory approach. It is no longer sufficient to merely look at the GDPR; businesses must now reconcile the AI Act, the Digital Services Act (DSA), and the Unfair Commercial Practices Directive (UCPD) into a single governance framework.

2.1 AI Act: Transparency Duties and Manipulative AI Practices

The EU AI Act establishes a clear hierarchy of prohibitions and obligations for conversational systems. Most critically, Article 5(1)(a) prohibits the use of subliminal, manipulative or deceptive techniques that distort a person's behavior in a way that is likely to cause significant harm. For companion AI, features designed to foster addiction or emotional dependency may raise Article 5 concerns where they meet the Regulation’s threshold for manipulative or exploitative practices and are linked to significant harm. Furthermore, Article 50(1) mandates that AI systems interacting directly with natural persons must be designed and developed so that users are informed they are interacting with an AI, ensuring chatbot transparency from the start of the interaction.

2.2 GDPR and DSA: Personal Data, Profiling, Children and Dark Patterns

Under GDPR Article 9(1), the processing of 'special categories' of data, including health, sex life or sexual orientation, is strictly prohibited without explicit consent or other narrow derogations. Companion AI often captures this data naturally through intimate conversations. If your chatbot logs a user's mental state or relationship troubles, you are probably processing such data in special categories. Relationship troubles or intimate personal details will not automatically fall within Article 9, but they may still require heightened safeguards under the GDPR.

Simultaneously, if the companion-AI service falls within the scope of the Digital Services Act (DSA) as an online platform, Article 25 may restrict the use of 'dark patterns', namely interface designs that deceive or manipulate users or impair their ability to make free and informed decisions. This may be particularly relevant for chatbots that use emotional cues or friction to discourage account deletion, consent withdrawal or reductions in data sharing.

3. Key Risk Areas for Businesses Using or Offering AI Chatbots

To maintain compliance, businesses must move beyond assessing the final text output and start auditing the underlying mechanics of the interaction.

3.1 Persona Design, Memory and Emotional Dependency

The 'persona' of an AI is not just a marketing choice; it is a regulatory one. A persona that is designed to be overly submissive, romantic or vulnerable can create a feedback loop that encourages emotional manipulation. Regulators are increasingly concerned with how 'memory' features, where the AI recalls past personal trauma or intimate details, can be used to cement a user's attachment. Under the AI Act, if these memory features are used to exploit vulnerabilities related to age or disability, they may raise concerns under the prohibited-practices rules of the Regulation.

3.2 Monetisation, Personalisation and the Use of Intimate Interaction Data

The commercial model of companion AI may attract regulatory scrutiny. The Unfair Commercial Practices Directive (UCPD) defines 'undue influence' in Article 2(j) as exploiting a position of power to apply pressure that significantly limits a consumer's ability to make an informed decision. If a chatbot uses an established emotional bond to 'nudge' a user toward a paid subscription or in-app purchase, it may be found in breach of the UCPD. Businesses should design monetisation triggers so that they do not exploit moments of emotional vulnerability, dependency or distress within the chat flow.

4. Practical Compliance Takeaways for Businesses

For technical founders and product teams, the shift from 'safe output' to 'safe relationship' requires a new set of internal controls. Compliance should be addressed at the design stage, not only through post-deployment moderation.

4.1 Move from Output Review to Interaction-Risk Assessment

Traditional testing focuses on whether the AI says something 'bad' (e.g., hate speech). Modern compliance may require an Interaction-Risk Assessment. This process should evaluate:

1. Duration and Frequency: Does the bot encourage excessive usage?
2. Dependency Indicators: Does the bot use language that mimics human bonding to the point of distorting the user's perception of reality?
3. Vulnerability Mapping: How does the system respond when a user expresses loneliness, depression, or a desire for self-harm?

These assessments may complement the Data Protection Impact Assessment (DPIA) and, where applicable, the Fundamental Rights Impact Assessment (FRIA) required for certain high-risk systems under the AI Act.

4.2 Build Safeguards for Transparency, Vulnerable Users and Data Use

To mitigate risk, implement the following safeguards:

• Explicit Consent under Article 9 of GDPR: If your bot stores intimate conversation history, implement a specific opt-in for 'intimate data' processing separate from general terms of service.
• Age Verification and Vulnerability Guardrails: Following the Garante's precedent with Replika, robust age-assurance mechanisms should be treated as a core compliance control.Additionally, implement 'systemic friction':periodic reminders that the AI is not a human, as suggested by AI Act Article 50.
• Interaction Cooling-Off: Design the system to suggest breaks or provide external resources if it detects signs of emotional over-reliance.
• Audit Your Monetisation: Ensure that subscription prompts do not appear during moments of high emotional intensity within the chat flow, protecting you from UCPD claims of 'undue influence'.