AI Hallucinations as a Compliance Risk: What SaaS Companies Need to Explain to EU Customers

Key Takeaways

• AI-generated inaccuracies concerning identifiable individuals may engage the GDPR accuracy principle. Compliance depends on the processing purpose, the parties’ GDPR roles and the measures available to prevent and rectify inaccurate personal data.
• Articles 13 to 15 of AI Act primarily concern high-risk AI systems. Where those provisions apply, providers must document relevant performance limitations, expected accuracy and human-oversight measures. Hallucination risks may form part of that assessment, although the AI Act does not use “hallucination” as a standalone legal category.
• Human-review workflows can reduce operational risk and clarify customer responsibilities, but they do not automatically transfer liability from the provider to the customer.
• Accuracy claims should be specific, substantiated and linked to defined use cases and evaluation methods. B2C claims may fall under the Unfair Commercial Practices Directive, while B2B marketing is more directly governed by misleading-advertising and national unfair-competition rules.

Why AI Hallucinations Require Early Risk Awareness

For technical founders, an AI hallucination, the phenomenon where a Large Language Model (LLM) generates factually incorrect but plausible-sounding text, is often viewed as a technical limitation that may be reduced through measures such as RLHF or RAG. However, in the context of the European regulatory landscape, these outputs may create a material AI compliance risk. When SaaS provides an enterprise tool to an EU customer, a hallucination is not merely a 'probabilistic quirk'; it may also create legal exposure, delay procurement and attract regulatory scrutiny.

From Technical Limitation to Business Risk

In a B2B SaaS environment, unreliable outputs can create operational, contractual and financial consequences. If your AI agent provides incorrect tax advice or hallucinates a clause in a legal summary, the enterprise customer may make an erroneous operational decision. This creates a chain of business risk: the customer may terminate the contract for breach of warranty, seek indemnification for third-party damages or fail their own internal risk assessments. In the EU, where 'trustworthy AI' is a policy pillar, failing to address these risks early may result in your product being considered unsuitable for the intended use during technical due diligence.

Potential Harm Caused by Hallucinations

The harms of AI hallucinations are multifaceted. Legally, they can lead to the processing of inaccurate personal data, potentially engaging data-protection requirements. Reputationally, a platform that confidently asserts false information can undermine user trust. From a procurement standpoint, enterprise buyers are increasingly wary of 'black box' risks. Without clear documentation on how you handle AI output reliability, your SaaS may struggle to meet the security and compliance requirements of large European enterprises.

Why Early Risk Assessment Matters

Waiting until the final stages of a sales cycle to address hallucination risks may create avoidable delays. Early risk assessment allows product teams to implement appropriate safeguards by design. By identifying where the model is most likely to fail, you can implement UI-level warnings or human-in-the-loop triggers. Furthermore, an early assessment ensures that your marketing team does not make unsupported accuracy claims that could later be flagged as misleading under applicable consumer-protection, advertising or unfair-competition law.

Legal and Compliance Issues: Accuracy, Transparency, Human Oversight and Liability Allocation

Navigating AI compliance risk requires a shift from technical metrics to legal standards. Technical metrics such as perplexity remain relevant, but they are not a substitute for assessing how the system interacts with applicable law. This section explores the four pillars of European regulation as they apply to AI-generated falsehoods.

GDPR: Accuracy and Data Subject Rights

The Accuracy principle is one of the most directly relevant GDPR issues for AI SaaS providers. Under GDPR Article 5(1)(d), personal data must be accurate and, where necessary, kept up to date. If an LLM generates false biographical details about an identifiable person, the output may constitute inaccurate personal data, depending on the processing context and purpose. The Italian Data Protection Authority (Garante) highlighted this issue in March 2023 when it temporarily restricted the processing of personal data through ChatGPT, citing a lack of accuracy in processing. Furthermore, the EDPB ChatGPT Taskforce report (May 2024) indicated that the 'probabilistic' nature of AI does not exempt providers from the duty to ensure data accuracy. Controllers should provide practical means for users to exercise their right to rectification under GDPR Article 16, even when the data is 'locked' in a model's latent space.

EU AI Act: Risk Management and Human Oversight

The EU AI Act contains separate requirements for high-risk AI systems and general-purpose AI. For high-risk AI systems, Article 15(1) requires that such systems achieve an 'appropriate level of accuracy' and robustness. Transparency is equally critical: Article 13(1) requires providers to supply instructions for use that include 'any known and foreseeable circumstances' which may lead to risks. Depending on the intended purpose, this may include known circumstances in which the system produces unreliable or fabricated outputs. To mitigate these risks, Article 14 requires effective human oversight for high-risk AI systems, requiring that systems be designed so humans can 'disregard, override or reverse the output.' If a SaaS product is classified as high-risk and does not enable human review of high-stakes AI outputs, it may be non-compliant.

DSA and Consumer Protection

While the Digital Services Act (DSA) applies, among other services, to online platforms, Article 25 prohibits 'deceptive design' (dark patterns). Where the service qualifies as an online platform, interface choices that materially obscure the AI-generated nature or known limitations of an output may require assessment under Article 25. Additionally, the Unfair Commercial Practices Directive (2005/29/EC) may apply where the tool is marketed to consumers; B2B marketing is generally governed by Directive 2006/114/EC and national law. If your sales deck claims '99% accuracy' while internal tests show a 10% hallucination rate in complex tasks, the claim may be considered misleading and may lead to enforcement or contractual disputes .

Contractual Liability

Liability allocation is an important part of risk management . Standard SaaS agreements often include 'as-is' clauses, but EU enterprise buyers often reject these for AI. The parties should clearly delineate who is responsible for the 'final' output. Where appropriate, and without treating AI Act Article 14 as a general liability rule, providers may contractually require the customer to perform a 'Human Review' before acting on AI outputs. Such a requirement may be relevant to causation and contractual risk allocation, but it does not automatically transfer the liability for 'hallucinated advice' to the user or relieve the provider of its own obligations.

Typical Questions from EU Customers

When selling into the EU, expect your CISO and legal counterparts to focus on AI output reliability. Your ability to answer these questions can directly affect procurement readiness.

Questions on Output Reliability

Customers will ask: 'How do you measure accuracy in a non-deterministic system?' You should be prepared to provide general benchmarking, where relevant, but more importantly evaluation results against 'representative ground-truth data specific to your application's domain. They may also ask about model settings, source grounding, abstention mechanisms and how you balance creativity against factual reliability.

Questions on Risk Controls

Enterprises want to know what happens when the AI is wrong. 'What technical guardrails (e.g., RAG, prompt engineering, output filtering) are in place to reduce and detect unreliable outputs ?' and 'Is there a mechanism for users to flag and correct inaccurate outputs?' are standard queries. For high-risk AI systems, they may also seek evidence of the Risk-management System required by the EU AI Act. For other systems, comparable controls remain relevant procurement evidence.

Procurement Focus

Legal teams will examine the Data Processing Agreement (DPA) where applicable, as well as the main services agreement, AI-specific terms and service levels . They will ask: 'How does your system comply with the GDPR Article 5(1)(d) accuracy principle if a user identifies a hallucination?' They will also examine liability caps, specifically whether you exclude 'indirect damages' arising from AI errors.

Preparation: Product Controls, Workflow Measures, Technical Evaluation and Documentation

To pass an EU procurement process, you should demonstrate a documented and proportionate approach to managing the AI hallucination risk. This involves addressing not only code, but also process, governance and documentation..

Product-Level Controls

Implement UI elements that remind users of the AI's nature. Depending on the use case and applicable law, this may include AI-generated labels, badges or notices near the output. Provide a 'Verify' button that links to source citations (particularly useful for RAG architectures) and a feedback loop (e.g., Thumbs Up/Down) to support monitoring and correction. Such feedback mechanisms do not, by themselves, satisfy the human-oversight requirements of AI Act Article 14.

Workflow-Level Measures

Design your software so that high-risk actions (e.g., sending an email to a client, approving a transaction) cannot be fully automated. Require a meaningful human-review step before execution. A simple 'Confirm' click is not sufficient where the reviewer lacks the information, competence, time or authority to assess the output. This workflow isn't just a UX choice; it can also support legal risk mitigation and contractual risk allocation.

Technical and Evaluation Measures

Consider maintaining a structured log of material hallucination incidents as part of your internal technical documentation. Use automated 'Eval' frameworks to test your model's factual consistency after material model or system changes. The AI Act does not specifically require a document called a 'Hallucination Log'. For high-risk systems, however, comparable records may support compliance with the risk-management, technical-documentation, logging, accuracy and post-market-monitoring requirements in Articles 9, 11, 12, 15 and 72.

Documentation and Contractual Materials

Update your Terms of Service to include specific 'AI Use' sections. Clearly define the 'Intended Purpose' of the AI to clarify supported uses and the consequences of out-of-scope use. Also consider creating an 'AI Transparency Factsheet' or 'Instructions for Use' that, for high-risk AI systems, meets the requirements of AI Act Article 13. The document should explain that the system is probabilistic and may generate inaccurate information, and should identify the categories of outputs or decisions that require verification by an appropriately qualified person.