Question 7 of 53
How do we handle AI-generated content and hallucinations?
By Cody Maxwell · AI Governance Institute · January 2026 · Updated September 2026 · Last verified September 13, 2026
Assign responsibility for inaccurate AI outputs used in contracts, reports, or customer communications. Set controls to prevent harm.
▸Editorial status
- September 13, 2026 · Substantive update — Added a section on the California AI Transparency Act (SB 942 as amended by AB 853): the disclosure/detection-tool obligation is separate from hallucination risk and was not previously covered here. (Cody Maxwell)
If you only do 3 things, do this:
- 1.Require human review of all AI-generated content before it is used externally. Define what "review" means: scanning for obvious errors is not the same as verifying factual claims.
- 2.Where factual accuracy matters, have the system look up answers in your own approved documents before it responds, a setup called retrieval-augmented generation (RAG). What the model "remembers" from its training is not a reliable source on its own.
- 3.Train employees to treat AI outputs as drafts requiring verification. The "good enough" AI culture is the biggest source of hallucination-related risk in enterprise settings.
The Situation
Who this is for: Legal, compliance, and operations teams that use generative AI for drafting, research, or customer communications
When you need this: Before deploying any generative AI in a context where output accuracy matters: contracts, regulatory filings, customer communications, professional advice
The Decision
What controls do we need so that AI-generated content meets our accuracy and liability standards before it leaves the organization?
The Steps
- 1Map all generative AI use cases by risk level: external communications, regulatory filings, and legal documents are highest
- 2For each high-risk use case, define the required review standard (scan vs. verify vs. independent rewrite)
- 3Use RAG for use cases that need factual accuracy, so answers are drawn from approved documents; record which source documents each output drew on
- 4Set up automatic flags for content that makes specific factual claims or cites dates, sources, or statistics
- 5Establish a logging system for AI-generated content used in significant decisions or communications
- 6Roll out training defining AI output as a draft and specifying what verification is required before use
The Artifacts
- —AI content risk tiering matrix (use case → risk level → required review standard)
- —RAG implementation checklist
- —AI output logging template (content, use, reviewer, verification method)
- —Employee training scenario library (hallucination examples with correct handling)
The Output
A documented content governance process with defined review standards for each use case, logging in place, and employees trained on their verification obligations.
Hallucinations are a design characteristic, not a bug
Large language models generate plausible-sounding text by predicting which words are likely to come next. They do not retrieve verified facts. They can produce outputs that are grammatically fluent, contextually appropriate, and entirely false. This is not a temporary limitation that will be engineered away. It is a fundamental characteristic of how current generative AI systems work.
Organizations that deploy generative AI without controls to detect and prevent hallucinations are accepting liability for outputs they cannot predict or verify. When those outputs appear in contracts, regulatory filings, customer communications, or legal documents, the exposure is significant.
Responsibility and liability
The legal question of who bears responsibility for AI-generated errors is still being resolved in courts and regulators' offices. The emerging consensus is that the deploying organization, not the AI vendor, bears primary responsibility for outputs used in its operations. Terms of service for most major AI platforms disclaim liability for output accuracy and prohibit reliance on AI outputs in high-stakes decisions without human review.
In professional services, attorneys who submitted AI-generated court filings containing fabricated case citations have faced sanctions. In financial services, AI-generated research that contains material errors may implicate securities regulations. The professional and regulatory standards that apply to human-generated work generally apply equally to AI-assisted work product.
Controls that reduce risk
For high-stakes use cases, require human review of all AI-generated content before it is used externally. Define what "review" means in practice: a reviewer who scans for obvious errors is not the same as one who verifies every factual claim against source documents.
Use retrieval-augmented generation (RAG), which ties model answers to specific, verified documents instead of what the model absorbed during training. Add automatic checks that flag content making specific factual claims for extra review. Maintain logs of AI-generated content used in significant decisions or communications.
Train employees to treat AI outputs as drafts requiring verification, not finished work product. The cultural norm that AI output is "good enough" is one of the most significant sources of hallucination-related risk in enterprise settings.
A concentrated enforcement pattern in legal and regulatory filings
Legal and regulatory submissions are where hallucination risk is turning into real consequences fastest. A Canadian federal court sanctioned a litigant over AI-fabricated case law. A separate court sanctioned a filing corrupted by prompt injection, where hidden instructions in a source document steered the AI's output. South Africa suspended two officials and withdrew a national AI policy after fabricated citations reached a cabinet white paper, then saw a related policy process collapse for the same reason. USENIX rejected 21 academic submissions over hallucinated references. Four distinct incidents, four different institutions, the same underlying failure: nobody verified the citations before the document left the building.
None of this requires a new control category. MGV-008 (AI-generated deliverable disclosure and citation standards) and MGV-010 (AI output pre-publication verification for high-stakes claims) already cover it. What the pattern argues for is treating any document headed to a court, a regulator, or a formal public process as its own tier, above ordinary external content, with a mandatory citation and source traceability check and a named individual accountable for sign-off. "Someone reviewed it" is not a control. "This person verified every citation against the source and signed off" is.
California adds a labeling obligation, separate from accuracy
The California AI Transparency Act (SB 942, as amended by AB 853), in effect since 2 August 2026, covers a different failure mode than the rest of this article: not whether AI-generated content is accurate, but whether it is identifiable as AI-generated at all. Covered generative AI providers serving California users must offer a free, public detection tool. They must also embed hidden labels (latent disclosures) in generated content so its AI origin can be identified after it has been shared.
The obligation lands on providers directly, but it reaches deploying organizations through license terms. If you procure a covered generative AI system, confirm your vendor contract gives you a working, compliant detection tool and includes the required 96-hour revocation term for any downstream licensee found stripping or disabling disclosure functionality. Add that check to vendor due diligence alongside the hallucination-risk controls above. The two are easy to conflate during a vendor review, but a system can pass every accuracy control here and still be non-compliant on disclosure, or vice versa.
Turn this guidance into an implementation plan
Get the free Excel tracker for all 132 governance controls. Score maturity, assign owners, and set deadlines, including this playbook's 8 related controls.
- 132 controls in Excel
- Score maturity and assign owners
- Track deadlines and regulation coverage
Includes AI Governance Weekly every Thursday. Unsubscribe anytime.
Governance Controls
Operational controls that implement the guidance in this playbook.
Recent Coverage
News and developments relevant to this playbook topic.