AI Governance Institute
← News
Standards2026-10-31

Major Labs Back Unified Scraping Opt-Out, Creating New Vendor Due Diligence Obligations

What happened

OpenAI, Google, Microsoft, and Anthropic have each committed to adopting a shared "policy token" signal proposed through the Internet Engineering Task Force (IETF), as reported by The biggest names in AI are backing a new way to opt out of scraping. The signal lets a website publisher add one line to its robots.txt file. That file is a standard web instruction file. It tells all participating AI crawlers to stop collecting content for AI training. Anthropic has already adopted the signal. Google and Microsoft have committed to supporting it in 2025, and OpenAI plans to follow in the coming months. Microsoft has added a notable distinction. Publishers will be able to separately opt out of AI training versus AI grounding. Grounding means using a live copy of a webpage to inform an AI answer without storing the content for future model training.

Why it matters

  • ·Organizations that publish proprietary content online now have a concrete mechanism to restrict its use in AI training. Failure to deploy the signal promptly could weaken copyright and licensing claims if content is collected before the opt-out is in place.
  • ·The training-versus-grounding distinction Microsoft is introducing creates two separate compliance exposures. Training raises data provenance and copyright questions already under litigation. Grounding raises accuracy, data currency, and potential liability questions that most vendor contracts do not yet address.
  • ·Enterprises buying AI products that use web-crawled data must now ask vendors whether and how they honor the new signal, and how they handle the training-versus-grounding distinction. Most existing vendor due diligence questionnaires and AI procurement contracts do not yet cover this. This gap could affect regulatory exposure under copyright law and emerging AI transparency requirements.

Stay ahead of stories like this

Get every Global AI governance development like this one, plus the rest of the week's developments. Every Thursday.

Free every Thursday. Unsubscribe anytime.

Governance controls affected

DGC-001Training Data ProvenancePRC-001AI Vendor Due DiligencePRC-002AI Contractual RequirementsPRC-008Vendor Model Update Disclosure and Re-Assessment ProtocolMGV-005Generative AI Input Data Classification

What to do now

  • ☐Ask your web team whether your organization's robots.txt files currently include any instructions limiting AI crawlers, and confirm whether the new IETF policy token signal has been added or is planned.
  • ☐Review your most valuable proprietary content assets and decide, with legal counsel, whether to opt out of AI training, AI grounding, or both, before the major labs complete their rollout.
  • ☐Update your AI vendor due diligence questionnaire to ask each vendor whether it honors the new policy token signal, how it distinguishes between training and grounding use of crawled content, and what happens to content collected before a publisher opt-out was in place.
  • ☐Review existing AI vendor contracts to determine whether training data sourcing and grounding data sourcing are addressed separately, and flag contracts that lack this distinction for renegotiation or addendum.
  • ☐Ask your legal team to assess whether any AI products your organization currently uses could expose you to third-party copyright claims based on training data collected before this standard existed.

What to watch next

Compliance teams should monitor whether the IETF formally adopts the policy token as a published standard, which would give it broader weight in contract negotiations and regulatory discussions. The training-versus-grounding distinction will likely attract scrutiny from regulators and courts. They are already examining AI training data practices, particularly in the EU under the EU AI Act (Regulation (EU) 2024/1689) and in ongoing US copyright litigation. Watch for vendor updates from Google, Microsoft, and OpenAI confirming live support for the signal. Also watch for any enforcement action or court ruling that treats failure to honor a publisher opt-out as a copyright violation.

Related Coverage

Corporate Policy2026-10-23

Meta's Revised Terms Still Permit AI Training, Legal Experts Warn

Meta updated its business account terms in October 2026 to remove explicit references to using scraped content for AI 'training,' following publisher pushback. Legal experts quoted by Nieman Lab argue the remaining 'develop' language is broad enough to cover model training. They add that robots.txt preferences carry no contractual force under the revised terms.

Corporate Policy2026-10-30

Wikimedia's NYT v. OpenAI Brief Draws a Copyright Line AI Buyers Must Track

The Wikimedia Foundation filed an amicus brief in the New York Times v. OpenAI and Microsoft litigation, arguing that pre-training AI on copyrighted works may qualify as fair use, but that AI systems reproducing content verbatim in outputs require a license. The brief introduces a specific two-part framework that, if adopted, would materially change how enterprises assess legal risk in AI procurement and deployment. The case remains pending, but the distinction Wikimedia draws is already relevant to compliance programs today.

Research2026-10-30

Five AI Coding Tools Found Vulnerable, Two Still Unpatched

Security researchers at Forever Security disclosed exploitable flaws in Cursor, GitHub Copilot, Anthropic Claude Desktop, Microsoft VS Code, and OpenAI Codex. The vulnerabilities allow attackers to run unauthorized code on developer machines or read sensitive files. Flaws in OpenAI Codex and Microsoft VS Code remain unpatched as of the disclosure date.