IETF Standards

The IETF (Internet Engineering Task Force) is the open standards body that writes the technical rules behind much of the internet, including protocols for secure communication, data transfer, and identity. Its working groups are now producing specifications for how AI agents identify themselves, authenticate, and exchange data with other systems. For compliance and risk teams, these drafts signal where future auditable requirements for AI-driven traffic are likely to land, so tracking them early helps shape internal controls before they become procurement or regulatory expectations.

1 item

StandardsGlobal2026-10-31

Major Labs Back Unified Scraping Opt-Out, Creating New Vendor Due Diligence Obligations

OpenAI, Google, Microsoft, and Anthropic have committed to a shared technical signal proposed through the Internet Engineering Task Force (IETF). It lets website publishers opt out of AI training data collection using a single instruction in their existing robots.txt file. Anthropic has already adopted the signal. Google, Microsoft, and OpenAI plan to follow in the coming months. Microsoft will also allow publishers to distinguish between opting out of AI training and opting out of AI grounding. AI grounding is the use of live web content to inform AI answers in real time.

web scraping training data data governance content licensing IETF standards OpenAI Google Microsoft Anthropic vendor due diligence