- Cloudflare (NET, Financial) introduces a new default setting blocking AI crawlers from accessing content without permission, enhancing content creator control.
- Website owners can now decide if and how AI companies can use their content, leading to a more sustainable future for the internet.
- Leading content and media companies endorse this permission-based approach to protect original content and foster a fair digital ecosystem.
Cloudflare, Inc. (NET) has announced a significant shift in its approach to online content management by becoming the first internet infrastructure provider to block AI crawlers from accessing content without explicit permission or compensation. This new default setting allows website owners the autonomy to choose if AI crawlers can access their content and under what terms, thus promoting a more sustainable and equitable internet.
This change addresses the longstanding issue where AI crawlers scrape content, such as text, articles, and images, without directing users back to the original source, thereby depriving content creators of potential revenue. With the inability to generate revenue from their content, creators might lose the incentive to produce quality content, thus threatening the future vibrancy of the internet.
Matthew Prince, Cloudflare’s CEO, emphasized the importance of creators maintaining control over their content to ensure the internet remains a dynamic and innovative space. He highlighted the need for a new economic model that benefits content creators, consumers, AI developers, and the future web.
Leading content, media, and technology companies, including BuzzFeed, Inc., Condé Nast, and Pinterest, have expressed strong support for Cloudflare’s initiative. They see this as a pivotal step toward ensuring fair compensation for original content and fostering sustainable innovation online.
The initiative is expected to reshape the digital landscape by requiring AI companies to be transparent about their use of crawlers, whether for AI training, inference, or search purposes. This transparency enables content creators to make informed decisions about which AI crawlers to permit, ensuring a fair value exchange.
As Cloudflare continues to bolster this permission-based model, it aims to provide website owners with an upfront choice regarding AI crawler access to their content. This move is supported by over one million of Cloudflare's customers who have already elected to block AI crawlers, marking a significant step towards redefining internet content management.
