Cloudflare (NET, Financial) slides 6.6% after announcing it will block AI crawlers by default and give publishers control over content access.
The move makes Cloudflare the first major internet infrastructure provider to require permission—or compensation via its private-beta “pay per crawl” feature—before automated bots can scrape data for AI training or inference.
Under the new settings, site owners can whitelist specific crawlers, designate their purpose, and enforce usage terms, putting creators back in the driver's seat amid surging AI demands.
CEO Matthew Prince warned that unchecked scraping threatens a “free and vibrant Internet,” and said the policy shift “safeguards the future” by balancing innovation with creator rights.
He noted that pay-per-crawl lets publishers charge AI firms for access, though it's limited to trusted partners in beta today. Reddit (RDDT, Financial) co-founder Steve Huffman backed the change, calling for transparency on crawler identities and intent.
Cloudflare also published a supporter list spanning more than 20 media and tech organizations—including The Associated Press, The Atlantic, BuzzFeed, TIME and Universal Music Group—underscoring broad industry demand for a permission-based model.
The timing coincides with rising tensions over AI's use of unlicensed content: recent estimates suggest AI crawlers have ingested billions of web pages without attribution or compensation.
By defaulting to block, Cloudflare shifts the burden to AI companies to negotiate access terms and could spark similar moves by competitors. While the policy may slow some data collection, Cloudflare believes most publishers will see revenue upside from licensing fees and stronger IP protections.
