Cloudflare is set to dramatically reshape the landscape of online content consumption by artificial intelligence. The company has announced a significant new policy, imposing a strict deadline on the AI industry to clearly distinguish between traditional web crawlers used for search and those deployed for AI agents and model training.
Come September 15, 2026, Cloudflare’s default settings will automatically block “mixed-use” crawlers from accessing any web pages that host advertisements. This pivotal change is designed to give content creators greater control and fair compensation in an increasingly AI-driven digital world.
A New Era for Web Content and AI Monetization
This policy update marks a critical turning point, impacting how AI model providers gather information for training purposes and power their advanced agentic services. Cloudflare’s proactive stance aims to strike a balance where publishers can remain discoverable without their valuable intellectual property being freely exploited.
Under the new default blocking settings, a broad spectrum of users will be affected. This includes all new Cloudflare customers, any new sites launched by existing customers, and importantly, all existing free customers. Unless site owners actively adjust their preferences, crawlers that conflate search, agent use, and AI training will simply be denied access to monetized content.
Cloudflare underscores a fundamental truth: while most website owners welcome discoverability through search and even some AI services, they also demand robust protections against their content being used without permission or proper attribution. This policy is a direct response to that pressing need, fostering a more equitable web ecosystem for everyone involved.
Addressing Industry Giants and the Rise of Bots
Without explicitly naming names, Cloudflare has pointedly critiqued the “world’s largest search engine” – a clear reference to Google – for having access to “2x more information” than other AI companies. This advantage, Cloudflare suggests, stems from the difficulty publishers face in remaining discoverable without inadvertently contributing their content for AI uses.
Google has historically defended its practices, highlighting its “Google Extended” bot, which allows site owners to opt out of their content being used for AI training and specific AI products like Gemini Apps and Vertex API. However, Google’s flagship Googlebot, essential for search inclusion, inherently supports AI features like AI Overviews and AI Mode, creating a complex dilemma for publishers.
Cloudflare co-founder and CEO Matthew Prince emphasized the urgency of these measures, referencing the recent, earlier-than-expected milestone where bot traffic surpassed human traffic online. Prince stated, “Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge.”
Introducing “Pay Per Use”: A Fairer Content Economy
Beyond simply blocking, Cloudflare is actively building solutions to empower publishers in the AI era. The company, which has previously launched tools to combat malicious AI bots, is now evolving its “Pay Per Crawl” marketplace into a more sophisticated and equitable “Pay Per Use” model.
This innovative framework will allow publishers to charge AI companies not merely for fetching their content, but specifically when that content actively creates value within an AI application or service. This represents a significant shift, focusing compensation on the actual utility and economic benefit derived from the content, rather than just its access.
The “Pay Per Use” model also promises practical efficiencies for content providers. Cloudflare’s internal data reveals that over 50% of crawl traffic from AI crawlers is spent re-fetching pages that remain unchanged. This new system can help conserve publishers’ valuable bandwidth and computing resources by reducing redundant data requests, leading to a more efficient web overall.
To kickstart this initiative, Cloudflare is partnering with Ceramic.ai and You.com. Publishers who opt-in can now receive direct payment when their content features in Ceramic’s AI search results or when You.com accesses premium content. This adaptable model can be customized by other AI companies, paving the way for a more transparent and financially sustainable future for web publishers.
Cloudflare’s new tools and partnerships aim to provide website owners with increased visibility and commercial opportunities, while also benefiting AI companies through bots with clear and transparent intent. This move is a strong signal to the industry, encouraging mixed-use crawlers to separate their search functions from their agent use and training activities for a healthier web ecosystem.
Source: TechCrunch – AI
