Cloudflare will change its default settings on September 15, 2026, to block crawlers that mix search indexing, AI agent use, and model training from accessing any web page that hosts advertisements, the company announced Wednesday. The move is the latest escalation in the tug-of-war between publishers and AI companies over how web content is used and compensated.
The change applies to new Cloudflare customers, new sites set up by existing customers, and all existing free-tier customers, the company said. Existing paid customers will keep their current settings unless they manually opt into the new defaults.
Why Cloudflare is drawing a line between search and AI bots
Cloudflare co-founder and CEO Matthew Prince said the company is acting because bots now account for the majority of internet traffic — a milestone that arrived earlier than the industry had predicted. “Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” Prince said in the announcement.
Cloudflare argues that most website owners want their content to remain discoverable through search engines and even AI-powered services, but they object to having their intellectual property used for training without compensation or control. The company specifically called out the “world’s largest search engine” — a clear reference to Google — as having access to “about 2x more information” than other AI companies because its main Googlebot crawls for both search and AI features like AI Overviews and AI Mode.
Also read: SpaceX reportedly showed investors a prototype AI device — Musk says it's 'utterly false'
Google has pushed back on that characterization. The company provides a separate bot called Google Extended that lets site owners opt out of having their content used for training and AI products like Gemini Apps and Vertex API, and it says opting out does not affect a site’s inclusion in Google Search. However, Google’s flagship Googlebot continues to crawl for search, including AI-generated search features.
New payment tools for publishers
Cloudflare is also expanding its tools for publishers to monetize AI access. The company previously launched a marketplace called Pay Per Crawl that lets websites charge AI bots for scraping. That product is now evolving into “Pay Per Use,” which will allow publishers to charge AI companies when their content creates value — for example, when it appears in an AI search result — rather than simply when it is fetched.
Cloudflare’s data suggests that over 50% of crawl traffic from AI bots is spent re-fetching pages that have not changed, wasting publisher bandwidth and compute resources. The new model aims to reduce that waste while creating a direct revenue stream.
Cloudflare is initially working with two partners: Ceramic.ai and You.com. When a publisher opts into the program, they are paid when their content appears in Ceramic’s AI search results or when You.com accesses premium content. Cloudflare says other AI companies can customize this model for how they work.
What this means for the AI industry
The policy could significantly reshape how AI model providers access web content. If other web infrastructure providers follow Cloudflare’s lead, AI companies may need to negotiate directly with publishers for training data and agent access, rather than relying on open web scraping.
Cloudflare’s default block applies only to “mixed-use” crawlers — those that combine search, agent, and training functions in a single bot. Bots that are clearly labeled for a single purpose, such as pure search indexing or pure AI training, will not be affected by the default block. The company said it hopes the change will encourage mixed-use crawler operators to separate their bots into clearly labeled, single-purpose crawlers.
“Cloudflare’s new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent,” Prince said.
Frequently Asked Questions
What exactly is a ‘mixed-use’ crawler?
A mixed-use crawler is a bot that combines multiple functions — such as indexing for search, powering AI agents, and training large language models — into a single crawl. Cloudflare’s new policy requires these functions to be split into separate, clearly labeled bots.
How will this affect Google?
Google’s main Googlebot serves search, AI Overviews, and AI Mode. Cloudflare argues this gives Google an advantage over competitors who must use separate bots. Google already offers Google Extended for opting out of AI training, but Cloudflare’s default block could pressure Google to further separate its crawlers.
What is Cloudflare’s Pay Per Crawl marketplace?
Pay Per Crawl is a Cloudflare tool that lets website owners charge AI companies for access to their content. It is evolving into ‘Pay Per Use,’ where publishers are paid when their content creates value in AI search results or products, not just when it is fetched.
Be the first to comment