Big changes are coming to the way AI bots crawl the web. Cloudflare just flipped the switch on new rules that will block many AI agent crawlers by default starting September 15, 2026. This is huge. Why? Because these bots now make up more than half of the web traffic Cloudflare handles. The web as we know it is about to get a major rewrite in who gets to see what—and when.
What’s Changing and Why It Matters
Cloudflare announced on July 1, 2026, that from September 15, AI agent crawlers will face new default restrictions. These bots fetch pages for AI agents that answer real-time queries. But here’s the catch: if a page shows ads, AI Training and Agent bots will be blocked by default. Only Search crawlers will still get through. That means the default setting flips for many sites, especially those depending on ads.
Why block AI Training and Agent bots on ad-supported pages? Cloudflare’s logic is clear. Ads prove a page was built for real humans, not machines. Pages with news, reviews, pricing, and product coverage live behind ads. These are exactly the pages AI agents want to scan and repurpose. But that’s a problem for creators and publishers who rely on those ads to make money.
Cloudflare’s blocking works at the network level, not just through robots.txt files that bots can ignore. This means the new rules are tougher to bypass. The change affects new domains joining Cloudflare, new sites by existing customers, and all existing free-tier customers. Anyone who dislikes these blocks can opt out before September 15 by changing security settings.
Who’s Impacted and What They’re Saying
This move has major players paying attention. Patreon’s CEO, Jack Conte, supports the change. He says it will stop AI crawlers from training on creators’ work without permission. That’s a big win for artists and content producers worried about AI using their content without fair credit or pay.
Cloudflare covers a massive chunk of the world’s web traffic. Its rules don’t just nudge bots; they slam the door on certain AI agents by default. Agentic AI deployments assumed the web would stay open to all. Now, that assumption faces a serious challenge. The shift means AI companies must rethink how their bots gather data.
Stephanie Cohen, Cloudflare’s chief strategy officer, revealed that more than half of its web requests come from AI agents. That’s staggering. It shows how deeply AI has woven itself into internet traffic. Yet, this flood of AI requests can strain servers and threaten the economics of ad-supported sites. Blocking bots from these pages helps protect the web’s financial backbone.
What’s Next for AI Crawlers and the Web
Starting September 15, the web’s AI ecosystem will look different. Search crawlers can still operate as usual, sending readers back to original sites. But AI Training and Agent bots will hit a wall on ad pages. This could reshape how AI models learn from web content. It may slow down some AI agents or push them to seek data from permissioned sources.
For site owners, the power shifts. They can now control whether AI agents access their pages. This gives creators a chance to protect their work and revenue. For AI developers, the rules force innovation in data sourcing and respect for web property.
The open web is evolving. Cloudflare’s new default restrictions mark a turning point in AI and internet relations. It’s a bold step to balance human-centered content with AI’s growing appetite. What happens next will shape the future of AI agents—and the very fabric of the internet.
Based on
- AI agent crawlers now need permission. Here’s how to get it — artificialintelligence-news.com
- Cloudflare puts prices on AI scraping | Semafor — semafor.com
- Patreon is working with Cloudflare to squash AI crawlers. | The Verge — theverge.com
- AI giants learn what everyone else on the modern internet already knows | Business Insider Africa — africa.businessinsider.com
