Cloudflare will filter out web crawlers that serve AI companies
Engadget · 2 min read · tech
Cloudflare will automatically block mixed-use web crawlers that serve both search and AI training by default starting September 15, 2026. New customers and websites will default to allowing search crawlers while blocking training and agent use on ad-supported pages. The company also relaunched its Pay Per Crawl feature as Pay Per Use, paying site owners when their content appears in AI chatbot answers.
Mentioned entities