Quick Take
Cloudflare has rolled out a trio of proactive tools (managed robots.txt, ad-page crawler blocking, and pay-per-crawl) that empower CIOs to regain control over AI bot access. Evaluate and enable these features in your Cloudflare settings to bolster efforts to protect content monetization and regain ownership over your data.
Why You Should Care
- Rising web crawler strain. The crawl-to-referral ratio can hit 1,500:1 for OpenAI and Anthropic’s can hit 60,000:1. That’s bandwidth and ad revenue drained, with little traffic returning to your site. Along with the aggressive crawling, chatbots may sometimes provide insufficient referral links or users are not using the referral links pointing to the original content.
- Automated crawler management. Cloudflare now creates and automatically updates robots.txt files to simplify the process and keep pace with evolving bot behaviors.
- Context-aware ad protection. The ad-page crawler blocking feature blocks web crawlers specifically on ad-supported pages, helping to prevent revenue leakage where it hurts most.
- Monetization via pay-per-crawl. Currently, in private beta, pay per crawl lets you charge bots for crawl access, effectively turning scrapers into paying customers.
What You Should Do Next
Audit current bot access settings in Cloudflare and start planning to implement a pay-per-crawl strategy if your content is heavily scraped.
Get Started
- Enable managed robots.txt feature. Let Cloudflare automatically adjust exclusions based on evolving AI crawler types.
- Activate ad-page crawler blocking. Prevent bots from accessing ad zones to protect your revenue streams.
- Apply for the pay-per-crawl beta. Define a per-request fee for bots to test monetization.
- Consider multiple guardrails for increased protection. Use other Cloudflare features like AI Labyrinth deception to trap web crawlers in a maze of fake content.
- Develop an AI bot policy that classifies web crawlers (allowed, paid, or blocked), and integrate this policy into your vendor/partner agreements.