AI web crawlers that gather data for training AI models may be getting smarter, but they’re still falling for digital decoys dressed as juicy content. Open-source developers and platforms like Cloudflare are fighting back with tarpits which create fake pages that waste scraper resources, pollute training data, and expose rogue bots that sidestep robots.txt files. IT decision makers should not count on robots.txt compliance or rate limits alone; instead, they need layered defenses that actively deceive, delay and detect unauthorized scraping at scale.
Why You Should Care
- AI web crawlers are getting more aggressive and not playing by the rules. These crawlers are ignoring robots.txt files and consuming anything that they can retrieve. TollBit reported that robots.txt files were bypassed 26 million times for AI web scraping in March 2025. Anthropic’s web crawler, ClaudeBot, was accused of sending 1 million requests in 24 hours to iFixit’s website.
- This is not about bandwidth. Scrapers feed directly into LLM training pipelines meaning your proprietary data might end up in a competitor’s chatbot or tool. The reputational, legal, and compliance implications are mounting, especially in regulated sectors.
- New defenses don’t just block, they confuse and consume. Cloudflare’s AI Labyrinth, Radware’s Bot Manager and open-source tools like Nepenthes create fake content designed to overwhelm and entrap scrapers. This wastes their compute cycles and pollutes training data with junk, making them pay dearly for ignoring access controls.
What You Should Do Next
- Review your existing anti-bot strategy to ensure it includes deterrence, not just detection.
- Deploy deception defenses like AI Labyrinth and Bot Manager. Tools like Nepenthes are radical and they do not differentiate between crawlers for web indexing and web crawlers for AI training. It is best to completely understand your preferred solution.
- Configure trap endpoints to capture insights into crawler behavior and refine your security posture.
Get Started
- Enable tarpits like AI Labyrinth and Bot Manager in your bot protection settings to start misleading scrapers with auto-generated decoy content.
- Track trap activity and update defenses based on insights. Flag suspicious user agents, enforce CAPTCHA on repeat offenders, and inform upstream providers.
- Integrate deception into your bot management policy as a permanent layer alongside rate limiting, behavioral detection, and token-based access controls.