@thom.code: Bots now send 57.5% of web page requests, and Anthropic's crawler takes roughly 70,000 pages for every single visitor it sends back. The dead internet theory finally has hard numbers behind it. This is the bot scraping economy from the server operator's side of the wire: what the three new kinds of AI crawler actually do, what they cost real sites like Read the Docs, iFixit and Wikimedia, and every rung of the defense ladder from robots.txt up to proof of work and feeding crawlers generated garbage. Every rung has a documented failure. Then the twist nobody planned for, where blocking the bots turned into a product and HTTP 402 Payment Required came back from thirty years of disuse. CHAPTERS 00:00​ Bots crossed the majority line 00:47​ Three AI crawlers: training, retrieval, user fetchers 01:54​ The receipts: Read the Docs, iFixit, Wikimedia 03:22​ The defense ladder: robots.txt, user agents, IP blocks 04:15​ Proof of work and Anubis 04:43​ Tarpits, mazes and poisoning the training data 05:14​ The polite bot problem and Perplexity 05:56​ Is scraping legal? The courts say maybe 06:30​ The toll booth: licensing deals and Pay Per Crawl 07:58​ What to actually do about it WHAT THIS VIDEO COVERS Why bots passed 57.5% of web page requests 18 months ahead of the prediction The split between training crawlers, retrieval crawlers and live user fetchers GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot and Claude-User Why AI crawler traffic re-downloads pages that never changed Real bandwidth bills from AI scraping, including 73TB in a month Why git forges and wikis get hit hardest, and what an uncached diff costs Whether robots.txt can block AI crawlers (RFC 9309 answers this directly) User agent blocking, IP blocklists, and why residential proxy swarms beat both Anubis proof of work, who runs it, and how bots learned to solve it Nepenthes, Iocaine and Cloudflare AI Labyrinth as crawler tarpits Cloudflare's honeypot test of crawler compliance and the Perplexity fallout The Google and Reddit scraping lawsuits against SerpApi Cloudflare Pay Per Crawl, HTTP 402, and the AWS payment protocol in CloudFront Reducing your URL space so crawlers cannot bankrupt endpoints that do not exist Web Bot Auth and cryptographically signed crawler requests WHAT IS THE BOT SCRAPING ECONOMY? Search crawlers used to trade traffic for access. They indexed your pages and sent readers back, and that exchange paid for the open web. AI crawlers broke the trade. Training crawlers copy your content to build the next model and return nothing, retrieval crawlers keep an index so an answer engine can quote you without a click, and user fetchers hit your server live because someone asked a chatbot a question you already answered. The result is a web where most page requests come from software, bandwidth bills land on small operators, and the compensation for the long tail is a firewall rule rather than a licensing check. SOURCES Cloudflare, "The crawl before the fall of referrals" (Jul 2025) and the crawler industry breakdown (Aug 2025) Matthew Prince (Cloudflare) on bots passing 57.5% of page requests (Jun 2026) RFC 9309, the Robots Exclusion Protocol Read the Docs, blog post on AI crawler bandwidth abuse iFixit, Kyle Wiens on ClaudeBot request volume Wikimedia Foundation Diff, "How crawlers impact the operations of the Wikimedia projects" (Apr 2025) Drew DeVault (SourceHut), "Please stop externalizing your costs directly into my face" (Mar 2025) Anubis by Xe Iaso, plus the Codeberg statement on solved challenges (Aug 2025) Nepenthes, Iocaine, and Cloudflare AI Labyrinth Cloudflare, honeytrap domain test and Verified Bots delisting (Aug 2025) Google v. SerpApi (N.D. Cal., Jul 2026) and Reddit v. SerpApi and others (S.D.N.Y., Jul 2026) Reddit S-1 filing, Reuters on the Google deal, WSJ on the News Corp and OpenAI deal Cloudflare Pay Per Crawl and AWS x402 support in CloudFront IETF Web Bot Auth drafts and RFC 9421 HTTP Message Signatures #AICrawl

Thom Code

Open In TikTok:

Region: NG

Wednesday 19 August 2026 07:25:33 GMT