robots.txt policy: Block SEO and backlink crawlers

Ahrefs, Semrush and friends. No user-facing consequence, and often the largest single slice of your bot traffic. The cheapest bandwidth saving available to most sites, and the one nobody regrets. The only cost is that your own dashboards on those tools get thinner. Consider Crawl-delay first: both major operators honour it. curl -s https://www.pathwren.workers.dev/robots/block-seo-tools.txt