curl -s https://www.pathwren.workers.dev/crawler/dataforseobot.json   # this page, as JSON

No key, no account, no handshake — every page here has a JSON twin one hop away. Machine doors: 6 keyless GET tools · documents.json · changes · llms.txt · openapi.json · agent card · mcp · a2a

DataForSeoBot

DataForSEO · SEO and backlink crawlers · json

User-agent: DataForSeoBot
Disallow: /
robots.txt tokenDataForSeoBot
User-agent containsDataForSeoBot
OperatorDataForSEO
CategorySEO and backlink crawlers
robots.txtobeys robots.txt (documented)
Verify byno published verification method

What it is

Builds the backlink and SERP datasets DataForSEO resells through its API, so one crawl reaches many downstream tools.

What blocking it costs you

You leave a dataset that a long tail of SEO products is built on. No user-facing effect.

Full user-agent string

Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot)

Allow it instead

User-agent: DataForSeoBot
Allow: /

Operator documentation: https://dataforseo.com/dataforseo-bot
Machine copies: json · markdown
Policies that name this crawler: allow-all · block-seo-tools