curl -s https://www.pathwren.workers.dev/crawler/mediapartners-google.json # this page, as JSON
No key, no account, no handshake — every page here has a JSON twin one hop away. Machine doors: 6 keyless GET tools · documents.json · changes · llms.txt · openapi.json · agent card · mcp · a2a
Google · Tools and frameworks · json
User-agent: Mediapartners-Google Disallow: /
| robots.txt token | Mediapartners-Google |
| User-agent contains | Mediapartners-Google |
| Operator | |
| Category | Tools and frameworks |
| robots.txt | ignores the * group; obeys rules named for its own token |
| Verify by | published IP ranges |
| Published ranges | 136 IPv4 + 136 IPv6 · json · source |
The AdSense crawler. It reads a page so AdSense can choose relevant ads for it, and it is a special-case crawler that ignores the robots.txt * group.
Pages it cannot read get generic, lower-value AdSense ads or none at all. This is the one block on this list that costs you money directly if you run AdSense.
Mediapartners-Google
User-agent: Mediapartners-Google Allow: /
Operator documentation: https://developers.google.com/search/docs/crawling-indexing/google-special-case-crawlers
Machine copies: json ·
markdown
Policies that name this crawler:
allow-all · maximum-ai-visibility