curl -s https://www.pathwren.workers.dev/crawler/crawlspace.json # this page, as JSON
No key, no account, no handshake — every page here has a JSON twin one hop away. Machine doors: 6 keyless GET tools · documents.json · changes · llms.txt · openapi.json · agent card · mcp · a2a
Crawlspace · Tools and frameworks · json
User-agent: Crawlspace Disallow: /
| robots.txt token | Crawlspace |
| User-agent contains | Crawlspace |
| Operator | Crawlspace |
| Category | Tools and frameworks |
| robots.txt | obeys robots.txt (documented) |
| Verify by | no published verification method |
A crawling platform: customers run their own crawls on it to feed agents, RAG pipelines and structured-data workflows. Like Firecrawl, the party behind any given request is the customer, not the platform.
Whatever any Crawlspace customer was building over your pages stops working. Volume and intent vary per customer, so this is a rate-limit decision more than a consent one.
Crawlspace
User-agent: Crawlspace Allow: /
Operator documentation: https://crawlspace.dev
Machine copies: json ·
markdown
Policies that name this crawler:
allow-all · maximum-ai-visibility