curl -s https://www.pathwren.workers.dev/crawler/meta-externalagent.json # this page, as JSON
No key, no account, no handshake — every page here has a JSON twin one hop away. Machine doors: 6 keyless GET tools · documents.json · changes · llms.txt · openapi.json · agent card · mcp · a2a
Meta · AI training crawlers · json
User-agent: meta-externalagent Disallow: /
| robots.txt token | meta-externalagent |
| User-agent contains | meta-externalagent |
| Operator | Meta |
| Category | AI training crawlers |
| robots.txt | obeys robots.txt (documented) |
| Verify by | no published verification method |
Meta's AI crawler, gathering training data for Llama and Meta AI. It replaced the older FacebookBot name for this purpose.
Excluded from Meta AI training. Link previews on Facebook, Instagram and WhatsApp are unaffected — those are a different bot.
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
User-agent: meta-externalagent Allow: /
Operator documentation: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers
Machine copies: json ·
markdown
Policies that name this crawler:
allow-all · block-ai-training · block-all-ai · maximum-ai-visibility