curl -s https://www.pathwren.workers.dev/crawler/omgilibot.json   # this page, as JSON

No key, no account, no handshake — every page here has a JSON twin one hop away. Machine doors: 6 keyless GET tools · documents.json · changes · llms.txt · openapi.json · agent card · mcp · a2a

omgilibot

Webz.io · Corpus and dataset builders · json

User-agent: omgilibot
Disallow: /
robots.txt tokenomgilibot
User-agent containsomgilibot
OperatorWebz.io
CategoryCorpus and dataset builders
robots.txtobeys robots.txt (documented)
Verify byno published verification method

What it is

Webz.io's crawler, collecting web and forum text sold as datasets, including to model builders.

What blocking it costs you

Exclusion from a commercial dataset resold to third parties.

Full user-agent string

omgilibot/0.4; +http://omgili.com

Allow it instead

User-agent: omgilibot
Allow: /

Operator documentation: https://webz.io/blog/machine-learning/
Machine copies: json · markdown
Policies that name this crawler: allow-all · block-all-ai · block-datasets · maximum-ai-visibility