HAWK-crawler

HAWK is a research project that maps the public domain-name market: which technologies, hosting providers and security settings websites use, measured in aggregate. If you see HAWK-crawler/1.0 in your logs, this page explains what it did.

What it does

• Reads /robots.txt first and follows it.
• Then requests your homepage (/) once, following at most three redirects, and reads at most 128 KB.
• Never submits forms, logs in, follows other links or crawls other pages.
• Keeps at most two connections to any one server address, with a pause between visits. Large content-delivery networks (such as Cloudflare), where one address answers for millions of different sites, get more parallel visits, each to a different site.

What it keeps

Only derived facts: the response status, the server software, security headers (such as HSTS), the technologies the page reveals (for example the CMS or a chat widget) and the page title. It does not store page content, and it does not collect personal data such as names, email addresses or phone numbers.

Opting out

Add this to your robots.txt and the crawler will not request your homepage:

User-agent: HAWK-crawler
Disallow: /

Or email ophawk@fastmail.com with the domain names to exclude, or with any question.

User agent: HAWK-crawler/1.0 (+https://op-hawk.fly.dev/crawler; ophawk@fastmail.com)