Our crawler

What it reads

Only what a Quinndatum customer asks about. When someone audits a page or a site, our server reads, with this crawler:

• the page they asked about, or, for a site audit, a sample of the site's pages;
• the site's robots.txt, and the first sitemap it names;
• /favicon.ico, when a page names no icon of its own;
• the canonical address and the language alternates a page names, to check they answer;
• in a site audit, up to 100 of the links on the pages read, to check that they answer;
• the http, https, www and non-www forms of the home address, to see where each one leads.

It reads the HTML your server sends. It does not run your pages' JavaScript, fill in forms or sign in to anything.

How it names itself

Every request carries this user agent:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Quinndatum/1.0; +https://quinndatum.com/bot.html) Chrome/124.0.0.0 Safari/537.36

Its robots.txt product token is Quinndatum. The browser words are there for the same reason as in Google's own crawler: some servers send a different page to anything that does not look like a browser, and we want to read the page your visitors get.

How fast it goes

During a site audit, page reads start at least half a second apart. A single page audit reads one page and the few files listed above.

How to stop it

Add a group for it to your robots.txt. To stop it reading anything on your site:

User-agent: Quinndatum
Disallow: /

A group that names Quinndatum is obeyed on every request, including a single page a customer asks about: they are told your robots.txt asks us not to read it, and nothing is fetched. A Disallow on a narrower path stops reads of that path only.

With no group naming us, a site audit follows your User-agent: * group. A single page a customer asks about is still read, the way their own browser would read it, and the report tells them your robots.txt disallows it.

We read robots.txt at the start of every site audit, and keep a copy for at most an hour between reads.

Questions

Email info@quintessentianetwork.com with "Crawler" in the subject. Quinndatum is a service of Quintessentia Network Inc., Ontario, Canada.