crawlsignals

How we read

• What we fetch: robots.txt and the addresses it names; the home page, read only for the links in its footer and never quoted; pages those links lead to whose address or link text names terms, legal, conditions, privacy, developers, a licence (or the Norwegian vilkår, betingelser, lisens), or whose link text names an API or data, and one more such link from those pages; /.well-known/ files and /llms.txt. Nothing else, unless a person approved that address: we never guess addresses no page links to. Never listings, posts, profiles, search results or data. A document in several languages is read once; a PDF is read as its text.
• How we fetch: one request at a time, at least two seconds apart per host (longer if its robots.txt asks), as crawlsignals/0.1 (+https://crawlsignals.com), with robots.txt obeyed for every page and every redirect. A page we are refused is never fetched another way.
• robots.txt is described as RFC 9309 reads it, over the whole file: an agent with its own group follows only that group, not the * rules; the longest matching rule wins; a 4xx answer means no rules; a server error means every path is disallowed. Example paths are only examples.
• Our crawler is stricter than the standard. Only a 404 or 410 counts as "no robots.txt"; any other refusal (such as 403, 418 or 429) we treat as every path disallowed, because refusing a crawler its robots.txt is a signal too. Where robots.txt's comments prohibit crawling, we fetch nothing else from that host, even where its rules would allow it.
• Not read by us marks a page we did not read: a refusal, no answer, a page that needs JavaScript, a format we do not read. It is our gap, not a statement about the site. Where our fetcher is refused, a person may read the page in a browser; such quotes say "read by hand".
• Quotes are copied from the page as read on the date shown, and a program checks that each appears word for word, on word boundaries, in the saved text of a successful answer.
• Open by design means an official API, feed or export that anyone, or anyone with a free key, may use, with the data under either a standard licence (CC0, CC BY, CC BY-SA, NLOD, ODbL, EU Commission Decision 2011/833) or the site's own terms granting reuse, each quoted. Meet these and send your site's address to hello@crawlsignals.com for a re-check.
• Nothing here is legal advice, and nothing says what you may do. We describe what each site says.

About

crawlsignals is for anyone who asks, before a project, what a website says about being read by machines. It is free and carries no advertising. No site pays to be listed, ranked or removed, and every site goes through the same steps.

• Contact: one address for everything, hello@crawlsignals.com. We aim to answer within 14 days.
• Adding a site takes its address, sent by anyone. We find its documents ourselves, as described under How we read. What we cannot find shows as "not found by us", and the site's owner may then send us the link. Sending an address again asks for a re-check.
• Corrections: if an entry misquotes a page, misses a newer version or shows text the site no longer has, write to us with the page's address. We judge a correction on the evidence, whether the text is on the page, not on who asks, so no proof of ownership is needed. We re-read the page the way it was first read, and the entry shows the new date. Where the mistake was ours, the entry says so, with the date. Which passages are quoted follows the same checklist for every site and does not change on request.
• Removing a site: an owner can ask us to remove their site. Write to us with the site's address, and we send a confirmation to an address on the site's own domain. Once it is confirmed, we remove the entry and our fetcher no longer reads the site. The index then shows "removed at the site's request" with the date.
• Who makes it: the makers of modelsentiment.com, which collects posts and comments from AI subreddits through Reddit's RSS feeds. Reddit's entry, its robots.txt included, is read the same way as every other site's.
• Data licence: our records (the fields we write, the readings we compute and the compilation) are under CC BY 4.0: reuse them, crediting crawlsignals.com. Text quoted from a site, its robots.txt included, belongs to that site's owner and is not under our licence. The records are one file, records.json. Our own robots.txt allows every crawler.

Privacy

• The controller is the operator of crawlsignals.com, in Norway, reachable at hello@crawlsignals.com.
• This page sets no cookies, runs no scripts and counts no visitors. Our server keeps no log of requests. Cloudflare (USA) carries traffic to it and processes your IP address to do so (its privacy policy).
• If you write to us, Cloudflare passes your message on to our mailbox at Microsoft (USA), and we keep it to answer and as the record of what was asked: for two years, except a message that confirmed a removal, which we keep while the removal stands. We never publish who wrote to us; a removal shows only that the site asked, and when.
• We do this on our legitimate interest in running the site and answering what is sent to us (GDPR Art. 6(1)(f)), and you may object to it.
• You can ask to see, correct or delete what we hold about you at hello@crawlsignals.com, and you can complain to Datatilsynet, the Norwegian data protection authority.