For site owners

Our crawler

News Quantified operates one automated fetcher, NewsQuantifiedAudit/1.0 (+https://newsquantified.com/bots). It checks how pages are structured for search engines and AI agents. It does not collect content for training, it does not resell what it reads, and it does not act on behalf of an end user — every request is started by us.

Seeing it in your logs and want it gone? Skip to blocking it.

How to recognise it

Requests carry this User-Agent, with the product token first:

NewsQuantifiedAudit/1.0 (+https://newsquantified.com/bots)

A user-agent string is not proof — anyone can send that text. So every request is also signed, and the signature is what you should verify.

Verifying that a request is really ours

We follow Web Bot Auth: HTTP Message Signatures (RFC 9421) with the profile from the IETF Web Bot Auth working group. Each request carries three headers:

  • Signature-Agent — "https://newsquantified.com", the origin publishing our keys.
  • Signature-Input — covers "@authority" and "signature-agent", tagged web-bot-auth, with created, expires, a nonce, and the keyid of the signing key.
  • Signature — an Ed25519 signature over those components.

Our public keys live at /.well-known/http-message-signatures-directory, served as a JSON Web Key Set. Take the keyid from Signature-Input, find the key with that kid, rebuild the signature base from the header, and verify. Because @authority is signed, a signature captured from one site cannot be replayed against another. The directory response is itself signed, so you can confirm the key set came from us.

Cloudflare verifies these signatures natively. If your site is behind Cloudflare, you do not have to implement anything.

What it requests, and how often

A handful of URLs per run: the page being checked, plus /robots.txt, /sitemap.xml and /llms.txt where they exist. Runs are started by hand during development, not on a schedule, so the traffic any single site sees is a few requests at a time and often none for weeks.

We never send more than one request per second to the same origin, and a Crawl-delay in your robots.txt raises that interval. It never lowers it. Requests to different sites run in parallel; requests to yours do not.

Blocking it

We read and obey robots.txt before every request, including the Allow, Disallow and Crawl-delay directives, and we honour the most specific matching rule. To turn us away entirely:

User-agent: NewsQuantifiedAudit
Disallow: /

That takes effect on our next run — there is no queue to drain and nothing to appeal. If your server returns a 5xx for robots.txt, we treat that as a full disallow and fetch nothing, rather than assume consent.

What it is not

  • It does not gather text for training models. Our dataset is built from licensed news feeds, not from crawling the open web.
  • It is not an AI assistant browsing on someone’s behalf. No end user directs it; we do.
  • It does not submit forms, follow links recursively, or attempt to reach anything behind authentication.

Contact

Questions, or something from us that looked wrong: support@newsquantified.com. Include the date, the path, and the Signature-Input header if you have it — that pins the request to a specific key and lets us tell our traffic from someone forging our user-agent.