The CoffeeCatalogue crawler

This page describes the crawler operated by wanted.coffee. If you found us in your webshop's access log, everything you need to identify, allow, or block us is below.

Who we are

wanted.coffee is a specialty-coffee catalogue. We list coffees that roasters are currently selling, and link customers back to the roaster's own shop to buy them. We do not sell coffee and we do not resell, republish, or mirror shop content as our own.

What we fetch

• Your robots.txt, before we look at anything else.
• Your public product feed or product listing pages — the same pages any customer can open without logging in.
• Periodic HEAD requests to individual product URLs we already list, to check whether that coffee is still on your shelf.

We send only GET and HEAD requests. We do not log in, submit forms, place orders, or attempt to reach anything behind authentication or payment.

How often

At most one full pass over a shop per day. Across all shops combined our crawler is paced to roughly two requests per second, and it waits at least one second between requests to the same shop.

How to identify us

Every request we make carries this User-Agent:

Mozilla/5.0 (compatible; CoffeeCatalogue/1.0; +https://wanted.coffee/coffeecatalogue)

The product token to match on in robots.txt is CoffeeCatalogue — that is the name our crawler answers to in a User-agent: line.

We crawl from one fixed address, 65.109.253.113. It resolves to crawler.wanted.coffee and that name resolves back to the same address, so you can confirm a visitor claiming to be us really is. Only crawl traffic leaves from it — our website and API use a different address.

How we behave

• We fetch and apply your robots.txt before crawling your product pages.
• If your address forwards somewhere else, we read the robots.txt of the place we end up as well, and stop there if it disallows us.
• We honour Crawl-delay, up to a ceiling of 30 seconds.
• We honour Retry-After when you send it.
• When a shop refuses us, we back off for that domain and slow down further on each repeat, rather than retrying at the same rate.

When two of your rules match the same path, the more specific one wins: we compare Allow and Disallow lines by how long the rule is, and the longest rule decides. If both rules are the same length, we follow Allow. This is the standard's own rule (RFC 9309), so a broad Allow: / next to a narrower Disallow: /out-of-stock/ works as you would expect: we stay out of /out-of-stock/ and fetch the rest. A blanket Disallow: / under our token has no Allow beside it, so it stops us completely, which is why that is the form we suggest below.

How to block us

Add this to your robots.txt and we will stop crawling your shop:

User-agent: CoffeeCatalogue
Disallow: /

That takes effect on our next pass, because we re-read robots.txt before each one.

That covers the stock checks too: for coffees we have already listed, we keep sending the occasional HEAD request to those product URLs to notice when they sell out, and those probes read your robots.txt first. Once it disallows us, we stop sending them and your coffees leave the catalogue until it allows us again.

To have your shop taken out of the catalogue altogether, email us — we will do it. robots.txt stops us fetching you; only a request to us removes what we already hold.

Contact

Questions, complaints, removal requests, or a rate that does not work for you: info@wanted.coffee. A human reads it.