The CoffeeCatalogue crawler

This page describes the crawler operated by wanted.coffee. If you found us in your webshop's access log, everything you need to identify, allow, or block us is below.

Who we are

wanted.coffee is a specialty-coffee catalogue. We list coffees that roasters are currently selling, and link customers back to the roaster's own shop to buy them. We do not sell coffee and we do not resell, republish, or mirror shop content as our own.

What we fetch

• Your robots.txt, before we look at anything else.
• Your public product feed or product listing pages — the same pages any customer can open without logging in.
• Periodic HEAD requests to individual product URLs we already list, to check whether that coffee is still on your shelf.

We send only GET and HEAD requests. We do not log in, submit forms, place orders, or attempt to reach anything behind authentication or payment.

How often

At most one full pass over a shop per day. Across all shops combined our crawler is paced to roughly two requests per second, and it waits at least one second between requests to the same shop.

How to identify us

Every request we make carries this User-Agent:

Mozilla/5.0 (compatible; CoffeeCatalogue/1.0; +https://wanted.coffee/coffeecatalogue)

The product token to match on in robots.txt is CoffeeCatalogue — that is the name our crawler answers to in a User-agent: line.

How we behave

• We fetch and apply your robots.txt before crawling your product pages.
• We honour Crawl-delay, up to a ceiling of 30 seconds.
• We honour Retry-After when you send it.
• When a shop refuses us, we back off for that domain and slow down further on each repeat, rather than retrying at the same rate.

One limit worth stating plainly, because it is not what the standard asks for: when we read your robots.txt we look at your Allow lines before your Disallow lines, and the first Allow that matches a path decides it — even when a longer, more specific Disallow also matches. A blanket Disallow: / under our token has no Allow beside it, so it stops us, which is why that is the form we suggest below. A broad Allow: / sitting next to a narrower Disallow: /out-of-stock/ does not: we read it as permission and still fetch that section. The gap runs one way only — we may fetch something you meant to fence off, never the reverse. Until that changes, email us which parts of your shop to leave alone and we will exclude them.

How to block us

Add this to your robots.txt and we will stop crawling your shop:

User-agent: CoffeeCatalogue
Disallow: /

That takes effect on our next pass, because we re-read robots.txt before each one.

One thing that does not yet cover: for coffees we have already listed, we keep sending the occasional HEAD request to those specific product URLs to notice when they sell out. Those probes do not currently consult robots.txt. To stop those too, and to have your shop taken out of the catalogue altogether, email us — we will do it.

Contact

Questions, complaints, removal requests, or a rate that does not work for you: info@wanted.coffee. A human reads it.