Tackle Watch

A personal restock monitor run by one angler. It checks a handful of public product listings for new items and restocks, usually every 15 minutes and at most once a minute for a page the owner is waiting on, so the owner gets a phone notification instead of refreshing the page by hand. It only reads pages of shops the owner has approved, prefers one brand or search page (or a feed) over many product pages, reads a shop's announcement feed (blog RSS) every 15 minutes, and reads a single product page only to see which colors are in stock when a listing it follows comes back. For a shop the owner is considering, it reads at most three pages (the home or store info page, the shipping guide, one product page), at most one page a minute, to learn whether it ships to the United States (a shop in the United States is never read for this). Marketplace searches use Rakuten's and Yahoo!'s official APIs. On Shopify stores its regular checks read only a single product's or collection's product data, never the whole catalog. When the owner asks for a price check of one product, it looks inside each approved shop once, with one or two requests (the shop's own search page where its robots.txt allows it, a page addressed by barcode, or one maker's product list), under the same per-site limits below; it doesn't follow a redirect to another site, and if the shop refuses it leaves the shop alone for at least six hours. It reads one public exchange rate (the ECB reference rate via frankfurter.dev) once a day when the owner turns on cost estimates, and otherwise only when a price check needs it, at most once an hour. It never adds to a cart, places orders or buys anything.

• It identifies itself as TackleWatch/2.0 (+https://tacklealert.com/bot).
• It reads and follows robots.txt, re-checked daily, and honors Crawl-delay.
• It sends one request to a site at a time, at most 2 a minute, started at least 30 seconds apart (longer if your Crawl-delay asks for it), and waits at least 5 seconds after one finishes before starting the next. Checks the owner starts by hand follow the same rules. Reading robots.txt (at most once a day per site) happens just before one of those requests.
• Rarely, for a product release the owner is waiting on, the owner can set a window of at most 60 minutes in which one page per site is checked every 30 seconds instead of once a minute. That stays within the per-site limits above, so the site's other pages are checked less often (or not at all) during the window.
• If a site answers 429 Too Many Requests, it waits as long as Retry-After says.
• It never works around bot protection, CAPTCHAs or blocks.

To opt out, add this to your robots.txt and it will stop within a day:

User-agent: TackleWatch
Disallow: /