Our crawler
This page exists so that anyone who sees our requests in their logs can find out immediately who we are, what we are doing, and how to make us stop.
Identification
Our collector identifies itself on every request:
DiscountVerdictBot/0.1 (+https://discountverdict.com/bot; price-transparency research; contact: bot@discountverdict.com)
We never disguise our crawler as a browser, never rotate user agents, and never use proxies or residential IP pools to appear to be someone else. If you block us, you have blocked us — we will not come back wearing a different hat.
What it does
- Requests the retailer’s own public product data — the JSON endpoint shop platforms publish, or the schema.org product data provided for search engines.
- Records the selling price, the advertised original price, and availability.
- Runs once per day. We have no interest in real-time pricing.
We do not create accounts, submit forms, add anything to a basket, attempt checkout, or access anything that requires logging in. We do not collect personal data. We do not copy product descriptions, images, or reviews.
How it behaves
| robots.txt | Read and obeyed, including Crawl-delay. If your rules disallow a path, we do not request it. |
|---|---|
| Concurrency | One request at a time per site. Never parallel. |
| Rate | At least two seconds between requests to the same host, longer where you ask for it. |
| Volume | Typically a handful of requests per retailer per day. Most of our sources return an entire catalogue in one response. |
| 429 / 503 | We back off immediately and honour Retry-After. |
| 403 | Treated as a refusal. We stop for the day and flag the retailer for removal. |
How to block us
Add this to your robots.txt and we will stop, usually within a day:
User-agent: DiscountVerdictBot Disallow: /
Or email bot@discountverdict.com and we will remove you from our collection list and confirm when it is done. You do not need to give a reason.
Corrections and questions
For anything about the data itself — a figure you believe is wrong, or a product you think we have mismatched — write to contact@discountverdict.com. We will re-examine our record and correct anything we got wrong.
Last updated 10 August 2026.