Ever wondered how we find all those prices? Here’s the full story of what happens behind the scenes,
from the moment we discover a retailer to the second you see results on your screen.
Step 1: Building our list of shops
We keep a database of over 500 UK online retailers. We’re not just grabbing random websites – we actively research and add legitimate
stores that UK shoppers actually use.
Some are obvious: the Amazons and Argoses of the world. Others are specialists you might not know about unless you’re shopping for something specific –
the kind of place that only sells hiking gear or kitchen gadgets.
Before we add a retailer, we check a few things:
- Are they a real UK business with proper registration?
- Do they have actual consumer protections in place?
- Can we actually access their website to check prices?
Some websites are built in ways that make it impossible for us to extract pricing data. Others actively block automated systems like ours.
We work through these challenges one by one, retailer by retailer.
Step 2: Visiting websites (the smart way)
Right, so we’ve got our list of retailers. Now we need to actually visit their websites and find products.
We don’t just download every single page on every website. That would be incredibly wasteful – most pages on a retailer’s site aren’t product pages at all.
There are contact forms, blog posts, help articles, privacy policies, career pages, all sorts of stuff we don’t care about.
Instead, we’ve built a system that’s selective about what it looks at. We start from obvious places (homepages and category pages),
then follow links, but we’re picky about which ones.
Our system looks at each link and asks: “Is this likely to be useful?”
- A link that looks like
www.shop.com/product/blue-widget-12345 is clearly a product page. High priority.
- A link like
www.shop.com/about-us or www.shop.com/careers is obviously not a product. We skip it.
- A link like
www.shop.com/electronics is a category page – worth visiting because it links to products.
How do we know what’s what?
We look for patterns in the web address:
- Product pages often contain words like product, item, or p
- They often end with numbers (product IDs)
- Category pages contain words like shop, browse, category
- Non-product pages have giveaways like login, basket, help, blog
We also look out for pagination – when there are 20 pages of results, each page has different products, so they’re all worth checking.
What we actively avoid
- Blog posts and news articles
- Help pages and FAQs
- Login screens, account pages, password resets
- Basket and checkout pages
- Store locators, delivery information, size guides
- Those horrifically long URLs with question marks everywhere (usually tracking junk)
Step 3: Only UK stuff, please
Loads of retailers run international versions of their sites. We only care about UK pricing, so we actively filter out foreign versions.
If we spot paths like /fr/, /de/, /es/ (or other non-UK country codes), we skip them.
Step 4: Grabbing the important bits
Once we find a product page, we extract the information we need. Every retailer builds their site differently, so we teach our system where to look
for each retailer’s product name, price, stock status, images and more.
What we grab:
- Product name – cleaned up so it’s readable
- Current price – handling sales, VAT, and common price quirks
- Stock status – in stock, out of stock, pre-order
- Product images
- Specifications (where available)
- Variants (sizes, colours, models)
Sometimes this breaks. Retailers redesign their websites and our extraction logic can stop working. We monitor for those breakages and fix them quickly.
Step 5: Working out what’s the same product
The same product appears across different retailers but is described in different ways. We use smart matching that considers brand, model numbers,
specifications, and other signals so we can group identical products together.
Get it wrong and the site becomes a mess. Show duplicates and users get confused. Miss matches and we fail to show the comparisons you came for.
We’re constantly refining this to get it right.
Step 6: Keeping everything up to date
Prices change all the time. Stock runs out. New products launch every day. We re-visit retailers on a schedule:
- Important product pages: checked multiple times per day
- Category pages: checked daily, to spot new products
- Less popular stuff: checked less frequently to save resources
Step 7: Quality checking everything
Not everything we find makes it onto the site. We run checks to filter out rubbish, for example:
- Dodgy prices (e.g. £9.99 or £999,999)
- Fake products and unreliable listings
- Stock status nonsense that doesn’t line up across sources
- Broken websites or data that goes stale
Step 8: Showing you results (instantly)
When you search Price Detector, you’re not triggering us to crawl dozens of websites in real time. That would take ages.
Instead, you search our database of products we’ve already found and catalogued.
Results are sorted by price by default – lowest first. And here’s the crucial bit:
no retailer can pay to appear higher in the results. No sponsored placements disguised as genuine results.
The hard bits (explained simply)
- Scale: millions of URLs to assess without hammering retailer servers
- Websites keep changing: extraction logic must adapt constantly
- Modern sites are complicated: JavaScript-heavy pages can be harder to read reliably
- Some sites don’t want us there: rate limiting, CAPTCHAs, blocking – we respect robots rules
- The price you see isn’t always the price you pay: VAT, delivery, promotions, memberships
- Matching products is genuinely hard: too strict misses matches, too loose merges different items
Why we do it this way
Many comparison sites only show retailers who provide a data feed (or pay to be included). That’s easier, but it means you don’t get a complete view.
We do it the hard way so we can include any UK retailer, whether they’re paying us or even know we exist.
What happens next
Right now, we’re actively showing products from
95
retailers. We have 500+ in our database that we’re working through.
Every week, that number goes up as we confirm extraction quality, fix issues, improve matching, and expand coverage.
The goal is simple: if you can buy it online in the UK, Price Detector should be able to show you where to get it cheapest.
If there’s a retailer you want us to add, see our retailers page or use our contact form.
Built in the UK. For UK shoppers. Since 2025.