All Collections
How do I prevent & fix duplicate pages?
How do I prevent & fix duplicate pages?
Jeff avatar
Written by Jeff
Updated over a week ago

Although this is not limited to eCommerce sites, we see this phenomenon happen a lot of them for product pages that have dynamic URLs like these:

Our crawling logic simply looks at VALID static <ahref> links it finds in pages and then structures the map tree based on the URL structure.

Today as far as it knows, those pages are unique, valid, and must be shown for a FULL audit.

>>> Most site owners are not even aware that their store app makes dynamic duplicate links. - this happens a lot for WooCommerce stores. <<

In this store's case, the Product's Add-To-Cart logic generates a unique add-to-cart ID URL every time our crawler hits that page.

you can see that here:
then by viewing the source and you will notice a dynamic link like this:
but if you refresh that page, and view-source again, that same add-to-cart link changes again to:

How to Prevent Duplicates ( Pre-Crawl first!) * your safest option.

This is the best way to generate a clean sitemap for ecommerce.

Use our powerful Advance Settings > Excluded Keywords Filter
It's that easy!

👉 For Woocommerce sites ( like the one above )
We recommend using these Exclude Filters > Keyword:


Exclude Directories:


👉 For Shopify sites :

We recommend using these Exclude Filters > Keyword:


How to Remove duplicates ( POST Crawl )

Ok, you did the filtering, GREAT!
but somehow a few dupes ( maybe 100's even ).. managed to get into the map and you just want to remove them all, rather than do a recrawl.

Solution: Use Search!

1. Click Search

2. Enter the Page Titles ( sorry only page titles work at this time.. URLs are coming )
3. Select All duplicate pages ( manually or click select all )
4. Click Delete.

Did this answer your question?