Although this is not limited to eCommerce sites, we see this phenomenon happen a lot of them for product pages that have dynamic URLs like these:
​
​https://mysecretdrawer.co/au/product/jacquard-bow-tie-stockings/?add-to-cart=16159
​https://mysecretdrawer.co/au/product/jacquard-bow-tie-stockings/?add-to-cart=9274
​https://mysecretdrawer.co/au/product/jacquard-bow-tie-stockings/?add-to-cart=15014
​
IMPORTANT:
Our crawling logic simply looks at VALID static <ahref> links it finds in pages and then structures the map tree based on the URL structure.
​
​Today as far as it knows, those pages are unique, valid, and must be shown for a FULL audit.
​
>>> Most site owners are not even aware that their store app makes dynamic duplicate links. - this happens a lot for WooCommerce stores. <<
​
In this store's case, the Product's Add-To-Cart logic generates a unique add-to-cart ID URL every time our crawler hits that page.
​
you can see that here:
​https://mysecretdrawer.co/au/product/jacquard-bow-tie-stockings/
then by viewing the source and you will notice a dynamic link like this:
​https://mysecretdrawer.co/au/product/jacquard-bow-tie-stockings/?add-to-cart=16159
but if you refresh that page, and view-source again, that same add-to-cart link changes again to: https://mysecretdrawer.co/au/product/jacquard-bow-tie-stockings/?add-to-cart=9274
​
​

Use our powerful Advance Settings > Excluded Keywords Filter
It's that easy!
​
👉 For Woocommerce sites ( like the one above )
We recommend using these Exclude Filters > Keyword:
​

Exclude Directories:
​

👉 For Shopify sites :
​
We recommend using these Exclude Filters > Keyword:
​

Ok, you did the filtering, GREAT!
but somehow a few dupes ( maybe 100's even ).. managed to get into the map and you just want to remove them all, rather than do a recrawl.

Solution: Use Search!
​
1. Click Search

2. Enter the Page Titles ( sorry only page titles work at this time.. URLs are coming )
3. Select All duplicate pages ( manually or click select all )
4. Click Delete.
​