All Collections
Some pages are missing...why?
Some pages are missing...why?

Here are some logical reasons..

Jeff avatar
Written by Jeff
Updated over a week ago

This can happen due to 1 of 2 reasons:

1 - you crawl a URL that also includes a /directory/ like this:

๐Ÿ”ฅ When you enter a URL with a /directory/, by default that directory will be auto-populated in the ADVANCED SETTINGS > Include Directories
This is because 99% of the time it is the user's intention to only crawl links inside that directory.

Therefore, it will ONLY crawl pages that have the /products/ in their URL.

๐Ÿ‘‰ However, If you wish to include pages outside of this directory simply select this option prior to your crawl.

2- by default we skip all URLs that have parameters ( "?..." )

example: /

Solution: Go to Advanced Settings > Turn this OFF!

We disable this because 99% of those pages are duplicate product or paginated pages that you usually don't want to crawl anyway,

Did this answer your question?