This happens when you crawl a URL that also includes a /directory/ like this:

🔥 When you enter a URL with a /directory/, by default that directory will be auto-populated in the ADVANCED SETTINGS > Include Directories
This is because 99% of the time it is the user's intention to only crawl links inside that directory.

Therefore, it will ONLY crawl pages that have the /products/ in their URL.

👉 However, If you wish to include pages outside of this directory simply select this option prior to your crawl.

Did this answer your question?