This can happen due to 1 of 3 reasons:
1 - you crawl a URL that also includes a /directory/ like this:
Site.com/products
🔥 When you enter a URL with a /directory/, by default that directory will be auto-populated in the ADVANCED SETTINGS > Include Directories
This is because 99% of the time it is the user's intention to only crawl links inside that directory.
Therefore, it will ONLY crawl pages that have the /products/ in their URL.
👉 However, If you wish to include pages outside of this directory simply select this option prior to your crawl.
2- by default we skip all URLs that have parameters ( "?..." )
example: /site.com/?some-link
Solution: Go to Advanced Settings > Turn this OFF!
We disable this because 99% of those pages are duplicate product or paginated pages that you usually don't want to crawl anyway,
3 - Some of those missing pages are NOT located inside your BASE URL, and they point to external sub-domains or domains.
REMEMBER: our crawler can only crawl INTERNAL urls its finds on the base url you entered to crawl. If you want to show links to other external pages, simply add them manually via our Editing options