All Collections
FAQs
I only see 1 page result on a Password-Protected Login Crawl
I only see 1 page result on a Password-Protected Login Crawl
Jeff avatar
Written by Jeff
Updated over a week ago

Reason 1: Site-Access is not Cookie based.

First thing you should know is that not every website uses cookies to authorize user access to private content. This could be one huge reason why we can't crawl it.

Some sites are programmed to use cookies as part of authentication, others like yours may simply not, as there are no standards for this yet.
​

If you have access to the site's dev team, then another option is to whitelist our crawlers.

​Reason 2: Base URL is not correct.

Ensure that you enter the CORRECT BASE URL to crawl first.

Sometime the right url is the one that you get redirected to AFTER you login with your credentials. This could be something like Site.com/dashboard or my.Site.com/ or something similar.

note: The typical mistake we see is when you entered the actual login url (login.site.com), which will confuse our crawler since it will be limited to only crawl that URL.

Note: that the crawler needs the actual dashboard to start crawling from, and not necessarily the literal login URL.

​Reason 3: Base URL is correct, but the crawl was limited to the /directory/

Once you figure out the BASE URL, if it is has a /directory/ in the URL, then you will need to select this additional option:

otherwise the crawler will only crawl pages inside that /directory/.

Did this answer your question?