All Collections
FAQs
Can I crawl a site behind a login/password-protected?
Can I crawl a site behind a login/password-protected?

Crawling

Artur M avatar
Written by Artur M
Updated over a week ago

Yes. It's finally here.

👉 Please note: it is not 100% effective on all types of log-in pages.

Therefore, you will simply need to test it on your specific site, because it all depends on how your site technically allows for authentication.
​
Some sites are programmed to use cookies as part of authentication, others simply don't, as there are no standards for this yet.

ALTERNATIVELY - You can select Include Custom HTTP Header  in the new sitemap
form under *Advanced Settings* so that we send the header with
a unique value assigned to your team with every request we send.
This is very useful for whitelisting our traffic.

Did this answer your question?