All Collections
FAQs
Can I crawl a site behind a login/password-protected?
Can I crawl a site behind a login/password-protected?

Crawling

Artur M avatar
Written by Artur M
Updated over a week ago

Yes. It's finally here.

👉 Please note: it is not 100% effective on all types of log-in pages.

Therefore, you will simply need to test it on your specific site, because it all depends on how your site technically allows for authentication.

Some sites are programmed to use cookies as part of authentication, others simply don't, as there are no standards for this yet.

ALTERNATIVELY - You can select Include Custom HTTP Header  in the new sitemap
form under *Advanced Settings* so that we send the header with
a unique value assigned to your team with every request we send.
This is very useful for whitelisting our traffic.

Did this answer your question?