All Collections
Tutorials
🔥 How to Crawl a Password-Protected / Private Website
🔥 How to Crawl a Password-Protected / Private Website

This is for only crawling password-protected websites.

Jeff avatar
Written by Jeff
Updated over a week ago

We're proud to announce that we've made it easier for you to crawl password-protected websites with this new secure method.
​

☞ Before you start

 : To ensure this works, we encourage you to first double-check and test that your credentials work on the site you wish to crawl.

  1.  Install our VisualSitemaps Chrome Add-on

  2.  Open a new tab in Chrome

  3.  Log into the site you want to crawl with your credentials.

  4.  Click on the Chrome extension "VisualSitemaps" 

> this will COPY ALL your Cookies into your clipboard to paste later...
​

OK Let's Do This!

  1.  Open a new tab and go to your VisualSitemaps Dashboard > "Create New Sitemap"
    ​

  2. Enter the URL you wish to crawl + Max Pages  and Max Depth  *we recommend first setting Max Pages to 3   for testing.
    ​

  3. Click Advanced Settings > Cookies

  1. PASTE all the Cookies ( from your Clipboard ) *this data is 100% encrypted and automatically deleted after every crawl. ( Mac = Command + V ) (PC = Cntrl + V)
    ​

  2. Click Crawl it 

  1. Once the test crawl is done and everything looks great, you are now ready to crawl the full site! 🚀
    ​
    ​ *NOTE: you will need to again copy all the cookies when setting up your new crawl. ( see step 4 > by going to the tab of the site you wish to crawl and click the Chrome icon again )
    ​
    · POWER TIP - you can use Advanced Filters to target specific directories or pages you want to filter out/in. Learn More>

-----------------------------------------------------------------------------------

SECURITY FAQ - Yes we want you to sleep well at night!

-----------------------------------------------------------------------------------

We use Enterprise-grade Symmetric Encryption (  AES 256 bits  )  to temporarily store the credentials that you provide. 

This means that none of VisualSitemap's employees have access to these values, and these are only decrypted and used when crawling.

Also, when the crawl is complete, we delete the encrypted credentials immediately!Seriously ...we don't want that liability.

-----------------------------------------------------------------------------------

Demo sitemap of Calendly's Dashboard >

Did this answer your question?