VisualSitemaps

We're proud to announce that we've made it easier for you to crawl password-protected websites with this new secure method. ​

: To ensure this works, we encourage you to first double-check and test that your credentials work on the site you wish to crawl.

Install our VisualSitemaps <a href="https://chrome.google.com/webstore/detail/visualsitemaps/acogpmfdhkmhbnhfkcamjgdffklfofho" rel="nofollow noopener noreferrer" target="_blank">Chrome Add-on</a>

&nbsp;Log into the site you want to crawl with your credentials.

&nbsp;Click on the Chrome extension "VisualSitemaps"&nbsp;

1.  Install our VisualSitemaps <a href="https://chrome.google.com/webstore/detail/visualsitemaps/acogpmfdhkmhbnhfkcamjgdffklfofho" rel="nofollow noopener noreferrer" target="_blank">Chrome Add-on</a>
2.  Open a new tab in Chrome
3.  Log into the site you want to crawl with your credentials.
4.  Click on the Chrome extension "VisualSitemaps"

&gt; this will COPY ALL your Cookies into your clipboard to paste later... ​

Open a new tab and go to your VisualSitemaps Dashboard &gt; "Create New Sitemap" ​

Enter the URL you wish to crawl + Max Pages  and Max Depth  *we recommend first setting Max Pages to <code>3</code>   for testing. ​

Click Advanced Settings &gt; Cookies

1.  Open a new tab and go to your VisualSitemaps Dashboard &gt; "Create New Sitemap" ​
2. Enter the URL you wish to crawl + Max Pages  and Max Depth  *we recommend first setting Max Pages to <code>3</code>   for testing. ​
3. Click Advanced Settings &gt; Cookies

PASTE all the Cookies ( from your Clipboard ) *this data is 100% encrypted and automatically deleted after every crawl. ( Mac = <code>Command + V</code> ) (PC = <code>Cntrl + V</code>) ​

1. PASTE all the Cookies ( from your Clipboard ) *this data is 100% encrypted and automatically deleted after every crawl. ( Mac = <code>Command + V</code> ) (PC = <code>Cntrl + V</code>) ​
2. Click <code>Crawl it</code>

Once the test crawl is done and everything looks great, you are now ready to crawl the full site! 🚀 ​ ​ *NOTE: you will need to again copy all the cookies when setting up your new crawl. ( see step 4 &gt; by going to the tab of the site you wish to crawl and click the Chrome icon again ) ​ · POWER TIP - you can use <a href="https://support.visualsitemaps.com/en/articles/3076283-advanced-settings-filters">Advanced Filters</a> to target specific directories or pages you want to filter out/in. Learn More&gt;

1. Once the test crawl is done and everything looks great, you are now ready to crawl the full site! 🚀 ​ ​ *NOTE: you will need to again copy all the cookies when setting up your new crawl. ( see step 4 &gt; by going to the tab of the site you wish to crawl and click the Chrome icon again ) ​ · POWER TIP - you can use <a href="https://support.visualsitemaps.com/en/articles/3076283-advanced-settings-filters">Advanced Filters</a> to target specific directories or pages you want to filter out/in. Learn More&gt;

-----------------------------------------------------------------------------------

SECURITY FAQ - Yes we want you to sleep well at night!

We use Enterprise-grade <a href="https://en.wikipedia.org/wiki/Advanced_Encryption_Standard" rel="nofollow noopener noreferrer" target="_blank">Symmetric Encryption</a> (  <code>AES 256 bits</code>  )  to temporarily store the credentials that you provide.

This means that none of VisualSitemap's employees have access to these values, and these are only decrypted and used when crawling.

Also, when the crawl is complete, we delete the encrypted credentials immediately!Seriously ...we don't want that liability.

Demo sitemap of <a href="https://app.visualsitemaps.com/share/aa8c0d228e52aa867247dc39e31a01bc" rel="nofollow noopener noreferrer" target="_blank">Calendly's Dashboard</a> &gt;

☞ Before you start

OK Let's Do This!

-----------------------------------------------------------------------------------

SECURITY FAQ - Yes we want you to sleep well at night!

-----------------------------------------------------------------------------------

This means that none of VisualSitemap's employees have access to these values, and these are only decrypted and used when crawling.

-----------------------------------------------------------------------------------

Demo sitemap of Calendly's Dashboard >