All Collections
FAQs
Can it crawl a password-protected WordPress site to generate the whole website map?
Can it crawl a password-protected WordPress site to generate the whole website map?

wordpress crawling

Jeff avatar
Written by Jeff
Updated over a week ago

yes!
​
However, you need to be very careful there because our crawler will CLICK ON EVERY LINK IT FINDS!
​
So if you set it up to crawl as an ADMIN user, it could trigger links on the page that are available to admins only.
​
Therefore you can either :


a) use a different limited-restriction user ( non-admin )

b) filter out specific links you want our crawler to skip.

**WE ARE NOT RESPONSIBLE FOR ANY CHANGES TO YOUR SITE.**
*PLEASE CRAWL RESPONSIBLY AND ALWAYS BACKUP YOUR SITE JUST IN CASE*


​

Did this answer your question?