All Collections
FAQs
Can it crawl a password-protected WordPress site to generate the whole website map?
Can it crawl a password-protected WordPress site to generate the whole website map?

wordpress crawling

Jeff avatar
Written by Jeff
Updated over a week ago

yes!

However, you need to be very careful there because our crawler will CLICK ON EVERY LINK IT FINDS!

So if you set it up to crawl as an ADMIN user, it could trigger links on the page that are available to admins only.

Therefore you can either :


a) use a different limited-restriction user ( non-admin )

b) filter out specific links you want our crawler to skip.

**WE ARE NOT RESPONSIBLE FOR ANY CHANGES TO YOUR SITE.**
*PLEASE CRAWL RESPONSIBLY AND ALWAYS BACKUP YOUR SITE JUST IN CASE*


Did this answer your question?