However, you need to be very careful there because our crawler will CLICK ON EVERY LINK IT FINDS!
So if you set it up to crawl as an ADMIN user, it could trigger links on the page that are available to admins only.
Therefore you can either :
a) use a different limited-restriction user ( non-admin )
b) filter out specific links you want our crawler to skip.
**WE ARE NOT RESPONSIBLE FOR ANY CHANGES TO YOUR SITE.**
*PLEASE CRAWL RESPONSIBLY AND ALWAYS BACKUP YOUR SITE JUST IN CASE*