Easy. Simply DO NOT set the Max Page
limits.
β
Because, If you do set this limit, the crawler will cold-stop when it reaches it, and you will not be allowed to restart the crawl from where it stopped.
β
By not specifying this setting, when the crawler reaches the plan's Max Sitemap Size
limit, it will
auto-pause
the crawl and give you the option to upgrade to a larger plan that offers larger-sized maps.
β
After the upgrade, simply return to your paused sitemap and click to
resume
the crawl. At this juncture, you will also have the ability to change some crawl settings like Max Depth
and add Max Page
limits.
Crawling Tips
Written by Jeff
Updated over 3 years ago