When starting, we highly recommend that you start crawling just 3 pages as a sample test. To limit your crawl amount, just use the 'Max Pages' field:
1 - This protects you from wasting all your monthly credits.
2 - So you can test and see if the site is fully crawl-able.
If you don't see anything out of the ordinary ( like a popup or other ad blocking content). If you happen to see a pesky popup, just let us know via chat and we will look into removing it manually for your next crawl.
If everything looks great, fantastic! It's now safe to start your full crawl!
Step 1 : Enter a valid Url (http or https)
You can enter just the main URL or a URL with a /path/
If you use a URL + /path/, it will be used as the base starting point. All other external paths will be ignored.
Step 2 : Enter Max Pages (optional)
If you enter
100 it will crawl up to 100 pages.
If you leave it blank, it will crawl the FULL site. *Be careful with this one ;-)
Step 3 : Enter Page Depth (optional)
How many levels should the crawler go into.
example: Depth = `3`
for : Site.com/products/shoes
will crawl up to 'Shoes'
If you leave it blank, it will crawl all levels.
Step 4 : Advanced Settings (optional)
you can enter multiple directories to filter out
Filter Keywords/Parameters in URL
you can enter multiple keywords to filter out
Over-Crawl Protection (optional)
highly recommended. More info here >
*only available on Paid plans.
Step 5 : Click `Crawl It`
Statuses you will encounter...
Pending = your crawl is in the queue. *this should take a few seconds.
If you find this stuck for more than 5 min - please let us know via chat.
In Progress = The pages are being crawled and the sitemaps are being generate. Meanwhile you can click
View to see the map progress.
Crawl Failed = this could be for several reasons. ie, our crawler can't load your site, or some temporary internal failure. Either way, please let us know via chat and we will investigate it.
Click Refresh icon to load the new updates.
Next Up > How to use your new map