​
​Prior to this date, all maps were generated using the /directory/ structure as the basis for generating the sitemap. It uses a breadth-first method, which simply means that URLs will be linked from their 'highest' link, in terms of their crawl depth.

For example:
​
​Site.com ( the base URL you crawled )
- Site.com/product/
--- Site.com/product/shoes.html
​

will generate this tree structure:

However, if your URLs are setup like this :
​
​Site.com ( the base URL you crawled )
- Site.com/product/
--- Site.com/shoes.html
​
We will generate this tree structure:
​

So if you wish to modify the structure to show a different relationship, then simply drag n drop the pages around as needed like so:
​
​

Other fundamental crawling rules we implemented: