Today, our crawler looks at the /directory/ structure as the basis for generating the sitemap. It uses a breadth-first method, which simply means that URLs will be linked from their 'highest' link, in terms of their crawl depth.

For example: ( the base URL you crawled )

will generate this tree structure:

However, if your URLs are setup like this : ( the base URL you crawled )

We will generate this tree structure:

So if you wish to modify the structure to show a different relationship, then simply drag n drop the pages around as needed like so:

Other fundamental crawling rules we implemented:

  1. We only show internal, indexable, and non-indexable, <ahref> HTML URLs.

  2. We only show URLs with incoming links (sorry - no orphan URLs at this time)


We know you want more here and in our upcoming Version 2,
we will offer an alternative parent < > child relational structure.
( regardless of the directory structure ) - stay tuned!

Did this answer your question?