Custom Headers: This allows your server to identify our crawlers, in case you need to white-list them to allow them through the security walls.
​

Crawl an XML URL:
The URL of an XML sitemap. We just crawl URLs found in the XML sitemap, links from the pages will not be followed.
​

Crawl a CSV file:
​Format = the first column must contain a URL.
We just crawl URLs found in the file, links from the pages will not be followed. Remember to use a comma , as a delimiter and " as an escape character.
The CSV may use headers as the first row.

Include & Exclude Directories
Directories are now NOT-STRICT ( aka fuzzy ) in their matches!
​
ie: /dog/ will match either of these URLs : 

ie: /pets/dog/  will match either of these URLs:

​Include & Exclude Keywords
​
Keywords are NOT STRICT, but "fuzzy" in their matches, and are more flexible than directories as they can match any part of the URL's path.
​
To include all urls containing 'dog' ie > https://site.com/dog-products.html  > Enter dog . This will also include urls such as https://site.com/categories/dogs/food
​

for URLs that have parameters like:

to Exclude all those pages only use the params for the Keywords:

All of the filters can be used in combination with each other:
ie: /directory/  + keyword  =  will return only results that strictly match both.
​
​Example:  Directories + Keywords together

will not crawl:  site.com/products/dog/monkey-pants.html ✘
will not crawl:  site.com/dog/monkey-pants.html ✘
will not crawl:  site.com/dog/monkeys.html ✘
​
it will crawl:  site.com/dog/foo.html  ✅
_________________

Multiple values within each filter,  use an “OR” logic. 

That is if you enter 3 keywords (dog , cat , monkey ), then the URL will pass the check as long as it has dog  OR cat  OR monkey  in its path.

Example:  Multiple Keywords
​
​

will not crawl:  site.com/dog/foo.html  ✘
will not crawl:  site.com/products/dog/... ✘
will not crawl:  site.com/tags/cat/...
will not crawl:  site.com/category/monkey/...  ✘
​
it will crawl:  site.com/cats/foo.html  ✅

Example : Multiple /Directories/ + Multiple Keywords
​
​

will not crawl  :  site.com/dog/monkey-pants.html  ✘
will not crawl  :  site.com/cat/monkeys.html  ✘
will not crawl  :  site.com/dog/birds/..  ✘
will not crawl  :  site.com/monkey/..  ✘
will not crawl  :  site.com/bird-cages/..  ✘

it will crawl :  site.com/dog-cages/  ✅
it will crawl :  site.com/categories/cat-litters/  ✅
​

​Example: Exclude + Include Combos:

will not crawl:  site.com/cake/monkeys.html  ✘
will not crawl:   site.com/cake/monkey-pants/foo.html ✘
will not crawl:  site.com/monkey/cakes.html ✘
it will crawl:  site.com/cake/foo.html ✅
​
*These can also cancel themselves if you use:
​ Exclude Directory ="/cat/" + Include keyword "cat"
​