FILTERS
The Basics:
Include & Exclude Directories
Directories are STRICT in their matches.
ie: /dog/ will only match this url :
site.com/dog/foo.html it wont match:
site.com/dogs/foo.html -------------------------------
Include & Exclude KeywordsKeywords are NOT STRICT, but "fuzzy" in their matches.
To include all urls containing 'dog' ie > https://site.com/dog-products.html > Enter dog .
-------------------------------
Advanced Combos:
All of the filters can be used in combination with each other:
ie: /directory/ + keyword = will return only results which match both.
Example : using both together
will not crawl this url : site.com/dog/monkey-pants.html ✘
but will crawl this url : site.com/dog/foo.html ✅
_________________
Multiple values within each filter, use an “OR” logic.
That is, if you enter 3 keywords (“dog”, “cat”, “monkey”), then the URL will pass the check as long as it has “dog” OR “cat” OR “monkey” in its path.
Example : using multiple keywords
will not crawl this url : site.com/dog/foo.html ✘
will not crawl this url : site.com/cat/stuff.html ✘
but will crawl this url : site.com/cats/foo.html ✅
_________________
Example : using /directory/ + multiple keywords
will not crawl this url : site.com/dog/cat/monkey-pants.html ✘
but will crawl this url : site.com/dog/monkey-pants.html ✅
_________________
Exclude + Include Combos:
will not crawl this url : site.com/cake/monkeys.html ✘
will not crawl this url : site.com/cake/monkey-pants/foo.html ✘
but will crawl this url : site.com/cake/foo.html ✅
but will crawl this url : site.com/monkey/cakes.html ✅