Advanced Settings & Filters

FILTERS

The Basics:

Include & Exclude Directories
D̶i̶r̶e̶c̶t̶o̶r̶i̶e̶s̶ ̶a̶r̶e̶ ̶V̶E̶R̶Y̶ ̶S̶T̶R̶I̶C̶T̶ ̶i̶n̶ ̶t̶h̶e̶i̶r̶ ̶m̶a̶t̶c̶h̶e̶s̶.̶ ̶S̶o̶ ̶y̶o̶u̶ ̶m̶u̶s̶t̶ ̶b̶e̶ ̶v̶e̶r̶y̶ ̶e̶x̶p̶l̶i̶c̶i̶t̶ ̶w̶i̶t̶h̶ ̶t̶h̶e̶ ̶F̶U̶L̶L̶ ̶d̶i̶r̶e̶c̶t̶o̶r̶y̶ ̶p̶a̶t̶h̶.̶ ̶ 

(UPDATED) Directories are now NOT-STRICT ( aka fuzzy ) in their matches!

ie: /dog/ will match either of these urls : 

site.com/dog/... OR
site.com/products/dog/...

ie: /pets/dog/  will match either of these urls:

site.com/pets/dog/... OR
site.com/products/pets/dog/... 

-------------------------------

Include & Exclude Keywords

Keywords are NOT STRICT, but "fuzzy" in their matches, and are more flexible than directories as they can match any part of the URL's path.

To include all urls containing 'dog' ie > https://site.com/dog-products.html  > Enter dog . This will also include urls such as https://site.com/categories/dogs/food

IMPORTANT!

for urls that have parameters like:

site.com/index.php?page=1

to Exclude all those pages only use the params for the Keywords:

page=

do not use 

index.php?page=

-----------------------------------

Advanced Combos:

All of the filters can be used in combination with each other:
ie: /directory/  + keyword  =  will return only results which strictly match both.

Example :  Directories + Keywords together

will not crawl:  site.com/products/dog/monkey-pants.html ✘
will not crawl:  site.com/dog/monkey-pants.html ✘
will not crawl:  site.com/dog/monkeys.html ✘

it will crawl:  site.com/dog/foo.html  ✅
_________________

Multiple values within each filter,  use an “OR” logic. 

That is, if you enter 3 keywords (dog , cat , monkey ), then the URL will pass the check as long as it has dog  OR cat  OR monkey  in its path.

Example :  Multiple Keywords 

will not crawl:  site.com/dog/foo.html  ✘
will not crawl:  site.com/products/dog/... ✘
will not crawl:  site.com/tags/cat/...
will not crawl:  site.com/category/monkey/...  ✘

it will crawl:  site.com/cats/foo.html  ✅

_________________

Example : Multiple /Directories/ + Multiple Keywords

will not crawl  :  site.com/dog/monkey-pants.html  ✘
will not crawl  :  site.com/cat/monkeys.html  ✘
will not crawl  :  site.com/dog/birds/..  ✘
will not crawl  :  site.com/monkey/..  ✘
will not crawl  :  site.com/bird-cages/..  ✘

it will crawl :  site.com/dog-cages/  ✅
it will crawl :  site.com/categories/cat-litters/  ✅
_________________

Example: Exclude + Include Combos:

will not crawl:  site.com/cake/monkeys.html  ✘
will not crawl:   site.com/cake/monkey-pants/foo.html ✘
will not crawl:  site.com/monkey/cakes.html ✘
it will crawl:  site.com/cake/foo.html ✅


*These can also cancel themselves if you use:
 Exclude Directory ="/cat/" + Include keyword "cat"