FILTERS
The Basics:
Include & Exclude Directories
D̶i̶r̶e̶c̶t̶o̶r̶i̶e̶s̶ ̶a̶r̶e̶ ̶V̶E̶R̶Y̶ ̶S̶T̶R̶I̶C̶T̶ ̶i̶n̶ ̶t̶h̶e̶i̶r̶ ̶m̶a̶t̶c̶h̶e̶s̶.̶ ̶S̶o̶ ̶y̶o̶u̶ ̶m̶u̶s̶t̶ ̶b̶e̶ ̶v̶e̶r̶y̶ ̶e̶x̶p̶l̶i̶c̶i̶t̶ ̶w̶i̶t̶h̶ ̶t̶h̶e̶ ̶F̶U̶L̶L̶ ̶d̶i̶r̶e̶c̶t̶o̶r̶y̶ ̶p̶a̶t̶h̶.̶ ̶
(UPDATED) Directories are now NOT-STRICT ( aka fuzzy ) in their matches!
ie: /dog/
will match either of these urls :
site.com/dog/... OR
site.com/products/dog/...
ie: /pets/dog/
will match either of these urls:
site.com/pets/dog/... OR
site.com/products/pets/dog/...
-------------------------------
Include & Exclude Keywords
Keywords are NOT STRICT, but "fuzzy" in their matches, and are more flexible than directories as they can match any part of the URL's path.
To include all urls containing 'dog' ie > https://site.com/dog-products.html
> Enter dog
. This will also include urls such as https://site.com/categories/dogs/food
IMPORTANT!
for urls that have parameters like:
site.com/index.php?page=1
to Exclude all those pages only use the params for the Keywords:
page=
do not use
index.php?page=
-----------------------------------
Advanced Combos:
All of the filters can be used in combination with each other:
ie: /directory/
+ keyword
= will return only results which strictly match both.
Example : Directories + Keywords together
will not crawl: site.com/products/dog/monkey-pants.html
✘
will not crawl: site.com/dog/monkey-pants.html
✘
will not crawl: site.com/dog/monkeys.html
✘
it will crawl: site.com/dog/foo.html
✅
_________________
Multiple values within each filter, use an “OR” logic.
That is, if you enter 3 keywords (dog
, cat
, monkey
), then the URL will pass the check as long as it has dog
OR cat
OR monkey
in its path.
Example : Multiple Keywords
will not crawl: site.com/dog/foo.html
✘
will not crawl: site.com/products/dog/...
✘
will not crawl: site.com/tags/cat/...
will not crawl: site.com/category/monkey/...
✘
it will crawl: site.com/cats/foo.html
✅
_________________
Example : Multiple /Directories/ + Multiple Keywords
will not crawl : site.com/dog/monkey-pants.html
✘
will not crawl : site.com/cat/monkeys.html
✘
will not crawl : site.com/dog/birds/..
✘
will not crawl : site.com/monkey/..
✘
will not crawl : site.com/bird-cages/..
✘
it will crawl : site.com/dog-cages/
✅
it will crawl : site.com/categories/cat-litters/
✅
_________________
Example: Exclude + Include Combos:
will not crawl: site.com/cake/monkeys.html
✘
will not crawl: site.com/cake/monkey-pants/foo.html
✘
will not crawl: site.com/monkey/cakes.html
✘
it will crawl: site.com/cake/foo.html
✅
*These can also cancel themselves if you use:
Exclude Directory ="/cat/" + Include keyword "cat"