Let's say you wish crawl the whole site, except for some very specific pages, paginated pages, and/or blog category pages.
OR in reverse ;-)
perhaps you wish to only crawl specific pages that have only "dog" and/or "mobile" in the urls.
You can by using our Filter Keywords options > enter multiple keywords.
For example, let's say you have a ton of product pages that have pagination which use URL parameters :
page=2 , etc
Just enter these keywords separately :
for urls that have urls with
.php parameters like:
to Exclude all those pages only use the params for the Keywords:
do not use