Let's say you wish crawl the whole site, except for some very specific pages, paginated pages, and/or blog category pages.

OR in reverse ;-)
perhaps you wish to only crawl specific pages that have only "dog" and/or "mobile" in the urls.

You can by using our Filter Keywords options > enter multiple keywords.

For example, let's say you have a ton of product pages that have pagination which use URL parameters : Somesite.com/bikes/?page=1 , page=2  , etc
or Somesite.com/blog?tag=consulting

Just enter these keywords separately : ?page= ?tag= 

Voila!

for urls that have urls with .php  parameters like:

to Exclude all those pages only use the params for the Keywords:

More full info and examples here >