All Collections
Tutorials
How to use Keyword Filters for URLs
How to use Keyword Filters for URLs

Crawling

Artur M avatar
Written by Artur M
Updated over a week ago

Let's say you want to crawl and screenshot some very specific, paginated , and/or blog category pages.

OR in reverse ;-)
perhaps you wish to only crawl specific pages that have only dog and/or mobile in the URL.

You can by using our Advanced Setting > Include/Exclude Keywords option.

For example, let's say you have a ton of product pages that have pagination which use URL parameters : Somesite.com/bikes/?page=1 , page=2  , etc
or Somesite.com/blog?tag=consulting

Just enter these keywords separately : ?page= ?tag= 

IMPORTANT!

Note: Our crawler will filter in/out urls only as it finds them procedurally from the base URL ( domain ) you initially enter, it does not crawl all URLs and then post-crawl, filter out the data.

other use-cases:

- for URLs that have urls with .php  parameters like:

site.com/index.php?page=1

to Exclude all those pages only use the params for the Keywords:

page=

do not use 

index.php?page=

Did this answer your question?