All Collections
FAQs
Can I restrict the crawl to just a part of a website?
Can I restrict the crawl to just a part of a website?

Crawling

Artur M avatar
Written by Artur M
Updated over a week ago

Absolutely. This is where the power of Advanced Filters shine!

For example let's say you only want to crawl every page under :  https://SomeUrl.com/products/

just enter  /products/ in the Include Directories field:


Result : this will crawl all these directories:

https://SomeUrl.com/products/mens/shoes
https://SomeUrl.com/products/kids/toys

but it will not crawl:

https://SomeUrl.com/blog/
https://SomeUrl.com/contact/
Did this answer your question?