Skip to main content

Do you crawl responsibly?

Crawling

Written by Jeff

Yes.
​
Our goal is to provide you with accurate and full-scoped data. If we go too fast, some servers will react by shutting down access. Who wants that right?  
​
Based on our research and constant benchmark testing, we've determined the safest acceptable rate limits and implemented dynamically throttled limits to avoid server overload. 

Did this answer your question?