Skip to main content

Why am I getting a "Crawl Failed - URL could not be processed" or 403/503, 5xx error?

Crawling Issies

Jeff avatar
Written by Jeff
Updated over 2 weeks ago

Unfortunately, these errors mean that


1 - you are using an ad blocker like UBlock Origin ( just disable it during a crawl )
or
2 - the website's server has denied access to our crawlers for security purposes.

NOTE: Some sites deploy highly protected security measures to prevent all automated crawlers. Except for Google crawlers.. of course!

If you see this issue, AND you own the site ( or have access to its servers )
then you can try to whitelist our crawler by selecting :

1. Click Advanced Settings > Include Custom ##### HTTP Header
2. this will show your UNIQE header ID.
3. copy/paste that into your server configs.


This will generate a unique header ID *value assigned just to your account/crawlers.


NOTE: Every server handles whitelisting configs differently.
Please contact your IT dept. to set up whitelisting headers!

Did this answer your question?