Skip to main content

Why am I getting a "Crawl Failed - URL could not be processed" or 403/503, 5xx error?

Crawling Issies

Written by Jeff

Unfortunately, these errors mean that
​
​
1 - you are using an ad blocker like UBlock Origin ( just disable it during a crawl )
​
or
​
2 - the website's server has denied access to our crawlers for security purposes.

​NOTE: Some sites deploy highly protected security measures to prevent all automated crawlers.
​
In cases where the server's security is hardened, we fallback to a global proxy service to attempt to by-pass such measures.
​
There will be edge-cases however where these proxies will still not work because of detection, and in those cases you will need to contact the site's IT dept to allow our crawlers in. ( see below* )


*If you see this issue, AND you own the site or have dev access, then you can try to whitelist our crawler by selecting :
​

1. Click Advanced Settings > Include Custom ##### HTTP Header
2. this will show your UNIQE header ID.
3. copy/paste that into your server configs.

​
This will generate a unique header ID *value assigned just to your account/crawlers.
​


NOTE: Every server handles whitelisting configs differently.
Please contact your IT dept. to set up whitelisting headers!

Did this answer your question?