Skip to main content

Why am I getting a "Crawl Failed - URL could not be processed" or 403/503, 5xx error?

Crawling Issies

Jeff avatar
Written by Jeff
Updated over 2 weeks ago

Unfortunately, these errors mean that


1 - you are using an ad blocker like UBlock Origin ( just disable it during a crawl )

or

2 - the website's server has denied access to our crawlers for security purposes.

NOTE: Some sites deploy highly protected security measures to prevent all automated crawlers.

In cases where the server's security is hardened, we fallback to a global proxy service to attempt to by-pass such measures.

There will be edge-cases however where these proxies will still not work because of detection, and in those cases you will need to contact the site's IT dept to allow our crawlers in. ( see below* )


*If you see this issue, AND you own the site or have dev access, then you can try to whitelist our crawler by selecting :

1. Click Advanced Settings > Include Custom ##### HTTP Header
2. this will show your UNIQE header ID.
3. copy/paste that into your server configs.


This will generate a unique header ID *value assigned just to your account/crawlers.


NOTE: Every server handles whitelisting configs differently.
Please contact your IT dept. to set up whitelisting headers!

Did this answer your question?