Skip to main content
All CollectionsFAQs
Why am I getting a "Crawl Failed - URL could not be processed" or 403/503, 5xx error?
Why am I getting a "Crawl Failed - URL could not be processed" or 403/503, 5xx error?

Crawling

Artur M avatar
Written by Artur M
Updated over 3 years ago

Unfortunately, these errors mean that the website's server has denied access to our crawlers for security purposes.

NOTE: Some sites deploy highly protected security measures to prevent all automated crawlers. Except for Google crawlers.. of course!

If you see this issue, AND you own the site ( or have access to its servers )
then you can try to whitelist our crawler by selecting :

1. Click Advanced Settings > Include Custom ##### HTTP Header
2. this will show your UNIQE header ID.
3. copy/paste that into your server configs.


This will generate a unique header ID *value assigned just to your account/crawlers.


NOTE: Every server handles whitelisting configs differently.
Please contact your IT dept. to set up whitelisting headers!

Did this answer your question?