Skip to main content
All CollectionsTutorials
How to crawl a list of specific links from one domain (CSV)
How to crawl a list of specific links from one domain (CSV)
Jeff avatar
Written by Jeff
Updated over a year ago

We allow for CSV importing of URLs for (1 domain) via the Advanced Settings:
​

REQUIREMENTS:
​
1 - the first column must contain a URL.

2 - all URLs must have either HTTP:// or HTTPS://
3 - all URLs must be from the same domain.
​

IMPORTANT NOTES!

  • We just crawl URLs found in the file, links from the pages will not be followed.

  • Remember to use a comma , as a delimiter and " as an escape character.

  • The CSV may use headers as the first row.

  • We automatically generate a root text-node using the root URL.

Did this answer your question?