NodeJS script to crawl sitemap for redirect, page time, page size and write results to a file. - 30/04/2018 18:21 EDT

Hello, As I understood you need a bot which will render the pages of a website from sitemap and record the metrics to a file. I'm a software developer working on PHP and NodeJS environments for over 5 years, I expertise in web scraping and automation. There are 2 ways to go around this project: 1. Handle everything with HTTP calls. 2. Use headless browser like PhantomJS or Headless Chrome to actually render the page and process javascript. The 2nd option would involve more heavy processing and bandwith usage, but we could use service like [login to view URL] If the first option is acceptable for your use-case, then the project is fairly simple. I'd use Async and Request modules in NodeJS to build a CLI tool which will take standard sitemap XML and visit the pages to figure out what are all the metrics you are looking for. I also have a few optimizations ideas, mostly based on statistical analysis of the already gathered data. Let's have a quick chat and go over all the details. Best, Nick.

I am an expert Nodejs/Javascript developer with good experience. I have worked on several website crawlers and scrapers I am interested in working on your project and also available for ongoing development. Please contact me via chat to discuss the details.

Hello, I'm a full-stack developer with 4-year experience in the industry. I have very strong experience with node.js and crawler applications so I should finish this quickly. I will also need more info on how many sites you want to crawl. Kind Regards, Mihael Mamula