Build a Large Scale Web Crawler System

Large Scale Crawler Looking for a developer (or company) to build a robust web crawler system. There are approximately 20,000+ websites that we want to crawl and extract data from. We want to be able to extract these data within 3-6 months. 1. Design the architecture of the crawler or use existing open source crawler as a template. Because we’re dealing with large volume of data the architecture needs to be: • Robust and scalable • Efficient and Fast • Support proxies (to bypass anti-scraping systems) 2. Create Admin dashboard where Admin can: a. Add, Edit, View, Delete, Stop, Search crawler b. Input the URL to crawl c. Specify the data that needs to be extracted (ie. Title, Title URL, etc.) d. View, Edit, and Delete extracted data e. Option to download the data in JSON, XML, CSV f. API of the data (either via Authorization Tokens or other means) for upload and integration h. Users Management with ACL (Access Control List), Create, Edit, View, Delete users 3. Data normalization and clean up. The data coming in are unformatted and unstructured; an example would be the location or city, some site list location or city as Houston, TX, while other list as Houston, Texas or USA-TX-Houston. Therefore, the location or city data needs to be formatted, we use Google Location. 4. Because the data changes daily on these 20,000+ websites, there needs to be notifications put in place to notify the system of the changes (ie. what’s been added and what’s been removed) and update the data automatically. 5. Once the data is verified and cleansed, it will be available for search either via Solr or ElasticSearch or any other recommendation. Some of the technical challenges that need to be addressed from the beginning: • Make sure that the crawler compresses the data before fetching it otherwise it will uses a huge amount of storage • No need to re-crawl a website every 1-2 days, because it would be a waste of resources, however we do want the data every 1-2 days • Ways to prevent crawler from DoS (Denial of Service) • Ways to prevent the system from crashing and overloading because there are so many crawlers running • System should be scalable to handle crawling 100,000 – 200,000 websites • Queuing: does the crawler start right away or does it run in batches at a certain time? How does it scale when we start adding more sites to crawl? Example Day 1: Admin adds 100 sites to crawl Day 2: Admin adds 200 sites to crawl Day 3: Admin adds 500 sites to crawl Day 4: etc.