Web site scraping/mapping

We need the following: For each of the six sites below, we need to get a map of the product hierarchy and the number of product within the lowest category. The result will look like the spreadsheet attached. There will be 6 spreadsheets, one for each brand. Each with 3 tabs We also need a screen grab of just the main menu for when you hover over the top menu and the left hand nav when you click on each of the top menu’s. So each spreadsheet will have twice the number of images as there are top level menu categories. You can use FireShot for this This is going to require some intelligence in the way it is approached to work out how the site actually navigates to each category. This can also vary by area within the site e.g. electronics different to toys. However once you figure it out, it is easy. E.g. John lewis – for Toys, click, on by age and then click on toys by age, and then view all. Then you can copy them to notepad and then into excel Each of the web sites approaches the category presentation differently. The way to get the hierarchy is to click on each of elements of the main menu, then scrape the categories on the left hand nav, then click on each and scrape the left hand nav categories again until there are no more categories. REMEMBER – each site gets you to categories differently. When there are no more categories below, you need to get the number of products in the lowest level category. This can be got from either the number of in the lister page, or the number of results in brackets next to the field of price in the left hand navigation. You can do this manually or automatically. We estimate it is 3-4 man hours per site, though have not proven this. You have to make your own judgement. The maximum revision to price if it takes longer than expected is 15% Needs to be completed within 3 days. Sites are: [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL]