Create webspiders / crawler to compare prices across multiple sites;
Should be implemented using configuration files*, separate modules for easy maintenance, change control (for example if the tag-tag has changed), map the locations of these pages with relevant information (eg price);
Stories
The user of the crawler will pass the name of a product, eg 32GB iPad 2, and the crawler will:
1. navigate through the search engines for each site
2. find the parameter passed
3. find a hash / unique identification code of the product
4. navigate to another site
5. if the products are identical to compare prices
# Knowledge required:
1. Experience in developing systems for processing and
data extraction using Object Oriented Perl, more specifically, developing webspiders / crawler.
2. Fluency in the use of parsers HTML / XML.
3. Domain in the use of Regular Expressions.
4. Knowledge of SQL.
# Specific knowledge:
1. LWP
2. DBI
# Sample of module file:
*[login to view URL]
#Attention:
--->> Available in an accessible via http
Hello,
10+ years exp full time freelancer here, can deliver a quality & professional work in the timeframe posted. Please contact me via PM for any question, Thanks.
Have 10 years experience with perl. Experience parsing HTML and XML and creating high-performance web crawlers/spiders with LWP. Experience with MySQL, PostgreSQL databases.
I have good experience using Perl APIs for website crawlers.
I have written to get the data from website and store at one location for further analysis for the company i work for.
Let me know i would be of any help.
Thanks
Pai.
I am a skilled,experienced IT professional with expertise in the field of developing web crawlers.
I found myself capable to fulfill the requirements specified in the project statement.
Have a look on attached script developed by me.
Hoping to hear from you soon.
Day 1: Information collection and requirements analysis.
Day 2: Environment analysis (target websites data structure).
Day 3: Application design.
Day 4: Engine construction.
Day 5: User interface build.
Day 6: Deploy and user acceptance.