Closed

Need an expert developer for firmware file and metadata scrapers

This project will develop set of crawlers based on Scrapy framework that can download and synchronize all of products' firmware (including all versions) from web pages of a given list of predefined vendors and store the firmware information (meta data) in PostgreSQL DB. Final number of crawlers would be ~100 and project milestones are defined per vendor and each milestone is max 65€ which is paid after we verify the completeness of each crawler and see no errors.

The mandatory metadata fields include (Manufacturer, Model, Version, Type, Name, Release Date (if available), Download link, (calculated Sha2 hash of the file)i.e. ( Cisco, Video Surveillance 6030 IP Camera, 2.7.0, IP Camera, [login to view URL], 21/08/2015, "link", “Sha2” ). There is a boolean field which indicates if the device is discontinued or not depending on the availability of such information on the website of the vendor. The firmware files itself will be stored in the file system and will be referenced in PostgreSQL.

The developer is required to follow DB schema and code templates provided by us which has been partially developed. It's also the responsibility of the developer to test crawler and ensure completeness of the solution in terms of full coverage of the firmware files and product pages. There are no GUI components on the server that runs crawlers. Therefore, headless browsing mode should be used.

Project Scope

1. Crawlers will be written per vendor. This is required because each vendor website will have its own implementation of the firmware download page.

2. The user should be able to pause and resume crawling jobs.

3. Crawlers should detect previously downloaded files and only download updated and new content and firmware files. At first execution of each crawler, it will download all the available firmware files but the subsequent crawler runs will only download new firmware files which are added since the last crawling.

4. The developer is required to manually analyze each provided vendor site before writing a crawler to identify the following required information:

a. URLs for the firmware download page including all of the firmware versions for each product

b. URLs/files for each product that include the following information, required to be scraped: "Manufacturer", "Model", "Version", "Type", "Release Date", "if the product is discontinued"

c. Credential Requirements (Simple Signups, Specific Signups, No Signups)

d. Any Captcha on the page

e. Any honeypot traps

5. If a vendor site requires credential for firmware download, the developer is required to sign up an account using an email address dedicated for this project

6. Script will try to imitate human like behaviour (to a limit) while scraping the web page as well as using Tor if required

Important Notes

The developer MUST test the completeness of each crawler before delivering to us and present test completion evidence in the form of a populated PostgreSQL database of that vendor.

*An NDA and a contract must be signed before the beginning of the project. A copy of the developer's identification document is required to verify the identity.

*Please apply just when you fully read and understand the project and agree with the conditions.

Skills: Scrapy, Web Scraping, SQLite, Python, Linux

See more: need php developer team, need web developer, need php developer delhi, need game developer, need upload large audio file, company asia need php developer offsite, need someone excel attached file, need lisp developer, need dotnetnuke developer, karachi need freelance developer, asp api expert developer, need flash developer, need full time sharepoint expert developer, urgent need expert ios developer, urgent: need expert ios developer, need to hire an expert in csv file, Need a developer expert

About the Employer:
( 4 reviews ) Brussels, Belgium

Project ID: #27831217

28 freelancers are bidding on average €7085 for this job

chirgeo

Hi, my name is George. I read the project description and we are happy to help you. At [login to view URL] we are developing a new tool that allows us to automate any task in the browser easier. This tool replaces More

€5000 EUR in 7 days
(134 Reviews)
7.6
evgeny1002

Hi I just read your job posting. It sounds like you need an expert in scraping with Python. As you can see in my work history I have been working with Python and scraping is one of my best skills. [login to view URL] More

€7500 EUR in 7 days
(26 Reviews)
6.4
technosystem

Would you like to show an interest for something ready for scraping the metadata ? For last 18 years i have been developing Websites, Mobile apps, Blockchain, Decentralized Exchanges, Crypto Coins based on ERC20, Bitco More

€14500 EUR in 60 days
(6 Reviews)
5.7
falconcapitalnyc

Hello there. This is Joshua and I have just read your job posting and read your requirements with care, I know you are looking for an experienced Scraping developer. I have been working as a software developer for over More

€7500 EUR in 7 days
(5 Reviews)
4.3
alexeygamil

Hi Nice to meet you. I hope you are safe from COVID I am Scrapy, Web scraping , Python, Linuxexpert and I only apply to jobs when I know I can do it. I am confident that I will deliver you the best solution possible a More

€5000 EUR in 7 days
(4 Reviews)
4.3
yanakhokhlova199

Hello there. I am very interested in your project. *** As python and webscraping expert ***. I can handle this and am confident of winning. So I have rich experience in this field and if you give a chance, perfect r More

€5000 EUR in 20 days
(3 Reviews)
4.3
visionvivante

Hi there, After reading over your application this looks like a perfect fit for my skill sets. My name is Naren, [login to view URL] Stack Developer with 9+ years of experience of Website Development and I think I am the guys you More

€8000 EUR in 40 days
(1 Review)
4.9
PKonstiantyn

Hi, I have read your post and understood your requirement. Looking for the freelancer to work on your next project? Or just need some issues/bugs/fixes ASAP? I have 11+ years of software experience in fortune 500 comp More

€7500 EUR in 7 days
(11 Reviews)
4.3
agushieduard450

Hi, manager! Nice to meet you. I hope you are well and staying safe without COVID-19 effect As checking your job description, I have noticed I'm a good fit for your project from my skills and experience. 1. My scrapin More

€5000 EUR in 30 days
(8 Reviews)
3.5
(3 Reviews)
3.1
EkoLike

Hello. how are you ? I am scrapping expert in python , node etc i have worked in many project and site for scraping data. and i saved data to csv or interface on other site etc anyway, i can provide several skill for y More

€6500 EUR in 7 days
(2 Reviews)
3.0
VaLer0chka

Hi, manager! I hope you are safe without the effect of covid-19. After checking your job description, I knew this job is a good fit for me because I have enough experience in the scraping field. I am as sharp as a tack More

€5000 EUR in 7 days
(2 Reviews)
2.2
Puneeths1996

Hi, My portfolio site, [login to view URL] My Github site: [login to view URL] Please leave a message, would be interested in hearing about the requirement and the objective th More

€7500 EUR in 7 days
(2 Reviews)
2.1
sabramihor

Hello. I am a node.js/python expert with 10+ year experience. I had read job description and fully understood about requiremnt. I have great experience of scraping several museums to build artist/artwork database whic More

€8500 EUR in 40 days
(2 Reviews)
2.2
marinaparshina05

Hi. With over 5 years of experience in Python and Proxy. I’ve gone through your complete project description. My main skills are as follows: Python, Machine Learning, Web Scraping, Selenium, BeautifulSoup... Thank More

€5000 EUR in 30 days
(1 Review)
2.4
(1 Review)
1.1
stepinnsolution

Yes i read all 6 points clearly and understand. We are ready to sign N|DA in this project we have no issue for NDA. We are team and working together sitting in the house. I am a full stack developer and around 12+ yea More

€7500 EUR in 7 days
(1 Review)
0.6
anpilogovfree

Hello! Your project is very interesting. As a professional scraper, I can complete your project successfully. I can do C++/C#/Javascript/Python/PHP scraping and well use of Fiddler or Charles Web debugging tools. Here More

€6000 EUR in 30 days
(0 Reviews)
0.0
iPHTechnologies

Hi there, Greetings!! I am an expert web designer and developer with 9+ years of experience who has a deep understanding of User experience, User Interface, Wireframes, Mockups, User Flows, Process Flows, and Site Maps More

€7500 EUR in 7 days
(0 Reviews)
0.0
simrantel

Hello, I have checked The requirement and understand that as well. I have created SIMILAR in past. Do you want to see the DEMO ??? Can will send you ?? Thanks.

€7500 EUR in 7 days
(0 Reviews)
0.0