We are hiring a Python developer to create a standalone crawler program for a specific website using Scrapy. Target website content: aggregated announcements for events or ticket gateway. Applicants will be asked to complete version 1.0 of this program, and deliver all source code to the client. Estimated budget is $200 with some room for negotiation.
The program requirements:
- Built using Scrapy, callable from Scrapy CLI.
- When executed, program must scrape ALL desired data from target website. Crawler may need to follow links within the target website.
- Crawled data items should be cleaned using a simple format (specified by client).
- Program MUST include a toggle-able test mode, which should check whether markup changes in the target website break the crawler. Please include a 1 to 2 line summary of how you would accomplish this with your proposal, in the format: "MY TEST MODE: (write your explanation)".
- [Extended] Exclusive rights to numerous followup project opportunities will be considered if the project is completed satisfactorily.
The applicant requirements:
- Strong communication skills, proficient with English, and comfortable speaking on [Offsite communication is prohibited in the site].
- Required programming languages: Python (HTML, CSS is an advantage).
- Strong preference given to those demonstrate previous experience with Scrapy.
- More consideration will be given to applicants who either 1) provide links to their own open source projects, 2) provide links to contributions to Python community via tech blogs, Q&A forums, etc 3) provide references for past clients.
Hi sir,
I am scraping expert, I have did too many similar projects, please check my feedback then you will know.
Can you tell me more details? then I will provide demo data for you.
Thanks,
Kimi
I have an experience in web scrapping. About python: you are able to see such program in my portfolio, with mechanize module. Actually, I have more web crawlers I can show you, if you want to see it.