Advanced Web Crawler

$750-1500 USD

Cancelled

Posted

over 11 years ago

$750-1500 USD

Paid on delivery

I am looking for an Advanced Web crawler built in Java. This crawler would at most be scanning 2 urls on a domain: A) Get a url from my database(ex: [login to view URL]) and scan the domain and the source of the domain for specific rules I will set such as: 1) Wordpress, Joomla, Drupal code signatures. - If yes, return Yes to the database. 2) Check to see if there is a link that is labeled "blog" or "contact us". IF either of these pages are found - Return the URL to the database for the correct one, IF yes for contact us THEN scan the page that was found for an email address. 4) Scan the site for a specific list of keywords, including 404 not found, and domain for sale. 3) Update the database with timestamp stating the site has been crawled. These are currently the 2 rules we need, but we will want the code to be written to easily add/modify these rules in the future. The most important process of this is Speed. Please place how many urls you think can be crawled in one day, as this will greatly influence who does the job.

Java

MySQL

PHP

Project ID: 4134206

About the project

10 proposals

Remote project

Active 11 yrs ago

Looking to make some money?

Email address

Benefits of bidding on Freelancer

Set your budget and timeframe

Get paid for your work

Outline your proposal

It's free to sign up and bid on jobs

10 freelancers are bidding on average $1,084 USD for this job

@creatorul

Professional solutions

$1,500 USD in 12 days

5.0

(81 reviews)

7.0

@javaphpsql

I am a Java developer with more than 10 years of experience in designing the Java multi-threaded applications and with more than 3 years experience in creating web harvesting applications. Please, check PM.

$799 USD in 12 days

4.9

(6 reviews)

4.7

@DenialWang

see pm please, thanks.

$750 USD in 4 days

4.9

(6 reviews)

4.3

@ciphe

Please see PMB.

$2,500 USD in 20 days

4.8

(5 reviews)

4.4

@saobiensoft

I'm interested in your job. Please check PMB for more details.

$750 USD in 10 days

4.5

(6 reviews)

3.5

@JavaGuruLV

Hi, I was finished similar project on this site. Target of this project was get urls from specified list, then crawl domain, parse results, remove html tags from page, count and store retrieved words to database. I used very fast crawler. It can simultaneously crawl many pages. My crawler has several important parameters, using it you can configure crawler inside your environment for best performance.

$750 USD in 5 days

4.8

(4 reviews)

2.8

@prologik

Hello. I'm a software engineer, and also a Senior Developer and Software Architect. Please see my private message for justification of my bid. Thanks in advance and good luck with your project. Best Regards! Esteban Esteban

$1,100 USD in 12 days

5.0

(3 reviews)

2.6

@paruikrishnendu

We are new on Freelancer, but since long time we are working internationally. We are confident that we can help you to fulfill your requirements. We'll give our best efforts to your project. We've a skilled team of developers and have worked on various requirements of complex projects. Request you to open private message board for genuine thread of communication and to get the ball rolling.

$750 USD in 15 days

0.0

(1 review)

0.0

@getveltrod

Hi, Veltrod Software services is a global software consulting company specialized in providing Mobile applications, Social media frameworks and eCommerce solutions. Leveraging best-in-class people, processes, and technologies, Veltrod provides high-quality software development and BPO services to independent software vendors and enterprises. We are specialized in developing Mobile applications, Search Engine optimizations, Social Network frameworks, Software Testing and ERP applications. We are pleased to take care of this project, if it is offered to us. We assure that we will work towards 100% client satisfaction. Thanks Vignesh

$1,000 USD in 28 days