Closed

Software Engineer I

Develop and validate solution architecture to support business requirements

Develop Spark applications to read transaction data and process business rules to report errors and transaction summary

Load and transform large sets of structured and semi structured.

Read data from Kafka and process data using Spark.

Adopt innovative architectural approaches to leverage in-house data integration capabilities.

Analyze existing processes and prepare functional and requirements documents.

Develop multiple spark streaming and core jobs with Kafka as a data pipe-line system.

Hive external tables on HBase using Insert Overwrite with S3 as data storage.

Troubleshoot developed Spark jobs.

Manage and review Yarn Application Logs, Spark Event Logs and Metrics sink CSV files.

Improve the performance and optimization of the existing jobs in Spark.

Develop solutions by utilizing commercial and open source software including Minifi and Nifi to interface big data and relational solutions.

Skills: Hadoop, Java, Software Architecture, Spark, Python

See more: how do i find a software engineer, i need a software engineer team, i want freelance software engineer to make a application, where can i find a software engineer, how do i find a qt software engineer, i need a software engineer, i am a engineer software and i am looking for projects in abroad, i am looking for a software engineer teachrr in nigeria, i am software engineer in india i want to come uk, i need a hl7 software engineer, i need a software engineer christchurch, i need a software engineer houston, i need a software engineer in toront, i want meet freelance software engineer in bangalore, what degree do i need to be a software engineer, where can i hire software engineer web developers and database designers, where do i get a software engineer, software engineer i job description, would i make a good software engineer, can i work as a software engineer part time

About the Employer:
( 0 reviews ) Los Angeles, United States

Project ID: #23958059

17 freelancers are bidding on average $45/hour for this job

helmot

Hello. I have a masters degree in AI and have worked with Hadoop, Spark, YARN, ... on a lot of projects. I can share some demos if you are interested. Thanks, Helmot

$35 USD / hour
(219 Reviews)
8.2
umg536

Hi there, Being an expert in Python and matlab programming I can do this project for you. please leave a message on my chat so we can discuss the budget and deadline of the project. Also, have a look at my profile for More

$50 USD / hour
(21 Reviews)
6.2
jinc772

Hi, there. I am a senior Python & Web developer and my main skills are Django and Flask. Fully experienced in each step of web development - design, development, deployment and maintenance, and mastered management of More

$40 USD / hour
(4 Reviews)
4.8
(2 Reviews)
4.5
asjkim

Highly effective Data Engineer and Data Scientist with over 25 years of experience. I am specilaizing Big Data and have extensive Python / Spark experience both in PySpark and Scala also in MLLib. I developed a number More

$55 USD / hour
(2 Reviews)
4.0
dineshrajputit

Hi, I have 9 years of experience and working on hadoop, spark, Kafka, nosql, java, BI tools(tableau, powerbi), cloud(Amazon, Google, Microsoft Azure)... Done end to end data warehouse management projects on aws cloud More

$50 USD / hour
(7 Reviews)
4.0
dinesh4nitb

Hello, hope you are well there, I've seen your requirements and I am interested in this. I'm ready to work from now on your app. I'm a full-stack developer. I've 7+ years of working experience in Mobile Apps Developm More

$35 USD / hour
(7 Reviews)
3.4
fabienbenoit1984

Hello! I'd like to optimize information aggregation. I'm familiar with python and theory of probabilities. I'll do the job blazingly fast. Please, give me a try!

$56 USD / hour
(2 Reviews)
2.5
stuartschade

Hi, I have readymade video streaming software. If you want to get the demo. I will give a demo. I have been working for the last 10+ years in development. Please have a look at my [login to view URL] Skills and Experience More

$43 USD / hour
(0 Reviews)
0.0
Beyaz1t

I have developed country wide log parsing application using Apache Spark on Hadoop. I have experienced memory issues deeply. Sqoop Metastore Oozie workflow pipelines were used. Machine Learning is another side of thmy More

$43 USD / hour
(0 Reviews)
0.0
nmogilip

I am currently working on each item which you described. I delivered end to end solution including kafka, spark, hive, nifi and all other required hadoop eco system components. Please let connect or have a call to di More

$44 USD / hour
(0 Reviews)
0.0
acka3a5

I saw your posting and thought we might be a good fit. We are a full stack development firm with a wide range of experience in development. Our team is highly capable and have worked with companies including Airbnb, PN More

$56 USD / hour
(0 Reviews)
0.0
LDaniel194

Hello, I read your project description and I think I'm fully qualified for this position. I'm a Software Developer with 5 years experience in technologies like: Java, Python, Spring, Hibernate, Redis, Gradle, Docker, More

$40 USD / hour
(0 Reviews)
0.0
andysteiner

Hi there, I have quite some experience in most of the big data tech out there (Hadoop, Spark, Kafka, Cassandra, ...), since I have been specializing on Data Management, Data Consolidation etc. for quite a long time, no More

$50 USD / hour
(0 Reviews)
0.0
awsdojo2020

Have the experience of over 5 years in developing Big data pipelines over multiple technologies, hold certification in Hortonworks and Cloudera platforms. Currently employed full time as analytics application engineer More

$40 USD / hour
(0 Reviews)
0.0
xcorp

Hi. I have a strong background in big data and Hadoop. I have worked as a Senior Data Platform Engineer and on technologies like spark, AWS, mapreduce, hbase, hive, Google bigquery, Kafka, Google cloud platform. I woul More

$55 USD / hour
(0 Reviews)
0.0
madirajujobs

Hi, Please refer my profile at linkedin [login to view URL] I have great experience in mentioned technologies. Awaiting for your response for further info Thanks, Vijay

$35 USD / hour
(0 Reviews)
0.0