Develop and validate solution architecture to support business requirements
Develop Spark applications to read transaction data and process business rules to report errors and transaction summary
Load and transform large sets of structured and semi structured.
Read data from Kafka and process data using Spark.
Adopt innovative architectural approaches to leverage in-house data integration capabilities.
Analyze existing processes and prepare functional and requirements documents.
Develop multiple spark streaming and core jobs with Kafka as a data pipe-line system.
Hive external tables on HBase using Insert Overwrite with S3 as data storage.
Troubleshoot developed Spark jobs.
Manage and review Yarn Application Logs, Spark Event Logs and Metrics sink CSV files.
Improve the performance and optimization of the existing jobs in Spark.
Develop solutions by utilizing commercial and open source software including Minifi and Nifi to interface big data and relational solutions.
17 freelancers are bidding on average $45/hour for this job
Hello. I have a masters degree in AI and have worked with Hadoop, Spark, YARN, ... on a lot of projects. I can share some demos if you are interested. Thanks, Helmot
Hello! I'd like to optimize information aggregation. I'm familiar with python and theory of probabilities. I'll do the job blazingly fast. Please, give me a try!
Hi, Please refer my profile at linkedin [login to view URL] I have great experience in mentioned technologies. Awaiting for your response for further info Thanks, Vijay