Hadoop Jobs

I need an observium specialist to do custom dashboard for: 1. Port Status Monitoring for servers 2. Monitor database health (MariaDB, MongoDB, Hadoop) 3. Server up and down status 4. Memory, hard disk, and resource utilisation

We have opportunities for qualified Data Engineering Specialist to work for a leading telecommunications company in Sydney, Australia. We're seeking an experienced data engineering specialist who has skills and experience in implementing large scale big data platforms. Our big data / data analytics technologies: Postgresql, Apache Spark, Apache Kafka, Apache HDFS, Apache Nifi, Apache Hive, Apache Flink, Apache Druid, Scala and Python Programming language, Kubernetes(K8s)/Rancher/Docker, ELK, Internet of Things, Data Science, AI and ML Platforms

Content based recommendation system using MapReduce, i.e. given a job description you should be able to suggest a set of applicable courses

Create a POC for data lakehouse solution using: 1. hdfs for storage, 2. deltalake, dremio to query delta lake, dremio-superset integration It should have folln features: 1. ACID support 2. data governance feature 3. Data/metadata discovery using Amundsen 4. data catalog with Apache atlas .. here Amundsen to be integrated with Atlas. 5. provision for masking/encrypting data based on role accessing the data. 6. security controls and audit. Deliverables: 1. demo showcasing all the above requirements. 2. setup instructions/scripts that will help install the POC setup. 3. any scripts used to configure the products used in solution. 4. explanation of each and every solution component used, its configuration etc. 5. spark jobs code used to showcase one end to end scenario

My project is about scala and spark optimization

I want to create the resource adapters in JBoss. The adapters will user the SAP JCo3 API for Client and Server communications Must provide some "admin" pages in Jboss/wildfly to add communication data for Client and Server connection and to Register/Start/Stop JCoServer It will have multiple SAP instances conneted to application. SAP wil make a call to send some informations, it will receive and store it in a database. SAP does have a J2EE server resource adapter, but it was made for SAP J2EE AS only. So we need to write one for JBoss, similar to SAP J2EE AS and after that we will write a aplication to receive SAP data and save it in a POSTGRES database Please look this document and see if it has enough information for your initiate.

One should select a public data set, think about a research project for example a clustering or classification problem or any machine learning model, and implement the model using cluster computing platforms like Apache spark, and evaluate the results.

Responsibilities: ● Review and understand business requirements ensuring that development tasks are completed within the timeline provided and that issues are fully tested with minimal defects ● Partner with software development team to implement best practices and optimize performance of Data applications. ● Collaborate across the company and interact with our customers to define, design and showcase new concepts and solutions ● Research on new Big Data technologies, assessing maturity and alignment of technology to business and technology strategy. ● Collaborate with other developers to ensure that client needs are met at all times ● Work in a rapid and agile development process to enable increased speed to market against a backdrop of appropriate controls. ● Implement good development a...

There is a scala Java udf which should be invoked to pyspark environment using transformed Function and load that transformed data back to hdfs but somehow I am unable to do it. So need to implement whole thing in scala which is call that transform function in scala. Need the extension code.