ETL Jobs

I am moving a single, fully structured PostgreSQL dataset (in the 1 TB – 10 TB range) into Snowflake and want the entire flow handled through Snowflake Openflow so that future incremental loads can piggy-back on the same orchestration. You will start by designing the Openflow pipeline, mapping every PostgreSQL schema, table, and index to its Snowflake equivalent, and putting in place the monitoring hooks Snowflake recommends for high-volume transfers. The goal is a like-for-like copy—no transformations—so row counts, constraints, and data types must match exactly once the job completes. Because of the data volume I need solid strategies for parallelisation, fault tolerance, and restartability. Please plan for a test run on a reduced slice, followed by a full push, a...

Hiring: Lead Data Engineer (Data Platforms & Analytics Infrastructure) Job ID: GKS2005_1 --- Role: Lead Data Engineer Mode: Remote (Monthly visit to Coimbatore office preferred) Budget: ₹50,000 – ₹60,000/month Contract Duration: 8 Months Experience: 7 – 10 Years --- About the Role We are looking for a Lead Data Engineer to build and scale a robust data platform supporting analytics and AI/ML use cases. This role involves end-to-end ownership of the data lifecycle along with leading a team to deliver reliable and high-performance data solutions. --- Key Responsibilities Build scalable batch and real-time data pipelines Design and implement Lakehouse architecture (Delta Lake / Iceberg) Develop ETL/ELT workflows using Airflow, dbt, or Prefect Ensure data quality,...

Informatica Data Quality (IDQ) Specialist Availability- Immediate Mode - WFH Job Summary: Informatica Data Quality (IDQ) Specialist Allocation: Dedicated / Full-Time Objective: Own the end-to-end Data Quality management framework for the project, covering data profiling, rules definition, and remediation pipelines across the 8 in-scope data sources. Key Responsibilities & Scope of Work (SoW): Data Profiling & Analysis: Conduct detailed data profiling and analysis for 8 complex data sources (including systems like SAP, Moamalat, Bawaba, etc.) to identify data anomalies and gaps. Rules Catalog Development: Collaborate with business stakeholders to define, document, and build a reusable catalog of Data Quality (DQ) rules, thresholds, and issue classifications. Remediation & Cle...

Hiring: Sr. Data Engineer Job ID: GKS1805_3 --- Role: Senior Data Engineer Mode: Work From Home Budget: Dm on freelancer chat Availability: Immediate Joiners Only Open Positions: 1 Interview Process: AI Interview → L1 → 2 Client Rounds --- Role Overview We are looking for a high-impact Senior Data Engineer to design and build scalable, multi-tenant data systems on AWS. You will work across data engineering, backend systems, APIs, and ML workflows, contributing to end-to-end product delivery. --- Key Responsibilities - Design & build scalable multi-tenant systems on AWS (Glue, Lambda, Step Functions, ECS, S3, DynamoDB, SageMaker) - Develop robust data pipelines (ETL/ELT) for high-volume datasets - Build Lambda-based microservices & workflow orchestration ...

## Microsoft Fabric / Power BI Expert Required We are looking for a highly experienced Microsoft Fabric / Power BI expert to enhance and optimize an enterprise analytics platform with advanced reporting, predictive analytics, and AI-driven insights. ### Scope of Work * Advanced Power BI dashboard development and optimization * Microsoft Fabric Data Lake / Data Warehouse implementation * RBAC, Multi-Tenant architecture, and security implementation * Performance tuning for Fabric F2 capacity * Predictive Analytics, RCA, Anomaly Detection, and forecasting * Integration with APIs, ERP, HRMS, IoT, and external systems * Enterprise-grade scalable architecture and governance setup ### Mandatory Requirements * Proven hands-on experience in Microsoft Fabric and Power BI enterprise implementati...

I currently run an in-house spreadsheet application that has outgrown its limits. I want to migrate everything into a purpose-built database that can accept multiple incoming feeds—CSV files arriving by batch, live API calls, and occasional manual entries (often straight from Excel). Before any record lands in the tables it needs to pass through an AI layer that cleans, classifies, and enriches the data. Once stored, different user groups must see only what their privilege level allows, so role-based security is essential. My short-term goal is to generate a print-ready publication from the data; longer term I will also push the same content to a website and, potentially, a mobile app. Deliverables • A relational schema that mirrors the existing spreadsheet logic whil...

I need a cloud-native ETL pipeline built end-to-end on AWS, coded in PySpark and designed for production reliability. The pipeline will ingest data from three sources—databases, APIs, and file systems—then standardise and load it into an analytics-ready destination. Source files arrive in a mix of CSV, JSON, and Parquet, so the job must include automatic format detection, schema inference, and efficient column-wise writes. Beyond raw transformation, I want solid engineering practices: parameter-driven jobs, modular Spark code, unit tests, logging, alerting, and retry logic. Leveraging AWS native services such as Glue, EMR, Lambda, and S3 is expected, but I’m open to other AWS components if they shorten development time or lower cost. Candidates must have expertise in da...

I’m setting up an intensive, expert-level course for a team that already designs and operates complex AWS workloads. The objective is to deepen their Data Engineering skills on AWS, so we need a trainer who can move straight past fundamentals and into real-world architecture, scaling strategies, and cost-efficient design. Core focus: Data Engineering with AWS, specifically • Data Storage Solutions • Data Processing & ETL • Big Data Analytics Your role is to craft and deliver a hands-on program that blends concise theory, architectural white-boarding, and live labs. Glue, EMR, Redshift, Lake Formation, Kinesis, Lambda, Step Functions, and related services should feature prominently, but I’m open to your proven toolset and workflows. Deliverables 1...

Over the coming weeks I want to master three core AWS services—EC2, S3 glue and Lambda—and immediately apply them to real-world ETL testing scenarios. The entire program needs to be delivered as live online classes so I can ask questions in real time and work through hands-on labs while you guide me. On the ETL side, the agenda should give equal attention to data validation, data transformation and performance testing. I’m aiming to finish the course able to design, automate and benchmark an end-to-end AWS-based ETL pipeline with confidence. Please structure the engagement around practical demonstration and practice rather than slides alone, using the AWS console, CLI or Terraform where appropriate. To keep us both on track, here’s what I expect to receive: &bull...

Looking for BlackRock Aladdin Experts with Implementation experience. Data engineer, technical Business Analyst, please detail your specific experience. I need hands-on practitioners who have delivered full Aladdin Implementations, with a sharp focus on Performance and FactSet integration. Your background must show you have handled Data Mapping, carried out precise System Configuration, and driven rigorous Testing and Validation for this specific stream. Please reference concrete projects where you owned or significantly contributed to these areas, noting any challenges you solved around data pipelines, integration touchpoints, and post-go-live support. Include the tools and techniques you relied on inside the Aladdin ecosystem.