Senior Data Quality Engineer (4 Months Contract ) Onsite in UAE - Octopus by RTG
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
Robusta seeks a Senior Data Quality Engineer for a 4-month contract in Abu Dhabi to lead data quality frameworks in a Databricks environment. The role involves designing scalable quality controls, profiling, AI-assisted monitoring, and remediation across 170+ datasets.
Key Skills for This Role
Responsibilities
- Configure and manage Databricks workspaces, compute clusters, PySpark notebooks, Delta Lake architecture, and Unity Catalog integrations
- Design scalable data quality processing frameworks across 170+ datasets and 1,346 prioritized Critical Data Elements (CDEs)
- Develop AI assisted profiling notebooks using PySpark to establish baseline data quality scores
- Assess data quality across six key dimensions: Completeness, Uniqueness, Validity, Consistency, Accuracy, Timeliness
- Design and build a scalable Data Quality Rule Factory using parameterized PySpark functions
- Integrate data quality controls within Bronze, Silver, and Gold Delta Lake layers
- Build automated data cleansing pipelines for standardization, deduplication, and schema harmonization
- Deploy MLflow managed machine learning models for anomaly detection and duplicate identification
- Design failed record handling frameworks and quarantine Delta tables
- Build Delta Lake aggregation tables for data quality metrics and deliver KPIs to Power BI dashboards
- Develop predictive models to identify datasets at risk of quality degradation
Requirements
- Bachelor's degree in Computer Science, Data Engineering, Information Systems, or a related field
- 5+ years of experience in Data Engineering or Data Quality Engineering
- 3+ years of hands on experience with Databricks and PySpark
- Strong expertise in Delta Lake architecture and data pipeline development
- Experience with Unity Catalog implementation and governance
- Hands on experience with MLflow and machine learning deployment
- Strong SQL skills and data modeling expertise
- Experience building enterprise scale data quality frameworks
- Experience integrating Databricks with Power BI and Azure services
- Strong understanding of data governance, metadata management, and data quality dimensions
Full Job Posting
About the Role
- We are seeking an experienced Senior Databricks Data Quality Engineer to lead the design, implementation, and automation of enterprise scale data quality frameworks within a Databricks environment.
- The successful candidate will play a key role in establishing data quality controls, profiling frameworks, remediation processes, and AI assisted quality monitoring across a large scale data platform consisting of 170+ datasets and over 1,300 Critical Data Elements (CDEs).
Key Responsibilities
- Configure and manage Databricks workspaces, compute clusters, PySpark notebooks, Delta Lake architecture, and Unity Catalog integrations.
- Design scalable data quality processing frameworks across 170+ datasets and 1,346 prioritized Critical Data Elements (CDEs).
- Develop AI assisted profiling notebooks using PySpark to establish baseline data quality scores.
- Assess data quality across six key dimensions: Completeness, Uniqueness, Validity, Consistency, Accuracy, Timeliness.
- Analyze null rates, duplicate records, invalid values, format violations, outliers, and schema drift.
- Design and build a scalable Data Quality Rule Factory using parameterized PySpark functions.
- Enable automated deployment of over 6,700 data quality rules without manual rule by rule development.
- Create reusable rule templates across datasets and data quality dimensions.
- Integrate data quality controls within Bronze, Silver, and Gold Delta Lake layers.
- Implement quality gates that prevent data progression unless predefined thresholds are met.
- Develop reusable Databricks Jobs for automated validation and monitoring.
- Build automated data cleansing pipelines for standardization, deduplication, and schema harmonization.
Requirements
- Bachelor's degree in Computer Science, Data Engineering, Information Systems, or a related field.
- 5+ years of experience in Data Engineering or Data Quality Engineering.
- 3+ years of hands on experience with Databricks and PySpark.
- Strong expertise in Delta Lake architecture and data pipeline development.
- Experience with Unity Catalog implementation and governance.
- Hands on experience with MLflow and machine learning deployment.
- Strong SQL skills and data modeling expertise.
- Experience building enterprise scale data quality frameworks.
- Experience integrating Databricks with Power BI and Azure services.
- Strong understanding of data governance, metadata management, and data quality dimensions.
Preferred Qualifications
- Microsoft Azure certifications.
- Databricks Certified Data Engineer Associate or Professional.
- Experience with enterprise data governance programs.
- Experience implementing AI assisted data quality and remediation solutions.
- Knowledge of Master Data Management (MDM) principles.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Robusta
Senior UX/UI Designers On-site Abu Dhabi - Octopus by RTG (8 months)
Abu Dhabi, UAE
Octopus by RTG seeks an exceptional Senior UX/UI Designer to lead the design of cutting-edge digital products across platforms. The role requires deep UX and UI craft, design system ownership, and collaboration with deve
Service Design Manager - On-site Abu Dhabi - Octopus by RTG (5 months contract)
, UAE
Who we are; Octopus by RTG is enabling a key partner organization to grow their tech teams while focusing on AI. We are currently looking for the right pioneers to join the team! Octopus is proud to be part of the Robust
Senior Service Designer - On-site Abu Dhabi - Octopus by RTG (5 months contract)
Abu Dhabi, UAE
Who we are; Octopus by RTG is enabling a key partner organization to grow their tech teams while focusing on AI. We are currently looking for the right pioneers to join the team! Octopus is proud to be part of the Robust
Senior Data Engineer - Hybrid - KSA - (10-12 Months) - RTG
جدة, KSA
Robusta assists organizations in transitioning to a digital-first approach, crafting unforgettable experiences for their customers. We provide strategy, design, product, and technology services to prominent businesses an
Client Growth Manager Onsite KSA - RTG
الرياض, KSA
We are seeking a results-driven Client Growth Manager to own and grow revenue across assigned industries and client portfolios. This role combines responsibilities traditionally handled by Sales and Account Management in
AI Platform Engineer - Hybrid - KSA - (10 Months) - RTG
جدة, KSA
Robusta assists organizations in transitioning to a digital-first approach, crafting unforgettable experiences for their customers. We provide strategy, design, product, and technology services to prominent businesses an
DevOps Engineer GCP - Hybrid - KSA - (10-12 Months) - RTG
جدة, KSA
Robusta assists organizations in transitioning to a digital-first approach, crafting unforgettable experiences for their customers. We provide strategy, design, product, and technology services to prominent businesses an
Java Software Engineer - Hybrid - KSA - (10-12 Months) - RTG
جدة, KSA
Robusta assists organizations in transitioning to a digital-first approach, crafting unforgettable experiences for their customers. We provide strategy, design, product, and technology services to prominent businesses an
Senior UX/UI Designers On-site Abu Dhabi - Octopus by RTG (8 months)
Abu Dhabi, UAE
Service Design Manager - On-site Abu Dhabi - Octopus by RTG (5 months contract)
, UAE
Senior Service Designer - On-site Abu Dhabi - Octopus by RTG (5 months contract)
Abu Dhabi, UAE
Senior Data Engineer - Hybrid - KSA - (10-12 Months) - RTG
جدة, KSA
Client Growth Manager Onsite KSA - RTG
الرياض, KSA
AI Platform Engineer - Hybrid - KSA - (10 Months) - RTG
جدة, KSA
DevOps Engineer GCP - Hybrid - KSA - (10-12 Months) - RTG
جدة, KSA
Java Software Engineer - Hybrid - KSA - (10-12 Months) - RTG
جدة, KSA