Data Quality Engineer (Databricks)
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
Datamatics is seeking a Data Quality Engineer to design and implement an enterprise data quality framework on Databricks for a 6-month contract in Abu Dhabi. You will build automated profiling, quality rules, cleansing pipelines, and monitoring across 170 datasets.
Key Skills for This Role
Responsibilities
- Configure and manage the Databricks environment supporting enterprise data quality operations
- Design and develop AI assisted profiling notebooks using PySpark
- Perform baseline data quality assessments across completeness, accuracy, consistency, validity, timeliness, and uniqueness
- Design and implement a reusable Data Quality Rule Factory with parameterized PySpark rule templates
- Deploy quality rules as reusable Databricks Jobs integrated into Delta Lake processing pipelines
- Develop automated remediation and cleansing pipelines using PySpark
- Design and manage exception handling processes for failed quality records
- Develop Delta Lake metric aggregation structures supporting enterprise quality reporting
- Configure automated quality monitoring and alerting mechanisms
- Support AI assisted root cause analysis and produce prioritized remediation datasets
Requirements
- Strong experience designing and implementing enterprise Data Quality frameworks
- Advanced Databricks engineering experience
- Strong PySpark development skills
- Experience with Delta Lake, Unity Catalog, Databricks Workflows and Jobs, Databricks SQL
- Experience building scalable data validation and quality rule frameworks
- Knowledge of machine learning techniques for anomaly detection and data quality monitoring
- Experience using MLflow for model management and deployment
- Strong understanding of data governance, metadata management, and data lifecycle processes
- Experience integrating data quality metrics into reporting platforms such as Power BI
- Knowledge of cloud based data engineering and modern lakehouse architectures
Full Job Posting
Role Overview
- The Data Quality Engineer will design, implement, and operate ADC's enterprise data quality framework within the Databricks platform.
- The role will deliver automated profiling, quality rule execution, cleansing, monitoring, remediation support, and quality reporting across 170 datasets and 1,346 prioritised Critical Data Elements (CDEs).
Key Responsibilities
- Configure and manage the Databricks environment supporting enterprise data quality operations.
- Establish and maintain compute clusters, PySpark notebook frameworks, Delta Lake storage structures, and Unity Catalog integration.
- Design and develop AI assisted profiling notebooks using PySpark.
- Perform baseline data quality assessments across completeness, accuracy, consistency, validity, timeliness, and uniqueness.
- Design and implement a reusable Data Quality Rule Factory with parameterized PySpark rule templates.
- Deploy quality rules as reusable Databricks Jobs integrated into Delta Lake processing pipelines.
- Develop automated remediation and cleansing pipelines using PySpark.
- Design and manage exception handling processes for failed quality records.
- Develop Delta Lake metric aggregation structures supporting enterprise quality reporting.
- Configure automated quality monitoring and alerting mechanisms.
- Support AI assisted root cause analysis and produce prioritized remediation datasets.
Required Skills and Experience
- Strong experience designing and implementing enterprise Data Quality frameworks.
- Advanced Databricks engineering experience.
- Strong PySpark development skills.
- Experience with Delta Lake, Unity Catalog, Databricks Workflows and Jobs, Databricks SQL.
- Experience building scalable data validation and quality rule frameworks.
- Knowledge of machine learning techniques for anomaly detection and data quality monitoring.
- Experience using MLflow for model management and deployment.
- Strong understanding of data governance, metadata management, and data lifecycle processes.
- Experience integrating data quality metrics into reporting platforms such as Power BI.
- Knowledge of cloud based data engineering and modern lakehouse architectures.
Deliverables
- Configured Databricks Data Quality environment.
- Enterprise Data Quality Rule Factory.
- AI assisted profiling notebooks and baseline assessment outputs.
- Automated data quality validation and gating processes.
- Data cleansing and remediation pipelines.
- Failed Record Register and reprocessing workflows.
- Data Quality metric aggregation tables.
- DQI reporting feeds and dashboard datasets.
- Predictive quality monitoring and alerting solutions.
- Root Cause Analysis and remediation support datasets.
- Governance, audit, and compliance artefacts supporting DDA and DGE reviews.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Datamatics Global Services Ltd
Data Platform Lead / Administrator (DevOps)
الرياض, KSA
Datamatics seeks a Data Platform Lead/Administrator to design, manage, and optimize enterprise data platform infrastructure in Riyadh. The role involves ensuring availability, security, scalability, and performance of da
Solution Architect
الرياض, KSA
Datamatics Global Services seeks an experienced Solution Architect for an onsite role in Riyadh. The architect will design, govern, and optimize enterprise-wide technology solutions, including data platforms, system inte
Solution Architect
Dubai, UAE
Datamatics Global Services seeks an experienced Solution Architect for an onsite role in Riyadh. The architect will design, govern, and optimize enterprise-wide technology solutions, including data platforms, system inte
Data Management Specialist
الرياض, KSA
Datamatics Global Services Ltd seeks a Data Management Specialist in Riyadh to implement and support enterprise Data Governance, Data Catalog, Metadata Management, and Data Quality solutions. The role requires 5+ years o
ServiceNow GRC & SecOps Implementation
الرياض, KSA
Datamatics Global Services Ltd seeks two ServiceNow Developers for a 6-month engagement in Riyadh implementing GRC and SecOps modules. Candidates must have minimum 3 years ServiceNow GRC implementation experience and kno
Data Privacy Specialist
الرياض, KSA
Datamatics Global Services is hiring a Data Privacy Specialist in Riyadh to design and maintain the organization's Data Privacy framework. The role involves conducting privacy assessments, managing risks, and ensuring co
Data Management Specialist (Data Governance, Catalog & Data Quality)
Dubai, UAE
Datamatics Global Services Ltd is seeking an experienced Data Management Specialist to support enterprise-wide Data Governance, Data Catalog, Metadata Management, Data Classification, and Data Quality initiatives. The ro
Data Management Specialist (Data Governance, Catalog & Data Quality)
الرياض, KSA
Datamatics Global Services is seeking an experienced Data Management Specialist to lead enterprise-wide Data Governance, Catalog, Metadata Management, and Data Quality initiatives in Riyadh. The role requires 5+ years of
Data Platform Lead / Administrator (DevOps)
الرياض, KSA
Solution Architect
الرياض, KSA
Solution Architect
Dubai, UAE
Data Management Specialist
الرياض, KSA
ServiceNow GRC & SecOps Implementation
الرياض, KSA
Data Privacy Specialist
الرياض, KSA
Data Management Specialist (Data Governance, Catalog & Data Quality)
Dubai, UAE
Data Management Specialist (Data Governance, Catalog & Data Quality)
الرياض, KSA