Data Quality Engineer (Databricks)
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
Datamatics Technologies is seeking a Data Quality Engineer with 5-7 years of experience to design and implement an enterprise data quality framework on the Databricks platform. The role involves building automated profiling, quality rule execution, cleansing, monitoring, and reporting across 170 datasets.
Key Skills for This Role
Responsibilities
- Configure and manage the Databricks environment for enterprise data quality operations
- Design and develop AI assisted profiling notebooks using PySpark
- Design and implement a reusable Data Quality Rule Factory
- Deploy quality rules as reusable Databricks Jobs integrated into Delta Lake processing pipelines
- Develop automated remediation and cleansing pipelines using PySpark
- Design and manage exception handling processes for failed quality records
- Develop Delta Lake metric aggregation structures for enterprise quality reporting
- Configure automated quality monitoring and alerting mechanisms
- Support AI assisted root cause analysis and produce remediation datasets
- Collaborate with Data Modellers and Data Catalogue Specialists to ensure quality controls align with metadata standards
Requirements
- 5 to 7 years of experience in data quality engineering
- Strong experience designing and implementing enterprise Data Quality frameworks
- Advanced Databricks engineering experience
- Strong PySpark development skills
- Experience with Delta Lake, Unity Catalog, Databricks Workflows and Jobs
Full Job Posting
Role Overview
- The Data Quality Engineer will be responsible for designing, implementing, and operating ADC's enterprise data quality framework within the Databricks platform.
- The role will deliver automated profiling, quality rule execution, cleansing, monitoring, remediation support, and quality reporting capabilities across 170 datasets and 1,346 prioritised Critical Data Elements (CDEs).
Key Responsibilities
- Configure and manage the Databricks environment supporting enterprise data quality operations.
- Establish and maintain compute clusters, PySpark notebook frameworks, Delta Lake storage structures, and Unity Catalog integration.
- Optimise platform performance for large scale profiling and rule execution across all in scope datasets and CDEs.
- Implement development, testing, and production deployment standards for data quality assets.
- Design and develop AI assisted profiling notebooks using PySpark.
- Perform baseline data quality assessments across completeness, accuracy, consistency, validity, timeliness, and uniqueness.
- Capture and analyse null value rates, duplicate records, invalid values, format violations, outliers, and schema drift.
- Produce quality profiling outputs for all prioritised CDEs and datasets.
- Design and implement a reusable Data Quality Rule Factory with parameterised PySpark based rule templates.
- Enable automated generation and management of approximately 6,730 data quality rules without manual rule by rule development.
- Deploy quality rules as reusable Databricks Jobs integrated into Delta Lake processing pipelines.
- Embed quality controls within Bronze, Silver, and Gold processing stages and implement automated quality gates.
Required Skills and Experience
- Strong experience designing and implementing enterprise Data Quality frameworks.
- Advanced Databricks engineering experience.
- Strong PySpark development skills.
- Experience with Delta Lake, Unity Catalog, Databricks Workflows and Jobs.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Datamatics Technologies
ServiceNow GRC & SecOps Implementation
Riyadh, KSA
Datamatics Technologies seeks two ServiceNow Developers for a 6-month engagement in Riyadh, focusing on GRC and SecOps implementation. Candidates need at least 3 years of ServiceNow GRC experience and expertise in IRM, S
Data Platform Lead / Administrator (DevOps)
Riyadh, KSA
The company is seeking a Data Platform Lead / Administrator to design, manage, and optimize enterprise data platform infrastructure, ensuring availability, security, scalability, and performance. The role involves collab
Data Privacy Specialist
Riyadh, KSA
The company is seeking a Data Privacy Specialist to design, implement, and maintain the organization's Data Privacy framework, ensuring compliance with privacy laws and regulations. The role involves conducting privacy a
Data Management Specialist
Riyadh, KSA
Datamatics Technologies seeks a Data Management Specialist in Riyadh to implement and support enterprise Data Governance, Data Catalog, Metadata Management, and Data Quality solutions. The role requires 5+ years of hands
AI / ML Engineer
Riyadh, KSA
We are seeking AI / ML Engineers across multiple experience levels (T1 T5) to design, develop, train, deploy, and optimize machine learning models and AI solutions throughout th...
Data Scientist / Advanced Analytics Consultant
Riyadh, KSA
We are seeking Data Scientists and Advanced Analytics Consultants across multiple experience levels (T1 T5) to deliver advanced analytics, statistical modeling, machine learning...
MLOps Architect / Engineer
Riyadh, KSA
We are seeking an experienced MLOps Architect / Engineer to design, build, and operate enterprise-grade Machine Learning Operations (MLOps) platforms. The ideal candidate will d...
Solution Architect
Riyadh, KSA
Datamatics Technologies seeks an experienced Solution Architect with 10+ years in enterprise solution architecture, data platform engineering, and system integration. The role involves designing scalable, secure solution
ServiceNow GRC & SecOps Implementation
Riyadh, KSA
Data Platform Lead / Administrator (DevOps)
Riyadh, KSA
Data Privacy Specialist
Riyadh, KSA
Data Management Specialist
Riyadh, KSA
AI / ML Engineer
Riyadh, KSA
Data Scientist / Advanced Analytics Consultant
Riyadh, KSA
MLOps Architect / Engineer
Riyadh, KSA
Solution Architect
Riyadh, KSA