Senior Data Quality Engineer
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
Robusta Studio seeks a Senior Databricks Data Quality Engineer to design and automate data quality frameworks. Requires 5-8 years experience with Databricks, PySpark, Delta Lake, and data quality management.
Key Skills for This Role
Responsibilities
- Configure and manage Databricks workspaces, compute clusters, PySpark notebooks, Delta Lake architecture, and Unity Catalog integrations
- Design scalable data quality processing frameworks across 170+ datasets and 1,346 prioritized Critical Data Elements (CDEs)
- Develop AI assisted profiling notebooks using PySpark to establish baseline data quality scores
- Assess data quality across six key dimensions: Completeness, Uniqueness, Validity, Consistency, Accuracy, Timeliness
- Design and build a scalable Data Quality Rule Factory using parameterized PySpark functions
- Enable automated deployment of over 6,700 data quality rules without manual rule by rule development
- Integrate data quality controls within Bronze, Silver, and Gold Delta Lake layers
- Implement quality gates that prevent data progression unless predefined thresholds are met
- Build automated data cleansing pipelines for standardization, deduplication, and schema harmonization
- Deploy MLflow managed machine learning models for anomaly detection, fuzzy duplicate detection, and exact duplicate identification
Requirements
- 5 to 8 years of experience in data quality engineering
- Strong expertise in Databricks, PySpark, Delta Lake, MLflow, and modern data quality management practices
- Experience with data profiling and quality assessment across six dimensions
- Experience building scalable data quality rule frameworks
- Experience with data cleansing and AI driven remediation
- Experience with data quality monitoring and reporting using Power BI
Full Job Posting
About the Role
- We are seeking an experienced Senior Databricks Data Quality Engineer to lead the design, implementation, and automation of enterprise scale data quality frameworks within a Databricks environment.
- The successful candidate will play a key role in establishing data quality controls, profiling frameworks, remediation processes, and AI assisted quality monitoring across a large scale data platform consisting of 170+ datasets and over 1,300 Critical Data Elements (CDEs).
- This role requires strong expertise in Databricks, PySpark, Delta Lake, MLflow, and modern data quality management practices.
Key Responsibilities
- Configure and manage Databricks workspaces, compute clusters, PySpark notebooks, Delta Lake architecture, and Unity Catalog integrations.
- Design scalable data quality processing frameworks across 170+ datasets and 1,346 prioritized Critical Data Elements (CDEs).
- Develop AI assisted profiling notebooks using PySpark to establish baseline data quality scores.
- Assess data quality across six key dimensions including: Completeness, Uniqueness, Validity, Consistency, Accuracy, Timeliness.
- Analyze null rates, duplicate records, invalid values, format violations, outliers, and schema drift.
- Design and build a scalable Data Quality Rule Factory using parameterized PySpark functions.
- Enable automated deployment of over 6,700 data quality rules without manual rule by rule development.
- Create reusable rule templates across datasets and data quality dimensions.
- Integrate data quality controls within Bronze, Silver, and Gold Delta Lake layers.
- Implement quality gates that prevent data progression unless predefined thresholds are met.
- Develop reusable Databricks Jobs for automated validation and monitoring.
- Build automated data cleansing pipelines for: Standardization, Deduplication, Schema harmonization.
Data Cleansing & AI Driven Remediation
- Deploy MLflow managed machine learning models for: Anomaly detection, Fuzzy duplicate detection, Exact duplicate identification.
- Ensure explainability of model outputs and support human in the loop validation processes.
- Design failed record handling frameworks and quarantine Delta tables.
- Capture failure reasons, affected CDEs, rule references, and timestamps.
- Develop automated reprocessing mechanisms for corrected records.
- Build Delta Lake aggregation tables for data quality metrics.
- Deliver data quality KPIs to Power BI dashboards including: Dimension level scores, Rule pass/fail rates, SLA adherence metrics.
- Configure automated alerting using Databricks SQL Alerts and Azure Monitor.
- Develop predictive models to identify datasets at risk of quality degradation.
- Support AI assisted Root Cause Analysis (RCA) using profiling outputs and machine learning techniques.
- Export and prepare remediation datasets for prioritization and governance reporting.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Robusta Studio
Sr. UX/UI Designer
Abu Dhabi, UAE
We are seeking an experienced Manager UX/UI Designer to lead the user experience design process for digital products related to Open Finance . The role focuses on creating intui...
Sr. UX/UI Designer (Open Finance) 6 Months Contract Onsite in UAE - Octopus
Abu Dhabi, UAE
Robusta Studio seeks an experienced Manager - UX/UI Designer to lead user experience design for Open Finance digital products. The role involves wireframing, prototyping, usability testing, and collaborating with cross-f
Senior UX/UI Designer
United Arab Emirates, UAE
Seeking an Experience Designer to create intuitive UX/UI designs, manage scalable design systems, and ensure seamless developer handoff for digital products.
Senior UX/UI Designers On-site Abu Dhabi - Octopus by RTG (8 months)
Abu Dhabi, UAE
Octopus by RTG seeks a Senior UX/UI Designer for an 8-month onsite contract in Abu Dhabi to lead design of digital products, own design systems, and ensure seamless developer handoff. Requires 5+ years experience, deep F
Senior Data Quality Engineer (4 Months Contract ) Onsite in UAE - Octopus by RTG
Abu Dhabi, UAE
Robusta Studio seeks a Senior Databricks Data Quality Engineer for a 4-month contract in Abu Dhabi. The role involves designing and implementing enterprise-scale data quality frameworks on Databricks, including profiling
Service Design Manager
United Arab Emirates, UAE
Seeking a strategic leader in service design to drive customer-centric solutions, mentor teams, and enhance business outcomes through innovative service strategies.
Senior Service Designer - On-site Abu Dhabi - Octopus by RTG (5 months contract)
Abu Dhabi, UAE
Who we are; Octopus by RTG is enabling a key partner organization to grow their tech teams while focusing on AI. We are currently looking for the right pioneers to join the team! Octopus is proud to be part of the Robust
Service Design Manager - On-site Abu Dhabi - Octopus by RTG (5 months contract)
Abu Dhabi, UAE
Who we are; Octopus by RTG is enabling a key partner organization to grow their tech teams while focusing on AI. We are currently looking for the right pioneers to join the team! Octopus is proud to be part of the Robust
Sr. UX/UI Designer
Abu Dhabi, UAE
Sr. UX/UI Designer (Open Finance) 6 Months Contract Onsite in UAE - Octopus
Abu Dhabi, UAE
Senior UX/UI Designer
United Arab Emirates, UAE
Senior UX/UI Designers On-site Abu Dhabi - Octopus by RTG (8 months)
Abu Dhabi, UAE
Senior Data Quality Engineer (4 Months Contract ) Onsite in UAE - Octopus by RTG
Abu Dhabi, UAE
Service Design Manager
United Arab Emirates, UAE
Senior Service Designer - On-site Abu Dhabi - Octopus by RTG (5 months contract)
Abu Dhabi, UAE
Service Design Manager - On-site Abu Dhabi - Octopus by RTG (5 months contract)
Abu Dhabi, UAE