{bc}
Apply
linkedin

Data Quality Engineer (Databricks)

RibbitZ LLC
Abu Dhabi, UAE
Full Time
Mid
1 months ago
DatabricksPySparkDelta LakeUnity CatalogData Quality FrameworksData Profiling
Free

Job Fit Check

Base Career helps you apply smarter for this job.

?%
Ready to Scan

Key skills for this role

DatabricksPySparkDelta Lake
Smart Apply

Full Job Posting

Role Overview

  • The Data Quality Engineer will be responsible for designing, implementing, and operating ADC's enterprise data quality framework within the Databricks platform.
  • The role will deliver automated profiling, quality rule execution, cleansing, monitoring, remediation support, and quality reporting capabilities across 170 datasets and 1,346 prioritised Critical Data Elements (CDEs).
  • Working closely with Data Modellers, Data Catalogue Specialists, business data owners, and platform engineers, the Data Quality Engineer will establish scalable and reusable quality controls that improve trust, accuracy, completeness, consistency, timeliness, validity, and uniqueness across ADC's da

Key Responsibilities

  • Configure and manage the Databricks environment supporting enterprise data quality operations.
  • Establish and maintain compute clusters, PySpark notebook frameworks, Delta Lake storage structures, and Unity Catalog integration.
  • Optimise platform performance for large scale profiling and rule execution across all in scope datasets and CDEs.
  • Implement development, testing, and production deployment standards for data quality assets.
  • Design and develop AI assisted profiling notebooks using PySpark.
  • Perform baseline data quality assessments across the six quality dimensions: Completeness, Accuracy, Consistency, Validity, Timeliness, Uniqueness.
  • Capture and analyse null value rates, duplicate records, invalid values, format violations, outliers, and schema drift.
  • Produce quality profiling outputs for all prioritised CDEs and datasets.
  • Design and implement a reusable Data Quality Rule Factory.
  • Build parameterised PySpark based rule templates capable of supporting large scale rule deployment.
  • Enable automated generation and management of approximately 6,730 data quality rules without manual rule by rule development.
  • Ensure rules are reusable, configurable, and maintainable across multiple datasets and domains.

Data Quality Controls and Pipeline Integration

  • Deploy quality rules as reusable Databricks Jobs integrated into Delta Lake processing pipelines.
  • Embed quality controls within Bronze, Silver, and Gold processing stages.
  • Implement automated quality gates preventing data progression where defined thresholds are not met.
  • Maintain rule traceability and execution history for audit and governance purposes.

Data Cleansing and Quality Improvement

  • Develop automated remediation and cleansing pipelines using PySpark.
  • Implement standardisation routines, data enrichment processes, deduplication logic, and schema harmonisation controls.
  • Deploy machine learning models managed through MLflow for anomaly detection, exact duplicate detection, and fuzzy matching.
  • Ensure all AI and ML recommendations are explainable, auditable, and routed through human in the loop validation processes where required.

Exception Management and Reprocessing

  • Design and manage exception handling processes for failed quality records.
  • Implement quarantine Delta Lake tables serving as the Failed Record Register.
  • Capture and maintain failure reason, associated CDE, rule reference, processing timestamp, and resolution status.
  • Develop reprocessing workflows to support correction and controlled re ingestion of remediated records.

Data Quality Metrics and Reporting

  • Develop Delta Lake metric aggregation structures supporting enterprise quality reporting.
  • Calculate and publish Data Quality Index (DQI) scores, dimension level quality metrics, rule pass/fail rates, dataset compliance scores, and SLA adherence indicators.
  • Provide curated outputs to support Power BI quality dashboards and executive reporting.

Monitoring, Alerting and Predictive Quality Management

  • Configure automated quality monitoring and alerting mechanisms.
  • Implement threshold based notifications using Databricks SQL Alerts and Azure Monitor integrations.
  • Develop predictive risk scoring models to identify datasets at risk of future quality degradation.
  • Support proactive quality management and operational intervention activities.

Root Cause Analysis and Remediation Support

  • Apply Databricks machine learning and pattern analysis techniques to profiling and rule execution outputs.
  • Support AI assisted root cause analysis across established remediation categories.
  • Identify recurring quality issues, systemic defects, and process breakdowns.
  • Produce prioritised remediation datasets for business and operational stakeholders.
  • Export remediation outputs to Power BI and Excel to support Remediation Tiering Matrix, Prioritisation Scoring Models, and governance reporting processes.

Governance, Compliance and Stakeholder Engagement

  • Collaborate with Data Modellers and Data Catalogue Specialists to ensure quality controls align with authoritative data definitions and metadata standards.
  • Support DDA and DGE governance processes by producing required quality artefacts and evidence.
  • Maintain documentation, version control, and audit trails for all quality assets, rules, models, and processes.
  • Participate in quality reviews, governance forums, and stakeholder workshops.

Required Skills and Experience

  • Strong experience designing and implementing enterprise Data Quality frameworks.
  • Advanced Databricks engineering experience.
  • Strong PySpark development skills.
  • Experience with Delta Lake, Unity Catalog, Databricks Workflows and Jobs, Databricks SQL.
  • Experience building scalable data validation and quality rule frameworks.
  • Knowledge of machine learning techniques for anomaly detection and data quality monitoring.
  • Experience using MLflow for model management and deployment.
  • Strong understanding of data governance, metadata management, and data lifecycle processes.
  • Experience integrating data quality metrics into reporting platforms such as Power BI.
  • Knowledge of cloud based data engineering and modern lakehouse architectures.

Apply for this job in 1 click

Skip the repetitive application forms

Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.

Sarah M.James T.Maya R.

Trusted by over 500,000 job seekers on Base Career

Start Free Today

More from this employer

More jobs at RibbitZ LLC

Data Modeller (Erwin Data Modeler)

Abu Dhabi, UAE

1 months agoFull Time

Data Engineer (Microsoft Fabric | Azure | Databricks)

Dubai, UAE

2 months agoContract

Informatica Technical Lead (DG/DQ)

Dubai, UAE

2 months agoContract

SAP MM Lead Solution Architect (Procurement)

Dubai, UAE

2 months agoFulltime

Java/Joget Developer

Abu Dhabi, UAE

3 months agoContract