Data Quality Engineer

3 Months

Overview

Data Quality Framework & Platform

• Configure and maintain Databricks, Delta Lake, Unity Catalog, and PySpark frameworks.
• Establish scalable data quality controls and deployment standards.
• Optimise platform performance for large-scale profiling and rule execution.

Data Profiling & Quality Assessment

• Develop profiling solutions to assess completeness, accuracy, consistency, validity, timeliness, and uniqueness.
• Analyse data quality issues including nulls, duplicates, invalid values, schema drift, and outliers.
• Produce baseline quality assessments and recommendations.

Rule Development & Automation

• Build a reusable Data Quality Rule Factory using PySpark.
• Develop scalable, parameterised validation rules.
• Integrate quality controls into data pipelines and implement automated quality gates.

Data Cleansing & Remediation

• Create automated cleansing, standardisation, enrichment, and deduplication processes.
• Utilise machine learning techniques for anomaly detection and duplicate identification.
• Support exception handling, failed record management, and reprocessing workflows.

Monitoring & Reporting

• Develop quality metrics, dashboards, and reporting datasets.
• Monitor data quality performance and implement automated alerts.
• Produce Data Quality Index (DQI) scores, compliance metrics, and operational reporting outputs.

Governance & Stakeholder Management

• Collaborate with data governance, modelling, and engineering teams.
• Maintain documentation, audit trails, and quality artefacts.
• Support governance reviews and stakeholder workshops.

Required Experience

• Strong experience implementing enterprise Data Quality frameworks.
• Advanced Databricks and PySpark expertise.
• Experience with Delta Lake, Unity Catalog, Databricks Jobs/Workflows, and Databricks SQL.
• Proven track record building scalable validation and quality rule frameworks.
• Knowledge of ML-driven data quality monitoring and anomaly detection.
• Experience integrating quality metrics into reporting tools such as Power BI.
• Strong understanding of data governance, metadata management, and modern cloud/lakehouse architectures.

Key Deliverables

• Databricks-based Data Quality framework and rule engine.
• Automated profiling, validation, cleansing, and remediation solutions.
• Data quality monitoring, reporting, and alerting capabilities.
• Governance and compliance documentation supporting enterprise data management initiatives