Data Test Engineer – Databricks Delta Lake

We are looking for a skilled Test Engineer with strong experience in data validation, ETL testing, and Delta Lake on Databricks.

The candidate will be responsible for ensuring data quality, integrity, and performance across data pipelines and analytics layers.

Perform data validation and testing on Delta Lake tables

Validate ETL pipelines built using Databricks (batch and incremental loads)

Write and execute complex SQL queries to identify data issues, duplicates, and inconsistencies

Validate incremental load logic, partitioning, and data refresh processes

Perform source-to-target data reconciliation

Identify and troubleshoot data quality issues (duplicates, nulls, mismatches)

Collaborate with data engineers and BI teams to resolve defects

Validate data models used in reporting tools such as Power BI

Conduct performance testing on large datasets

Create and maintain test cases, test plans, and test reports

Hands-on experience with Databricks and Delta Lake

Experience in ETL/Data Warehouse testing

Solid understanding of:

Data pipelines

Incremental load concepts

Partitioning and indexing

Experience in data validation techniques

Familiarity with Azure ecosystem (preferred)

Good to Have

Knowledge of Power BI testing

Exposure to automation tools (Python / PySpark)

Duplicate data handling in Delta tables

Incremental load validation (StartRange / EndRange issues)

Late-arriving data scenarios

Partition refresh validation

Schema evolution testing

Data consistency across layers (Bronze → Silver → Gold)

Pay: AED7,000.00 - AED10,000.00 per month