Sr Data Engineer

Senior Data Engineer must have prior hands-on experience designing and delivering data solutions on Databricks, including building and maintaining lakehouses using Delta Lake with a medallion (Bronze/Silver/Gold) architecture.

Strong knowledge working with data from financial and operational systems, with proven experience implementing Slowly Changing Dimensions (SCD Types 1, 2, and 3) using Delta Lake MERGE operations and Databricks SQL within a unified lakehouse model.

Experience running and optimizing cloud data platforms on Databricks, including cluster configuration, autoscaling policies, job scheduling via Databricks Workflows, and adherence to daily runbook SLAs through proactive monitoring and alerting.

Strong experience with Git-based version control integrated into Databricks (Databricks Repos / Git folders) and project management tools such as Jira, operating within Agile/Kanban delivery frameworks.

Strong experience with modern data architecture principles, including Unity Catalog for data governance, Delta Sharing, and cloud-native lakehouse design patterns on AWS with Databricks.

Ability to translate business requirements into technical designs and deliver production-grade data solutions within Databricks, from initial scoping through deployment.

Design and develop parallel and distributed ETL/ELT pipelines using Apache Spark (PySpark/Scala) on Databricks, applying partitioning, caching, and broadcast join strategies for optimal resource efficiency and throughput.

Understand data mapping and transformation requirements and implement them using Databricks-native constructs including Spark transformations (aggregations, joins, unions, window functions, lookups, and pivot/unpivot operations) and Delta Live Tables (DLT) for declarative pipeline development.

Develop and maintain Databricks Workflows and job orchestration logic (including dependency management, retry policies, and alerting), replacing traditional shell-based wrapper patterns with cloud-native, maintainable pipeline automation.

Proven experience designing and building integrations that support standard data modeling constructs — fact tables, dimension tables, star and snowflake schemas, and aggregations — implemented as Delta tables within Unity Catalog.

Ability to provide end-to-end technical guidance across the full software development life cycle, from requirements gathering and architecture design through implementation, testing, and production deployment on Databricks.

Ability to produce high-quality solution design documentation, including data flow diagrams, pipeline architecture specs, and Unity Catalog data asset definitions, ensuring clarity for both technical and business stakeholders.