Senior Data Engineer

Role Summary

Data Engineering Responsibilities

• Design and maintain pipelines with Databricks, PySpark, Azure Data Factory, Azure Logic Apps, and Apache Airflow.
• Build orchestration frameworks integrating Airflow DAGs with ADF and Logic Apps.
• Develop event-driven, micro-batch, hybrid scheduling, and dependency-based workflows.
• Build ETL/ELT pipelines and transformation frameworks for large structured and semi-structured datasets.
• Implement pipeline monitoring, dependency management, error handling, and recovery.
• Integrate pipelines with ADLS Gen2, Azure Blob Storage, and event-driven Azure services.
• Optimize pipelines for performance, scalability, reliability, and cost efficiency.
• Implement CI/CD using GitHub Actions and automated deployment processes.
• Ensure data quality, governance, security, and compliance.
• Collaborate on AI/ML data platforms, modernization, architecture reviews, and technical planning.
• Mentor junior data engineers and establish engineering best practices.

Required Qualifications

• Bachelor’s degree in Computer Science, Information Technology, Engineering, Data Science, or a related discipline.
• At least 7 years of experience in data engineering.
• Experience designing enterprise-scale data platforms and cloud-native data solutions.
• Experience managing complex production data pipelines.
• Strong Databricks, PySpark, Delta Lake, distributed processing, Spark optimization, and large-scale ETL/ELT experience.
• High proficiency in Azure Data Factory, Azure Logic Apps, ADLS Gen2, Azure Blob Storage, and Azure cloud-native architectures.
• Strong Apache Airflow experience, including DAGs, scheduling, dependencies, orchestration, and failure recovery.
• Strong Python and advanced SQL programming skills.
• Understanding of ETL/ELT patterns, pipeline architecture, lakehouse concepts, data modeling, quality, and governance.

Preferred Qualifications

• Experience with event-driven architecture, API integrations, AI/ML data engineering, ML pipelines, or MLflow.
• Knowledge of lakehouse architecture and governance frameworks.
• Experience with the Azure Databricks ecosystem and monitoring or observability tools.
• Azure or Databricks certifications and Agile/Scrum experience are preferred.

Technical Competencies

• Databricks engineering, PySpark development, Azure data engineering, pipeline architecture, workflow orchestration, Airflow DAG development, ETL/ELT, Delta Lake, cloud modernization, reliability engineering, data governance, security, and performance optimization.

Role Outcomes