Lead / Manager - Data Engineering

Key Responsibilities

• Lead and manage a team of Data Engineers and provide technical guidance and mentorship.
• Design, develop, and optimize scalable data pipelines in an on-premise environment.
• Build robust ETL/ELT workflows using Python, Spark, SQL, and Airflow .
• Design and manage complex Apache Airflow DAGs for data pipeline orchestration.
• Develop and optimize Spark-based data processing solutions for large datasets.
• Write complex and optimized SQL queries for data extraction, transformation, and analysis.
• Troubleshoot pipeline failures, performance issues, and data quality challenges.
• Work closely with architects, business stakeholders, and cross-functional teams to understand requirements and deliver solutions.
• Conduct code reviews and ensure adherence to engineering and development best practices.
• Drive technical design, estimation, planning, and end-to-end project delivery.
• Monitor team performance, project timelines, risks, and dependencies.
• Mentor engineers and contribute to building a strong data engineering practice.
• 8+ years of overall experience in Data Engineering.
• Strong hands-on experience with Python .
• Strong experience with Apache Spark / PySpark .
• Advanced SQL skills, including complex joins, CTEs, window functions, subqueries, and query optimization.
• Strong experience with Apache Airflow for workflow orchestration and scheduling.
• Experience working with on-premise data environments .
• Strong understanding of ETL/ELT processes and data pipeline architecture.
• Experience handling large volumes of data and optimizing data processing workloads.
• Good understanding of data quality, monitoring, troubleshooting, and performance optimization.

Leadership Requirements