Azure Databricks Lead - Airline Domain
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
HTC Global Services is hiring an Azure Databricks Lead for an airline project in Abu Dhabi. The role involves leading a team of data engineers, designing data pipelines using Azure Databricks and PySpark, and implementing lakehouse solutions.
Key Skills for This Role
Responsibilities
- Lead and manage a team of Data Engineers, providing technical guidance, mentorship, and performance coaching
- Design, develop, and optimize large scale data pipelines using Azure Databricks and PySpark
- Architect and implement modern data lake and lakehouse solutions using Azure Data Lake Storage (ADLS)
- Build and maintain batch and real time streaming data pipelines using technologies such as Azure Event Hubs, Kafka, Spark Streaming, or Structured Streaming
- Collaborate with Solution Architects, Data Scientists, Product Owners, and business stakeholders to define data requirements and technical solutions
- Establish data engineering best practices, coding standards, CI/CD processes, and governance frameworks
- Ensure high availability, scalability, security, and performance of data platforms
- Drive Agile delivery, sprint planning, effort estimation, and technical reviews
- Troubleshoot complex data engineering challenges and optimize platform performance
- Evaluate emerging technologies and recommend improvements to the organization's data architecture
Requirements
- Strong experience with Azure Data Engineering ecosystem
- Expert level proficiency in Azure Databricks
- Strong hands on experience with PySpark and Apache Spark
- Experience with Azure Data Lake Storage (ADLS Gen2)
- Experience with streaming data solutions (Spark Structured Streaming, Azure Event Hubs, Apache Kafka)
- Experience with Azure Data Factory for orchestration and ETL
- Strong SQL and data modeling skills
- Experience with Delta Lake and Lakehouse architecture
- Knowledge of CI/CD pipelines, Azure DevOps, Git, and Infrastructure as Code
- Experience with Databricks Unity Catalog for enterprise data governance
- Hands on experience with Lakeflow Declarative Pipelines (formerly Delta Live Tables)
- Experience with Databricks Genie and AI powered conversational analytics
Full Job Posting
Job Overview
- HTC Global Services is hiring an Azure Databricks Lead for an airline project in Abu Dhabi, UAE.
- This is a 1 year extendable contract, onsite position.
Key Responsibilities
- Lead and manage a team of Data Engineers, providing technical guidance, mentorship, and performance coaching.
- Design, develop, and optimize large scale data pipelines using Azure Databricks and PySpark.
- Architect and implement modern data lake and lakehouse solutions using Azure Data Lake Storage (ADLS).
- Build and maintain batch and real time streaming data pipelines using technologies such as Azure Event Hubs, Kafka, Spark Streaming, or Structured Streaming.
- Collaborate with Solution Architects, Data Scientists, Product Owners, and business stakeholders to define data requirements and technical solutions.
- Establish data engineering best practices, coding standards, CI/CD processes, and governance frameworks.
- Ensure high availability, scalability, security, and performance of data platforms.
- Drive Agile delivery, sprint planning, effort estimation, and technical reviews.
- Troubleshoot complex data engineering challenges and optimize platform performance.
- Evaluate emerging technologies and recommend improvements to the organization's data architecture.
Required Skills & Experience
- Strong experience with Azure Data Engineering ecosystem.
- Expert level proficiency in Azure Databricks.
- Strong hands on experience with PySpark and Apache Spark.
- Experience with Azure Data Lake Storage (ADLS Gen2).
- Expertise in designing and implementing streaming data solutions using Spark Structured Streaming, Azure Event Hubs, Apache Kafka, or real time data processing frameworks.
- Experience with Azure Data Factory (ADF) for orchestration and ETL workflows.
- Strong SQL and data modeling skills.
- Experience with Delta Lake and Lakehouse architecture.
- Knowledge of CI/CD pipelines, Azure DevOps, Git, and Infrastructure as Code.
- Experience working with large scale enterprise data platforms.
- Strong experience with Databricks Unity Catalog for enterprise data governance, access control, metadata management, and secure data sharing.
- Hands on experience with Lakeflow Declarative Pipelines (formerly Delta Live Tables) for building scalable, reliable, and maintainable data ingestion and transformation pipelines.
Preferred Qualifications
- Databricks Certified Data Engineer Professional.
- Experience implementing enterprise wide data governance frameworks using Unity Catalog.
- Exposure to Databricks AI/BI, Genie, Mosaic AI, Vector Search, and Retrieval Augmented Generation (RAG) architectures.
- Experience supporting self service analytics and business intelligence solutions using Databricks Lakehouse Platform.
- Experience leading modernization initiatives involving migration from traditional data warehouses to Databricks Lakehouse.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at HTC Global Services
NOC Lead (Zabbix Expertise)
Dubai, UAE
HTC Global Services is seeking an experienced NOC Lead with strong Zabbix expertise for a 1-year contract in Dubai. The role involves leading NOC operations in a 24x7 environment, managing incidents, and mentoring junior
SAP FICO S/4 Hana Consultant
Abu Dhabi Emirate, UAE
HTC Global Services is hiring an SAP FICO S/4 Hana Consultant for a 1-year extendable contract in Abu Dhabi, UAE. The role involves leading SAP S/4HANA Finance transformation, stakeholder engagement, solution design, tes
Senior Delivery Manager cum Portfolio Manager – (SAP SuccessFactors)
Abu Dhabi Emirate, UAE
HTC Global Services seeks a Senior Delivery Manager cum Portfolio Manager with deep SAP SuccessFactors expertise to own the end-to-end HR Technology roadmap, manage budgets, and drive digital transformation in Abu Dhabi.
Scrum Master - Airline Domain
Abu Dhabi Emirate, UAE
HTC Global Services is hiring a Scrum Master with Airline domain expertise for a 1-year contract in Abu Dhabi. The role involves facilitating Agile ceremonies, removing impediments, and driving delivery of airline digita
AI Tester
Abu Dhabi Emirate, UAE
HTC Global Services is hiring an AI Tester for a 1-year contract in Abu Dhabi. The role involves testing conversational AI and airline booking solutions, with strong automation expertise. Requires 4+ years of software te
Change Manager (Airline Domain)
Abu Dhabi Emirate, UAE
HTC Global Services seeks a Change Manager with airline domain expertise for a 6-month contract role in Abu Dhabi. The role involves leading change management activities, conducting impact assessments, developing communi
Product Owner - (Airline + Data)
Abu Dhabi Emirate, UAE
HTC Global Services is hiring a Product Owner with Airline domain expertise for a 1-year contract in Abu Dhabi. The role involves defining and prioritizing user stories, working with multiple Scrum teams, and ensuring de
AI Engineer (Innovation)
Abu Dhabi, UAE
HTC Global Services seeks an AI Engineer (Innovation) with 6+ years of experience to design and develop AI/ML and Generative AI solutions. The role involves building predictive models, LLM-based applications, and deployi
NOC Lead (Zabbix Expertise)
Dubai, UAE
SAP FICO S/4 Hana Consultant
Abu Dhabi Emirate, UAE
Senior Delivery Manager cum Portfolio Manager – (SAP SuccessFactors)
Abu Dhabi Emirate, UAE
Scrum Master - Airline Domain
Abu Dhabi Emirate, UAE
AI Tester
Abu Dhabi Emirate, UAE
Change Manager (Airline Domain)
Abu Dhabi Emirate, UAE
Product Owner - (Airline + Data)
Abu Dhabi Emirate, UAE
AI Engineer (Innovation)
Abu Dhabi, UAE