Data Engineer (Azure & Databricks) - Lead Analyst

Position Description:

Your future duties and responsibilities:

• Job Title: Data Engineer (Azure & Databricks) Position: Lead Analyst Experience: 10–12 Years Category: Software Development / Engineering Shift: General Shift Work Mode: Work from Office / Hybrid (as per business requirement) Primary Location: India, Karnataka, Bangalore CGI Office: Bangalore Employment Type: Full Time

Required qualifications to be successful in this role:

• About the Role : We are seeking a highly experienced and technically hands on Lead Data Engineer to design, build, and optimize our next generation Enterprise Data Warehouse and data Lakehouse solutions. You will be responsible for leading complex data integration initiatives, migrating legacy workloads, and building highly scalable ETL/ELT pipelines using Python, Databricks, and the Azure Cloud ecosystem. As a technical leader, you will collaborate closely with enterprise architects, business stakeholders, and data scientists to ensure our data infrastructure is robust, secure, and aligned with business objectives. Mandatory Key Responsibilities : . Architecture & Design: Design and implement modern, scalable Data Warehousing and Data Lakehouse architectures on Microsoft Azure using Databricks and Delta Lake. . ETL/ELT Development: Architect, build, and maintain highly efficient data pipelines to ingest, clean, transform, and load massive volumes of data from various sources (relational, non relational, APIs, flat files). . Databricks & PySpark Mastery: Write optimized, production grade PySpark and Python code in Azure Databricks for complex data transformations and advanced analytics. . Legacy Modernization: Lead the migration of legacy Enterprise Data Warehouse (EDW) workloads (e.g., traditional SQL stored procedures, Unix scripts, legacy ETL tools) to a modern Azure/Databricks cloud stack. . Azure Infrastructure: Utilize and orchestrate Azure native services including Azure Data Factory (ADF), Azure Data Lake Storage (ADLS Gen2), Azure Key Vault, and Azure SQL/Synapse. . Performance Tuning: Monitor, troubleshoot, and optimize Spark clusters, SQL queries, and ETL jobs for maximum speed, efficiency, and cost effectiveness. . Leadership & Mentorship: Lead a team of data engineers, conduct code reviews, enforce engineering best practices (CI/CD, automated testing), and guide the team on technical roadblocks. Required Skills & Qualifications . Experience: 10 to 12 years of core experience in Data Engineering, Business Intelligence, or Enterprise Data Warehousing. . Data Warehousing: Deep understanding of data modeling concepts (Kimball, Inmon, Data Vault), schema design (Star/Snowflake), and historical data tracking (SCD Types 1, 2, 3). . ETL/ELT: Extensive background in designing enterprise grade data integration frameworks and orchestrating complex workflows. . Python: Strong programming proficiency in Python (Pandas, core Python, OOP) for data manipulation, automation, and API integration. . Databricks & Spark: 4+ years of hands on experience with Databricks, Apache Spark (PySpark & Spark SQL), and Delta Lake architecture. . Azure Cloud: Proven expertise in the Azure Data ecosystem, specifically Azure Data Factory (ADF) and ADLS Gen2. . Advanced SQL: Expert level SQL skills, capable of writing and tuning highly complex queries, window functions, and migrating legacy stored procedures. Preferred Skills . Experience migrating legacy on premise EDW environments (SQL Server, Teradata, Oracle, Unix Shell scripting) to the cloud. . Experience with CI/CD pipelines and DevOps practices using Azure DevOps, Git, or GitHub Actions. . Understanding of Data Governance, Data Quality frameworks, and compliance standards (e.g., RBAC, Unity Catalog in Databricks). Certifications: Azure Data Engineer Associate (DP 203) or Databricks Certified Data Engineer Professional #LI-SS65