Consultant | Azure Databricks | Bengaluru | Engineering | Data Modernization & Migration

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Databricks.
• Develop data transformation workflows using PySpark, Spark SQL, and Python.
• Build and orchestrate data pipelines using Azure Data Factory (ADF).
• Ingest data from various sources including databases, APIs, files, and cloud storage.
• Develop and optimize Delta Lake tables following Medallion Architecture (Bronze, Silver, Gold).
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Work with structured, semi-structured, and unstructured datasets.
• Implement data quality checks, logging, and monitoring for data pipelines.
• Collaborate with data architects, analysts, and business stakeholders to understand business requirements.
• Troubleshoot production issues and provide timely resolutions.
• Participate in code reviews and follow coding standards and best practices.
• Support deployment activities using Azure DevOps or Git-based CI/CD pipelines.
• Create and maintain technical documentation for data solutions.

Required Technical Skills

• 3–6 years of experience in Data Engineering.
• Hands-on experience with Azure Databricks.
• Strong expertise in PySpark, Apache Spark, and Spark SQL.
• Proficiency in Python and SQL.
• Experience with Azure Data Factory (ADF).
• Experience with Azure Data Lake Storage Gen2 (ADLS Gen2).
• Good understanding of Delta Lake concepts.
• Knowledge of ETL/ELT development and data integration.
• Experience in Spark performance tuning and optimization.
• Familiarity with Git and Azure DevOps.
• Understanding of data modeling concepts and relational databases.
• Experience working in Agile/Scrum environments.

Preferred Skills

• Azure Synapse Analytics
• Databricks Workflows
• Unity Catalog
• Azure Key Vault
• Azure Event Hubs or Kafka
• REST API integration
• Power BI integration
• Basic knowledge of Terraform or Infrastructure as Code (IaC)
• Experience with streaming data processing

Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
• Strong problem-solving and analytical skills.
• Good communication and collaboration skills.
• Ability to work independently and as part of a team.

Preferred Certifications

Key Technologies

Nice to Have

Preferred Qualifications

• Informatica IDMC or MDM certifications
• Experience in cloud migration programs or hybrid architectures
• Exposure to real-time streaming or event-driven architectures

Experience and Qualification

• B.E./ B.Tech. or M.C.A. in Computer Science from a reputed University
• 4-7 years of relevant industry experience