Senior Associate _Azure Databricks Pyspark Developer -_Data & Analytics _Advisory _Mumbai & Pune

Role Overview

Data Lake and Pipeline Responsibilities

• Design, implement, and manage Data Lake and Lakehouse architectures.
• Develop and maintain scalable data pipelines and workflows.
• Use Azure Data Lake Services for data storage and management.
• Use PySpark for data processing and transformations.
• Implement Delta Live Tables for real-time data processing and analytics.
• Ensure data quality and consistency across the data lifecycle.

Governance and Integration

• Use Unity Catalog for data governance and metadata management.
• Ensure robust data security and compliance with industry standards.
• Extract, transform, and load data from multiple sources, including SAP and Dynamics 365.
• Use Azure Data Factory and Synapse Analytics for data integration and orchestration.
• Optimize job performance and data storage and retrieval for performance and cost efficiency.

Collaboration and Improvement

• Work with data scientists, analysts, and stakeholders to understand data requirements.
• Provide technical guidance and mentorship to junior team members.
• Stay updated on trends and technologies in data engineering and cloud computing.
• Continuously improve data processes and infrastructure for efficiency and scalability.

Mandatory Skills

• Proficiency in PySpark and Python is required.
• Strong experience with Azure Data Lake Services and Data Lake architecture is required.
• Hands-on experience with Databricks for data engineering and analytics is required.
• Knowledge of Unity Catalog, Delta Live Tables, Azure Data Factory, and Synapse Analytics is required.
• Experience with ETL and data from multiple sources is required.
• Excellent problem-solving, analytical, communication, collaboration, and data-quality skills are required.

Preferred Skills

• Hands-on ETL and Data Lake experience is good to have.
• Experience with SAP and Dynamics 365 is good to have.
• Databricks Certified Data Engineer Professional certification is good to have.
• Technical guidance and mentorship experience is good to have.

Certifications and Education

• Azure Data Engineering or relevant Azure certification is listed.
• DP203 certification is required.
• Databricks Certified Data Engineer Associate certification is required.
• A regular full-time B.E., B.Tech, MCA, M.E., M.Tech, MBA, or PGDM qualification is required.
• The required experience is 2–4 years.