Data Engineer II

Job Responsibilities

• Provide technical guidance and support to development teams and clients throughout the solution lifecycle. Troubleshoot and resolve complex issues related to Cloud based Data Lake and Data Mesh environments. Stay current with emerging technologies and trends and recommend best practices.
• Develop and implement scalable and secure cloud solutions for Big Data Platforms. Optimize data pipelines using distributed cloud computing to ensure that they support high volume / high velocity data streams and are scalable to meet growth expectations.
• Perform tuning of a Big Data /NoSQL environment with underlying impact analysis of distributed jobs to enhance data processing, analytics, and machine learning capabilities. Implement a mixed batch/near-real time architecture to analyze, index, and publish data for applications.
• Create and maintain comprehensive documentation for solution designs, configurations, and best practices. Provide training and support to clients and internal teams on solution architecture. Conduct presentations, demos, and workshops to showcase solution
• Required Qualifications
• 5+ years of experience in Database Development, Master Data Management (MDM), and Data Integration initiatives.
• 5+ years of hands-on experience designing and implementing ETL/ELT solutions using Azure data integration technologies, including Azure Data Factory, Azure Functions, Logic Apps, and Azure Databricks.
• Strong expertise in Azure Databricks and PySpark for developing scalable, high-performance data pipelines.
• Proficient in SQL development, with extensive experience writing, optimizing, and troubleshooting complex queries in Azure-based or other relational database platforms.
• Solid understanding of Software Development Life Cycle (SDLC) best practices, source code management, CI/CD processes, and Azure DevOps/GitHub integration.
• Strong organizational, analytical, and problem-solving skills with exceptional attention to detail.
• Experience supporting multiple clients in regulated and compliance-driven environments.
• Ability to collaborate effectively across teams to implement and promote data management frameworks and governance standards.
• Proven ability to translate business requirements into scalable technical solutions.
• Excellent verbal and written communication skills, with the ability to engage both technical and business stakeholders.