Senior Data Engineer

Responsibilities

• Develop & Optimize Data Pipelines Build, test, and maintain ETL/ELT data pipelines using Azure Databricks & Apache Spark (PySpark) . Optimize performance and cost-efficiency of Spark jobs. Ensure data quality through validation, monitoring, and alerting mechanisms. Understand cluster types, configuration, and use-case for serverless
• Build, test, and maintain ETL/ELT data pipelines using Azure Databricks & Apache Spark (PySpark) .
• Optimize performance and cost-efficiency of Spark jobs.
• Ensure data quality through validation, monitoring, and alerting mechanisms.
• Understand cluster types, configuration, and use-case for serverless
• Implement Unity Catalog for Data Governance Design and enforce access control policies using Unity Catalog. Manage data lineage, auditing, and metadata governance . Enable secure data sharing across teams and external stakeholders.
• Design and enforce access control policies using Unity Catalog.
• Manage data lineage, auditing, and metadata governance .
• Enable secure data sharing across teams and external stakeholders.
• Integrate with Cloud Data Platforms Work with Azure Data Lake Storage / Azure Blob Storage/ Azure Event Hub to integrate Databricks with cloud-based data lakes, data warehouses, and event streams . Implement Delta Lake for scalable, ACID-compliant storage.
• Work with Azure Data Lake Storage / Azure Blob Storage/ Azure Event Hub to integrate Databricks with cloud-based data lakes, data warehouses, and event streams .
• Implement Delta Lake for scalable, ACID-compliant storage.
• Automate & Orchestrate Workflows Develop CI/CD pipelines for data workflows using Azure Databricks Workflows or Azure Data Factory . Monitor and troubleshoot failures in job execution and cluster performance .
• Develop CI/CD pipelines for data workflows using Azure Databricks Workflows or Azure Data Factory .
• Monitor and troubleshoot failures in job execution and cluster performance .
• Collaborate with Stakeholders Work with Data Analysts, Scientists, and Business Teams to understand requirements. Translate business needs into scalable data engineering solutions .