Lead Data Engineer - Azure Databricks/Kafka
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
Virtusa is seeking a Lead Data Engineer with 5-8 years of experience to design and build streaming ingestion pipelines using Apache Spark, Databricks, and Kafka. The role involves implementing CDC, monitoring, and DevOps processes.
Key Skills for This Role
Responsibilities
- Design and develop streaming ingestion pipelines using Apache Spark (Structured Streaming) and Databricks Auto Loader
- Implement CDC and deduplication logic by capturing change events from source databases using Debezium or native CDC tools
- Scale ingestion through configuration by building a config driven framework using Airflow, DBX Jobs, or Delta Live Tables
- Implement monitoring, observability, and security by capturing streaming query metrics and publishing to Prometheus and Grafana
- Participate in DevOps processes using CI/CD pipelines (Jenkins or GitHub Actions) and manage infrastructure with Terraform
Requirements
- 5 8 years of experience designing and building data pipelines using Apache Spark, Databricks, or equivalent big data frameworks
- Hands on expertise with streaming and messaging systems such as Apache Kafka, Confluent Cloud, RabbitMQ, or Azure Event Hub
- Deep understanding of relational databases and CDC, with proficiency in SQL Server, Oracle, or other RDBMSs
- Proficiency in programming languages such as Python, Scala, or Java
- Solid knowledge of SQL for data manipulation and transformation
- Cloud platform expertise, specifically with Azure or AWS services for data storage, compute, and orchestration
- Knowledge of data Lakehouse architectures, Delta Lake, partitioning strategies, and performance optimization
- Familiarity with Git, CI/CD pipelines, and infrastructure as code
Full Job Posting
Responsibilities
- Design and develop streaming ingestion pipelines using Apache Spark (Structured Streaming) and Databricks Auto Loader to consume files from cloud storage or messages from Kafka/RabbitMQ/Confluent Cloud and ingest them into Delta Lake, ensuring schema evolution and exactly once semantics.
- Implement CDC and deduplication logic by capturing change events from source databases using Debezium, built in CDC features of SQL Server/Oracle, or other connectors, and apply watermarking and drop duplicate strategies based on primary keys and event timestamps.
- Scale ingestion through configuration by building a config driven framework such as using Airflow, DBX Jobs, or Delta Live Tables that iterates over metadata tables to deploy/update ingestion pipelines for hundreds of tables/sources without code duplication.
- Implement monitoring, observability, and security by capturing streaming query metrics and publishing them to monitoring platforms like Prometheus and Grafana, setting up dashboards for lag, files processed, and processing duration, and enforcing role based access control, encryption, and data maski
- Participate in DevOps processes by using CI/CD pipelines, such as Jenkins or GitHub Actions, to automate the deployment of jobs, managing infrastructure with Terraform or similar tools, and following best practices for version control and code reviews.
Requirements
- 5–8 years of experience designing and building data pipelines using Apache Spark, Databricks, or equivalent big data frameworks.
- Hands on expertise with streaming and messaging systems such as Apache Kafka, Confluent Cloud, RabbitMQ, or Azure Event Hub, including creating producers, consumers, and topics and integrating them into downstream processing.
- Deep understanding of relational databases and CDC, with proficiency in SQL Server, Oracle, or other RDBMSs and experience capturing change events using Debezium or native CDC tools.
- Proficiency in programming languages such as Python, Scala, or Java.
- Solid knowledge of SQL for data manipulation and transformation.
- Cloud platform expertise, specifically with Azure or AWS services for data storage, compute, and orchestration.
- Knowledge of data Lakehouse architectures, Delta Lake, partitioning strategies, and performance optimization.
- Familiarity with Git, CI/CD pipelines, and infrastructure as code.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Virtusa
Business Analyst - LOS
Dubai, UAE
Virtusa is seeking a Business Analyst with expertise in Loan Origination Systems (LOS) to engage with retail lending teams and define future-state origination journeys. The role requires 8-10 years of banking experience
GTB - Technology Delivery Manager
Dubai, UAE
Virtusa seeks a GTB Technology Delivery Manager to own end-to-end technology delivery for GTB applications including Payments, Cash Management, Trade Finance, and Supply Chain Finance as part of a Core Banking transforma
Infrastructure Engineer
Doha, QAT
Virtusa seeks an Infrastructure Engineer specializing in Nutanix AHV-based platforms. The role involves migrating server workloads using lift and shift and parallel build strategies, managing Nutanix HCI, and supporting
Dotnet Technical Lead
Doha, QAT
Virtusa is seeking a Dotnet Technical Lead with 8-12 years of experience to lead a team of .NET and React developers. The role involves designing scalable enterprise applications, developing REST APIs and microservices,
Treasury Testing
Dubai, UAE
Virtusa is looking for a Manual Test Engineer with automation knowledge and 5+ years of banking/payments testing experience. The role requires strong knowledge of SWIFT MT/MX messages, ISO 20022, and experience with paym
Treasury Testing
Dubai, UAE
Virtusa is looking for a Manual Test Engineer with automation knowledge for a Treasury Testing role in Dubai. The role requires 5+ years of banking/payments testing experience, strong knowledge of SWIFT MT/MX messages an
Java Developer
Doha, QAT
Virtusa is looking for a Java Developer with expertise in Agile, Web Services, Java, Spring Boot, and Microservices. The role involves backend development using Java 8/11, Spring Boot, REST APIs, and cloud platforms. Nic
Java Developer
Doha, QAT
Virtusa is looking for a Java Developer with strong backend development skills in Java, Spring Boot, and Microservices. The role requires experience with REST APIs, cloud platforms, and AI integration. Candidates should
Business Analyst - LOS
Dubai, UAE
GTB - Technology Delivery Manager
Dubai, UAE
Infrastructure Engineer
Doha, QAT
Dotnet Technical Lead
Doha, QAT
Treasury Testing
Dubai, UAE
Treasury Testing
Dubai, UAE
Java Developer
Doha, QAT
Java Developer
Doha, QAT