Lead Data Engineer - Azure Databricks/Kafka
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
Virtusa seeks a Lead Data Engineer to design and develop streaming ingestion pipelines using Apache Spark, Databricks, and Kafka. Requires 5-8 years of experience with big data frameworks, streaming systems, and cloud platforms like Azure or AWS.
Key Skills for This Role
Responsibilities
- Design and develop streaming ingestion pipelines using Apache Spark (Structured Streaming) and Databricks Auto Loader
- Implement CDC and deduplication logic using Debezium or built in CDC features
- Scale ingestion through config driven frameworks using Airflow, DBX Jobs, or Delta Live Tables
- Implement monitoring, observability, and security for streaming pipelines
- Participate in DevOps processes using CI/CD pipelines and infrastructure as code tools like Terraform
Requirements
- 5 8 years of experience designing and building data pipelines using Apache Spark, Databricks, or equivalent big data frameworks
- Hands on expertise with streaming and messaging systems such as Apache Kafka, Confluent Cloud, RabbitMQ, or Azure Event Hub
- Deep understanding of relational databases and CDC, with proficiency in SQL Server, Oracle, or other RDBMSs
- Proficiency in programming languages such as Python, Scala, or Java
- Solid knowledge of SQL for data manipulation and transformation
- Cloud platform expertise with Azure or AWS services for data storage, compute, and orchestration
- Knowledge of data Lakehouse architectures, Delta Lake, partitioning strategies, and performance optimization
- Familiarity with Git, CI/CD pipelines, and infrastructure as code
Full Job Posting
Job description
- Design and develop streaming ingestion pipelines using Apache Spark (Structured Streaming) and Databricks Auto Loader.
- Implement CDC and deduplication logic using Debezium or built in CDC features.
- Scale ingestion through config driven frameworks using Airflow, DBX Jobs, or Delta Live Tables.
- Implement monitoring, observability, and security for streaming pipelines.
- Participate in DevOps processes using CI/CD pipelines and infrastructure as code.
Requirements
- 5 8 years of experience designing and building data pipelines using Apache Spark, Databricks, or equivalent big data frameworks.
- Hands on expertise with streaming and messaging systems such as Apache Kafka, Confluent Cloud, RabbitMQ, or Azure Event Hub.
- Deep understanding of relational databases and CDC, with proficiency in SQL Server, Oracle, or other RDBMSs.
- Proficiency in programming languages such as Python, Scala, or Java.
- Solid knowledge of SQL for data manipulation and transformation.
- Cloud platform expertise with Azure or AWS services.
- Knowledge of data Lakehouse architectures, Delta Lake, partitioning strategies, and performance optimization.
- Familiarity with Git, CI/CD pipelines, and infrastructure as code.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Virtusa
Business Analyst - LOS
Dubai, UAE
Virtusa is seeking a Business Analyst with expertise in Loan Origination Systems (LOS) to engage with retail lending teams and define future-state origination journeys. The role requires 8-10 years of banking experience
GTB - Technology Delivery Manager
Dubai, UAE
Virtusa seeks a GTB Technology Delivery Manager to own end-to-end technology delivery for GTB applications including Payments, Cash Management, Trade Finance, and Supply Chain Finance as part of a Core Banking transforma
Infrastructure Engineer
Doha, QAT
Virtusa seeks an Infrastructure Engineer specializing in Nutanix AHV-based platforms. The role involves migrating server workloads using lift and shift and parallel build strategies, managing Nutanix HCI, and supporting
Dotnet Technical Lead
Doha, QAT
Virtusa is seeking a Dotnet Technical Lead with 8-12 years of experience to lead a team of .NET and React developers. The role involves designing scalable enterprise applications, developing REST APIs and microservices,
Treasury Testing
Dubai, UAE
Virtusa is looking for a Manual Test Engineer with automation knowledge and 5+ years of banking/payments testing experience. The role requires strong knowledge of SWIFT MT/MX messages, ISO 20022, and experience with paym
Treasury Testing
Dubai, UAE
Virtusa is looking for a Manual Test Engineer with automation knowledge for a Treasury Testing role in Dubai. The role requires 5+ years of banking/payments testing experience, strong knowledge of SWIFT MT/MX messages an
Java Developer
Doha, QAT
Virtusa is looking for a Java Developer with expertise in Agile, Web Services, Java, Spring Boot, and Microservices. The role involves backend development using Java 8/11, Spring Boot, REST APIs, and cloud platforms. Nic
Java Developer
Doha, QAT
Virtusa is looking for a Java Developer with strong backend development skills in Java, Spring Boot, and Microservices. The role requires experience with REST APIs, cloud platforms, and AI integration. Candidates should
Business Analyst - LOS
Dubai, UAE
GTB - Technology Delivery Manager
Dubai, UAE
Infrastructure Engineer
Doha, QAT
Dotnet Technical Lead
Doha, QAT
Treasury Testing
Dubai, UAE
Treasury Testing
Dubai, UAE
Java Developer
Doha, QAT
Java Developer
Doha, QAT