Kafka Engineer
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
DICETEK LLC is seeking a highly skilled Confluent Kafka Engineer for development, UAT, and production support of Kafka-based systems. The role requires deep expertise in Kafka architecture including Confluent Control Center, KSQLDB, Kafka Streams, and Kafka Connect.
Key Skills for This Role
Responsibilities
- Design, develop, and implement Kafka based solutions using Confluent Control Center, KSQLDB, Kafka Streams, and Kafka Connect
- Write and maintain high quality code for Kafka producers, consumers, and stream processing applications
- Develop and manage Kafka connectors for seamless integration with external systems
- Utilize Kafka Streams for real time processing of streaming data
- Employ KSQLDB for stream processing tasks, including real time analytics and transformations
- Collaborate with data engineers, software developers, and DevOps teams to integrate Kafka solutions
- Develop and execute UAT plans to validate Kafka solutions
- Monitor and maintain Kafka clusters to ensure high availability and reliability
- Troubleshoot and resolve issues related to Kafka performance, latency, and data integrity
- Provide 24/7 support for production systems, including on call rotation
Requirements
- Bachelor's degree in Computer Science, Information Technology, or a related field
- 3+ years of hands on experience with Apache Kafka and Confluent Kafka in a production environment, including Confluent Control Center, KSQLDB, Kafka Streams, and Kafka Connect
- Proven experience in Kafka development, including producer and consumer API, stream processing, and connector development
- Experience with Kafka cluster management, including setup, configuration, monitoring, and troubleshooting
- Proficiency in programming languages such as Java, Python, or Scala
- Strong knowledge of Kafka internals, including brokers, zookeepers, topics, partitions, and offsets
- Experience with monitoring tools (e.g., Prometheus, Grafana) and logging frameworks (e.g., Log4j, ELK Stack)
- Familiarity with distributed systems, microservices architecture, and event driven design patterns
- Experience with cloud platforms (e.g., AWS, Azure) and containerization (Kubernetes) is a plus
Full Job Posting
Job Summary
- We are seeking a highly skilled Confluent Kafka Engineer to join our dynamic team.
- The ideal candidate will be responsible for the development, User Acceptance Testing (UAT), and production support of our Confluent Kafka based systems.
- This role requires deep expertise in Kafka architecture, including Confluent Control Center, KSQLDB, Kafka Streams, and Kafka Connect.
Key Responsibilities Development
- Design, develop, and implement Kafka based solutions to meet business requirements, utilizing Confluent Control Center, KSQLDB, Kafka Streams, and Kafka Connect.
- Write and maintain high quality code for Kafka producers, consumers, and stream processing applications.
- Develop and manage Kafka connectors for seamless integration with external systems, ensuring data consistency and reliability.
- Utilize Kafka Streams for real time processing of streaming data, transforming and enriching data as it flows through the pipeline.
- Employ KSQLDB for stream processing tasks, including real time analytics and transformations.
- Collaborate with data engineers, software developers, and DevOps teams to integrate Kafka solutions with existing systems.
- Ensure all Kafka based solutions are scalable, secure, and optimized for performance.
Key Responsibilities User Acceptance Testing (UAT)
- Develop and execute UAT plans to validate Kafka solutions, including components such as Kafka Streams, KSQLDB, and Kafka Connect, before deployment to production.
- Work closely with QA teams to identify and resolve defects during the UAT phase.
- Ensure all UAT activities comply with the organization’s standards and best practices.
- Provide detailed reports on UAT outcomes and work with developers to implement necessary fixes.
Key Responsibilities Production Support
- Monitor and maintain Kafka clusters, including components like Confluent Control Center, to ensure high availability and reliability of data streaming services.
- Troubleshoot and resolve issues related to Kafka performance, latency, and data integrity, including issues specific to Kafka Streams, KSQLDB, and Kafka Connect.
- Perform routine maintenance tasks such as patching, upgrades, and backups for Kafka clusters and associated components.
- Implement monitoring solutions to proactively identify and mitigate potential production issues, leveraging Confluent Control Center for comprehensive cluster visibility.
- Provide 24/7 support for production systems, including participation in on call rotation.
Qualifications Education
- Bachelor’s degree in Computer Science, Information Technology, or a related field.
Qualifications Experience
- 3+ years of hands on experience with Apache Kafka and Confluent Kafka in a production environment, including experience with Confluent Control Center, KSQLDB, Kafka Streams, and Kafka Connect.
- Proven experience in Kafka development, including producer and consumer API, stream processing, and connector development.
- Experience with Kafka cluster management, including setup, configuration, monitoring, and troubleshooting.
- Familiarity with distributed systems, microservices architecture, and event driven design patterns.
- Experience with cloud platforms (e.g., AWS, Azure) and containerization (Kubernetes) is a plus.
Technical Skills
- Proficiency in programming languages such as Java, Python, or Scala.
- Strong knowledge of Kafka internals, including brokers, zookeepers, topics, partitions, and offsets.
- Experience with monitoring tools (e.g., Prometheus, Grafana) and logging frameworks (e.g., Log4j, ELK Stack).
- Proficiency in using Confluent Control Center for monitoring, managing, and optimizing Kafka clusters.
- Expertise in Kafka Streams for building scalable, fault tolerant stream processing applications.
- Experience with KSQLDB for real time processing and analytics on Kafka topics.
- Strong understanding of Kafka Connect for integrating Kafka with external data sources and sinks.
- Understanding of networking, security, and compliance aspects related to Kafka.
- Familiarity with CI/CD pipelines and automation tools (e.g., Jenkins, GitLab CI).
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at DICETEK LLC
Senior PLSQL Developer
Abu Dhabi, UAE
DICETEK LLC seeks a Senior PLSQL Developer in Abu Dhabi for contract work. The role requires strong Oracle Database and PL/SQL expertise, advanced SQL optimization, ETL development, and experience in regulatory reporting
DMF Delivery Manager – Data Management
Abu Dhabi, UAE
DICETEK LLC seeks a DMF Delivery Manager for Data Management in Abu Dhabi. The role requires 8-12 years of experience in data project delivery, technology transformation, and data governance/platform implementation. Hand
NOC Officer
Dubai, UAE
DICETEK LLC seeks a proactive Monitoring & Support Engineer to manage alerts, incidents, and basic server/network checks. The role involves infrastructure monitoring, incident response, documentation, and basic Windows/L
Data Operations Consultant
Abu Dhabi, UAE
Dicetek LLC seeks a Data Operations Consultant for a contract role in Abu Dhabi. Requires 5-10 years experience in data governance, data management, or data quality within banking, financial services, or government. The
Data Governance-GDO Strategy
Abu Dhabi, UAE
DICETEK LLC seeks a Data Governance Specialist for GDO Strategy in Abu Dhabi. The role focuses on risk and financial reporting governance, data lineage, data quality management, and stakeholder engagement within banking.
Project Lead – Data Management Framework
Abu Dhabi, UAE
DICETEK LLC seeks a Project Lead for Data Management Framework in Abu Dhabi. The role requires 8-12 years of experience in data project delivery, technology transformation, and data governance/platform implementation. Ha
System Analyst
Abu Dhabi, UAE
DICETEK LLC is seeking a System Analyst with AML domain knowledge for a contract role in Abu Dhabi. The role involves requirements gathering, data analysis, SQL, testing (SIT, UAT), defect management, and data reconcilia
Data Governance Specialist
Abu Dhabi, UAE
DICETEK LLC seeks a Data Governance Specialist with 8-11 years of experience in data governance, data management, or data quality. The role requires expert-level hands-on experience with Axon, data lineage, and data qual
Senior PLSQL Developer
Abu Dhabi, UAE
DMF Delivery Manager – Data Management
Abu Dhabi, UAE
NOC Officer
Dubai, UAE
Data Operations Consultant
Abu Dhabi, UAE
Data Governance-GDO Strategy
Abu Dhabi, UAE
Project Lead – Data Management Framework
Abu Dhabi, UAE
System Analyst
Abu Dhabi, UAE
Data Governance Specialist
Abu Dhabi, UAE