Sr. Data Engineer - Data Engineering 4B

Key Responsibilities

Design, develop, and maintain high-performance real-time data processing applications using Apache Flink.

Develop in-flight transformation logic leveraging Apache Flink, Kafka Streams, KSQLDB, and Kafka Single Message Transforms (SMTs).

Build and maintain scalable mapping frameworks for transforming source system data into harmonized enterprise data models.

Implement complex event processing, enrichment, filtering, aggregation, and windowing logic for streaming workloads.

Optimize streaming pipeline performance, ensuring low latency, high throughput, and efficient resource utilization.

Design solutions to handle late-arriving and out-of-order events using event-time processing, watermarks, checkpointing, and state management.

Integrate streaming applications with Kafka, Schema Registry, APIs, and downstream analytical platforms.

Implement robust error handling, retry mechanisms, dead-letter queues (DLQ), and monitoring for streaming applications.

Collaborate with Solution Architects, Data Architects, API teams, and business stakeholders to define transformation rules and streaming integration patterns.

Develop automated testing frameworks and CI/CD pipelines for streaming applications.

Monitor production streaming jobs and troubleshoot performance, scalability, and reliability issues.

Required Skills & Experience

6–10+ years of experience in Data Engineering with strong expertise in real-time streaming solutions.

Hands-on experience with Apache Flink for enterprise-scale stream processing.

Strong knowledge of Apache Kafka, Kafka Streams, KSQLDB, Kafka Connect, and Single Message Transforms (SMTs).

Experience designing event-driven data architectures and real-time data integration pipelines.

Strong understanding of event-time processing, watermarks, windowing, checkpointing, stateful stream processing, and fault tolerance.