Senior Associate | Data Engineering | Bengaluru | Engineering as a Service/ Operate (Bengaluru, IN)

We are seeking a Sr Data Engineer I to support the reliability, scalability, and performance of

Disney’s Ad Technology big data platforms and pipelines.

This role focuses on hands-on

development and operations for Databricks, Spark, Flink, and Airflow (MWAA) environments,

ensuring that production data workflows are resilient, cost-efficient, and performant at scale.

You will be responsible for building, optimizing, and supporting data pipelines, while

contributing to automation, monitoring, and incident response.

This role will work closely with

platform, BI, and analytics teams to ensure data is delivered accurately and on time.

As a senior

IC, you’ll also mentor junior engineers, contribute to runbooks, and help evolve best practices

under the guidance of a Lead Engineer and Manager.

• Design, maintain, and optimize Spark and Flink pipelines running in Databricks and

Airflow.

• Ensure reliability and availability of data workflows in production.
• Debug and resolve issues in notebooks, jobs, and orchestration workflows.

• Tune Spark jobs, clusters, and data pipelines for performance and cost efficiency.
• Analyze logs, queries, and workloads to identify bottlenecks and implement

improvements.

• Build and maintain dashboards, alerts, and metrics with Datadog, CloudWatch, or

Prometheus.

• Proactively monitor pipeline health and escalate issues as needed.

• Participate in incident response for pipeline issues, helping diagnose and resolve

outages.

• Contribute to RCA and preventative actions to improve long-term stability.

• Partner with data engineering, BI, and platform teams to support delivery and

operations.

• Document best practices, contribute to runbooks, and mentor junior engineers.

• 4+ years of experience in data engineering or big data platform operations.
• Hands-on experience with Databricks, Spark, Flink, or Airflow (MWAA) in production.
• Strong SQL skills and experience with ETL/ELT pipeline development.
• Familiarity with AWS services (S3, Glue, Athena, EMR, EKS) for pipeline management.
• Working knowledge of observability tools (Datadog, CloudWatch, Prometheus).
• Proficiency in Python, Scala, or Java for pipeline development and automation.
• Familiarity with infrastructure-as-code (Terraform or similar).
• Strong collaboration and problem-solving skills, with experience in distributed teams.

• Experience with incident response and RCA in production data environments.
• Familiarity with CI/CD pipelines (GitLab, Jenkins, or GitHub Actions).
• Exposure to data governance, access control, and cost optimization practices.
• Certifications such as Databricks Data Engineer Associate or AWS Solutions Architect –

Associate.

• Bachelor’s degree in Computer Science, Engineering, or related technical field (or

equivalent experience