Senior Data Engineer

About Air Products

Location and Work Model

• The role is based in Pune and follows a hybrid work model.
• The posting states an experience level of 8+ years.

Core Technologies

• Core technologies include Databricks, Apache Spark, Delta Lake, and Azure or AWS.
• The role uses PySpark, Python, SQL, Terraform, Databricks Jobs, Workflows, and Structured Streaming.

About the Role

Key Responsibilities

• Gather requirements and design, develop, and optimize Databricks and Apache Spark data pipelines.
• Build and manage Delta Lake Medallion architectures.
• Develop batch and streaming pipelines with Structured Streaming.
• Optimize Spark workloads for performance and cost efficiency.
• Implement Databricks Jobs, Workflows, and SQL.
• Ingest data from relational databases, SAP, APIs, and other sources into Databricks.
• Contribute to AWS Glue and Redshift migration initiatives into Databricks.
• Implement data quality, security, and governance best practices.
• Collaborate with cross-functional teams and mentor junior engineers.

Required Skills and Experience

• 8+ years of experience in data engineering.
• Strong hands-on Databricks experience.
• Advanced Apache Spark knowledge, with PySpark preferred.
• Experience with Terraform, Delta Lake, SQL, and Python.
• Understanding of cloud-native data architectures on Databricks.
• Experience building scalable, production-grade data pipelines.
• Strong problem-solving and communication skills.

Nice to Have

• Knowledge and experience with security and governance topics.
• Experience with Unity Catalog and data governance.
• Familiarity with CI/CD pipelines and DevOps practices for data solutions.
• Familiarity with Terraform, dbt, or Airflow.
• Databricks certification, which is preferred but not mandatory.