Data Engineer II - Databricks, Python, PySpark, AWS

Role overview

Job responsibilities

• Organize, update, and maintain gathered data to make it actionable.
• Determine controls needed for secure data access using knowledge of data system components.
• Use authorized AI capabilities for data analysis support and technical documentation, validating outputs against security requirements.
• Improve data quality checks and model or change validation routines using reuse-first, AI-assisted approaches.
• Make custom configuration changes in one or two tools to generate requested products.
• Update logical or physical data models for new use cases with minimal supervision.
• Contribute to a diverse, inclusive, respectful team culture.

Required qualifications

• Formal training or certification in data engineering concepts and at least 2 years of applied experience.
• Knowledge of the data lifecycle, data management, data frameworks, and data lakes.
• Experience with batch and real-time processing using Spark or Flink.
• Experience with relational and NoSQL databases, ETL pipelines, and data warehousing.
• Experience with Databricks, Python or Java, and PySpark.
• Working knowledge of AWS Glue, EMR, Spring Boot or Flask, and AWS EKS or Kubernetes.
• Experience with statistical data analysis and selecting appropriate analysis tools.

AI and data handling

• Use authorized AI capabilities while following data sensitivity and security requirements.
• Review and validate AI-assisted outputs before use and escalate when uncertain.

Preferred qualifications

• Expertise in AWS, Docker, and Kubernetes for cloud-native and containerized data solutions is preferred.
• Experience with Hadoop, Spark, Kafka, or distributed system design is preferred.