Python, PySpark, ETL Developer

Role overview

Responsibilities

• Develop and maintain scalable batch ETL pipelines using Python and PySpark.
• Implement reusable, modular, testable transformation logic.
• Optimize Spark jobs through partitioning, caching, joins, and shuffle management.
• Apply validation checks and handle schema evolution for accurate and complete datasets.
• Troubleshoot failures, analyze logs, and implement error handling and retries.
• Monitor job runs using alerts and runbooks and resolve incidents promptly.
• Gather requirements, define data mappings, and deliver business-aligned datasets.
• Participate in code reviews and continuous improvement of engineering standards and tooling.
• Document pipeline logic, dependencies, and operational procedures.

Qualifications

• A bachelor's degree in computer science, engineering, information systems, or a related field, or equivalent practical experience.
• 2–5 years of hands-on experience building data pipelines with Python and PySpark.
• Strong understanding of ETL, data transformations, and large-scale datasets.
• Proficiency in clean, maintainable code and production debugging.
• Working knowledge of data structures, algorithms, and software development best practices.

Technical skills

• Python and PySpark for data ingestion, transformation, and loading.
• Spark performance optimization, data validation, schema evolution, and data quality practices.
• Production troubleshooting, monitoring, error handling, and retry mechanisms.