Python-Pyspark Developer

Primary Skills

• Python is listed as a primary skill.
• PySpark is listed as a primary skill.

Role Overview

Data Engineering and Development

• Develop and maintain data pipelines using Python and PySpark.
• Process and transform large datasets in distributed environments.
• Build scalable ETL and ELT workflows.

Big Data Processing

• Use Apache Spark for batch and real-time processing.
• Optimize Spark jobs for performance and efficiency.
• Handle structured and unstructured data.

Data Integration

• Ingest data from databases, APIs, and CSV, JSON, and Parquet files.
• Integrate data with Hadoop HDFS and AWS, Azure, or GCP platforms.

Performance Optimization

• Tune Spark jobs through partitioning, caching, and parallelism.
• Optimize SQL queries and transformations to improve efficiency and reduce cost.

Collaboration and Support

• Work with data engineers, data scientists, and analysts.
• Translate business requirements into technical solutions.
• Participate in code reviews and agile development practices.

Monitoring and Troubleshooting

• Debug and resolve data pipeline issues.
• Monitor job execution and data quality.
• Ensure the reliability and availability of data workflows.

Requirements

• Strong Python and PySpark skills are required.
• Experience designing, developing, and optimizing large-scale data processing systems is expected.
• Experience building scalable ETL or ELT workflows is expected.
• Knowledge of distributed data processing with Apache Spark is expected.
• Ability to work with structured and unstructured data is expected.
• Ability to integrate data from databases, APIs, and files is expected.
• Ability to collaborate with data engineers, data scientists, and analysts is expected.