PySpark Developer

PySpark Developer

Key Skills

• Strong experience in PySpark, Apache Spark, Python, and SQL .
• Expertise in building and optimizing ETL/Data Processing Pipelines .
• Good understanding of Spark Architecture (RDDs, DataFrames, Spark SQL, DAGs).
• Experience with Hadoop Ecosystem (Hive, HDFS) and large-scale data processing.
• Exposure to AWS/Azure/GCP , object storage, and cloud-based data solutions.
• Knowledge of Data Modeling, Data Engineering, and Pipeline Design .
• Hands-on experience with Git and Linux/Unix environments.
• Strong debugging, performance tuning, and problem-solving skills.

Responsibilities

• Design, develop, and maintain scalable PySpark-based data pipelines .
• Build ETL workflows for ingesting, transforming, and loading large datasets.
• Optimize Spark jobs for performance, scalability, and reliability.
• Process data from multiple sources, including databases, APIs, HDFS, and cloud storage.
• Ensure data quality, validation, and governance standards.
• Troubleshoot and tune Spark applications for maximum efficiency.
• Collaborate with data engineers, analysts, and data scientists to deliver business solutions.
• Support production deployments and ongoing application maintenance.
• Follow security and best practices for enterprise data engineering.

Good to Have