Pyspark Data Engineer

About the Unit

Title Description for Opportunity

Responsibilities

• Design, develop, and optimize large-scale data processing solutions using PySpark, Apache Spark, and Python , ensuring high-performance batch and real-time data processing.
• Build and maintain scalable data ingestion, transformation, and storage pipelines using Spark SQL, DataFrames, Kafka, and cloud-based data platforms such as AWS, Azure, or GCP.
• Develop and support distributed computing frameworks and streaming applications , leveraging Spark Streaming, multithreading, and big data technologies to process high-volume datasets efficiently.
• Collaborate with cross-functional teams to design, implement, and deploy data engineering and machine learning solutions , utilizing Python, Scala, Java, TensorFlow, PyTorch, and Scikit-learn.
• Drive best practices for code quality, performance tuning, monitoring, troubleshooting, and optimization of Spark applications , while ensuring reliable and scalable data platforms for enterprise analytics and business needs.

Must Have Skills

• PySpark: In-depth knowledge of PySpark, Apache Spark, and Python.
• Data Processing: Strong understanding of data processing concepts, including data ingestion, data transformation, and data storage.
• Cloud Experience: Experience with cloud-based data platforms, including AWS, Azure, or Google Cloud.
• Expertise in DataFrames & Spark SQL, Spark Streaming with Apache Kafka real-time data ingestion pipelines
• Very good conceptual understanding of Multithreading, distributed computing concepts of Pyspark
• Programming: Proficiency in programming languages, including Python, Java, or Scala.
• Communication: Excellent communication and collaboration skills.
• Preferred Skills:
• Apache Spark Certifications: Relevant certifications, such as Apache Spark Certification or Cloudera Certified Spark Developer.
• Big Data: Experience with big data technologies, including Hadoop, HBase, or Cassandra.
• Machine Learning: Experience with machine learning frameworks, including TensorFlow, PyTorch, or Scikit-learn.
• Data Science: Experience with data science tools, including Jupyter Notebook, Pandas, or NumPy

Walk-in Drive Locations - Bengaluru, Chennai, Hyderabad, Kolkata and Pune.

• Experience Range: 4 to 10 years
• Qualifications: Any Technical Degree(BE/B,Tech/M.tech/MCA/BCA/BSC/Diploma/Masters)

Eligibility

Documents required for interview

Solution-

Research insights

Case studies-

Leadership Video

Big Bet hiring page link