{bc}
Apply
linkedin

Pyspark Developer(Python,Airflow,Cloudera is mandate) - 8+ YOE - Onsite - Dubai and Chennai - Immediate - 20 Days Joiners

ValueLabs
Dubai, UAE
Fulltime
Mid-Senior
2 months ago
PythonScala
Free

Job Fit Check

Base Career helps you apply smarter for this job.

?%
Ready to Scan

Key skills for this role

PythonScala
Smart Apply

Full Job Posting

Education And Experience

  • Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, or a related field.
  • 8+ years of experience as a Data Engineer, with a strong focus on PySpark and the Cloudera Data Platform.

Pyspark Job Description

  • Data Pipeline Development: Design, develop, and maintain highly scalable and optimized ETL pipelines using PySpark on the Cloudera Data Platform, ensuring data integrity and accuracy.
  • Data Ingestion: Implement and manage data ingestion processes from a variety of sources (e.g., relational databases, APIs, file systems) to the data lake or data warehouse on CDP.
  • Data Transformation and Processing: Use PySpark to process, cleanse, and transform large datasets into meaningful formats that support analytical needs and business requirements.
  • Performance Optimization: Conduct performance tuning of PySpark code and Cloudera components, optimizing resource utilization and reducing runtime of ETL processes.
  • Data Quality and Validation: Implement data quality checks, monitoring, and validation routines to ensure data accuracy and reliability throughout the pipeline.
  • Automation and Orchestration: Automate data workflows using tools like Apache Oozie, Airflow, or similar orchestration tools within the Cloudera ecosystem.
  • Monitoring and Maintenance: Monitor pipeline performance, troubleshoot issues, and perform routine maintenance on the Cloudera Data Platform and associated data processes.
  • Collaboration: Work closely with other data engineers, analysts, product managers, and other stakeholders to understand data requirements and support various data-driven initiatives.
  • Documentation: Maintain thorough documentation of data engineering processes, code, and pipeline configurations.

Technical Skills

  • PySpark: Advanced proficiency in PySpark, including working with RDDs, DataFrames, and optimization techniques.
  • Cloudera Data Platform: Strong experience with Cloudera Data Platform (CDP) components, including Cloudera Manager, Hive, Impala, HDFS, and HBase.
  • Data Warehousing: Knowledge of data warehousing concepts, ETL best practices, and experience with SQL-based tools (e.g., Hive, Impala).
  • Big Data Technologies: Familiarity with Hadoop, Kafka, and other distributed computing tools.
  • Orchestration and Scheduling: Experience with Apache Oozie, Airflow, or similar orchestration frameworks.
  • Scripting and Automation: Strong scripting skills in Linux.

Apply for this job in 1 click

Skip the repetitive application forms

Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.

Sarah M.James T.Maya R.

Trusted by over 500,000 job seekers on Base Career

Start Free Today

More from this employer

More jobs at ValueLabs

Retail Banking -Business Analyst

Dubai, UAE

TodayFull Time

Advent Dotnet Engineer

Dubai, UAE

1 weeks agoFull Time

DevOps Engineer

Dubai, UAE

1 weeks agoFull Time

Advent + .Net Developer (Advent APX Domain is mandate ) - 5+ YOE - Onsite - Dubai - Immediate Joiners Preferable

Dubai, UAE

2 weeks agoFull Time

Finacle Test Lead

Sharjah Emirate, UAE

2 weeks agoFull Time

Python Developer

Dubai, UAE

2 weeks agoFull Time

Senior Finacle Automation Engineer

Dubai, UAE

3 weeks agoFull Time

Finacle Automation Testing

Dubai, UAE

3 weeks agoFull Time