Senior Data Engineer Pyspark with data modelling
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
About The Role We are seeking a highly skilled Data Engineer with deep expertise in PySpark and the Cloudera Data Platform (CDP) to join our data engineering team.
Key Skills for This Role
Full Job Posting
About The Role
We are seeking a highly skilled Data Engineer with deep expertise in PySpark and the Cloudera Data Platform (CDP) to join our data engineering team.
As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines that ensure high data quality and availability across the organization.
This role requires a strong background in big data ecosystems, cloud-native tools, and advanced data processing techniques.
The ideal candidate has hands-on experience with data ingestion, transformation, and optimization on the Cloudera Data Platform, along with a proven track record of implementing data engineering best practices.
You will work closely with other data engineers to build solutions that drive impactful business insights.
Responsibilities
- Data Pipeline Development: Design, develop, and maintain highly scalable and optimized ETL pipelines using PySpark on the Cloudera Data Platform, ensuring data integrity and accuracy.
- Data Ingestion: Implement and manage data ingestion processes from a variety of sources (e.g., relational databases, APIs, file systems) to the data lake or data warehouse on CDP.
- Data Transformation and Processing: Use PySpark to process, cleanse, and transform large datasets into meaningful formats that support analytical needs and business requirements.
- Performance Optimization: Conduct performance tuning of PySpark code and Cloudera components, optimizing resource utilization and reducing runtime of ETL processes.
- Data Quality and Validation: Implement data quality checks, monitoring, and validation routines to ensure data accuracy and reliability throughout the pipeline.
- Automation and Orchestration: Automate data workflows using tools like Apache Oozie, Airflow, or similar orchestration tools within the Cloudera ecosystem.
- Monitoring and Maintenance: Monitor pipeline performance, troubleshoot issue.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Virtusa
Business Analyst - LOS
Dubai, UAE
Virtusa is seeking a Business Analyst with expertise in Loan Origination Systems (LOS) to engage with retail lending teams and define future-state origination journeys. The role requires 8-10 years of banking experience
GTB - Technology Delivery Manager
Dubai, UAE
Virtusa seeks a GTB Technology Delivery Manager to own end-to-end technology delivery for GTB applications including Payments, Cash Management, Trade Finance, and Supply Chain Finance as part of a Core Banking transforma
Infrastructure Engineer
Doha, QAT
Virtusa seeks an Infrastructure Engineer specializing in Nutanix AHV-based platforms. The role involves migrating server workloads using lift and shift and parallel build strategies, managing Nutanix HCI, and supporting
Dotnet Technical Lead
Doha, QAT
Virtusa is seeking a Dotnet Technical Lead with 8-12 years of experience to lead a team of .NET and React developers. The role involves designing scalable enterprise applications, developing REST APIs and microservices,
Treasury Testing
Dubai, UAE
Virtusa is looking for a Manual Test Engineer with automation knowledge and 5+ years of banking/payments testing experience. The role requires strong knowledge of SWIFT MT/MX messages, ISO 20022, and experience with paym
Treasury Testing
Dubai, UAE
Virtusa is looking for a Manual Test Engineer with automation knowledge for a Treasury Testing role in Dubai. The role requires 5+ years of banking/payments testing experience, strong knowledge of SWIFT MT/MX messages an
Java Developer
Doha, QAT
Virtusa is looking for a Java Developer with expertise in Agile, Web Services, Java, Spring Boot, and Microservices. The role involves backend development using Java 8/11, Spring Boot, REST APIs, and cloud platforms. Nic
Java Developer
Doha, QAT
Virtusa is looking for a Java Developer with strong backend development skills in Java, Spring Boot, and Microservices. The role requires experience with REST APIs, cloud platforms, and AI integration. Candidates should
Business Analyst - LOS
Dubai, UAE
GTB - Technology Delivery Manager
Dubai, UAE
Infrastructure Engineer
Doha, QAT
Dotnet Technical Lead
Doha, QAT
Treasury Testing
Dubai, UAE
Treasury Testing
Dubai, UAE
Java Developer
Doha, QAT
Java Developer
Doha, QAT