Senior Big Data Engineer
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
**Responsibilities:** * Design, implement, and optimize data pipelines for batch and real\-time data processing using Cloudera (Hadoop, Hive, Spark, Impala) and Informatica (PowerCenter, Cloud Data Integration) * Build data extraction, transformation, and loading (ETL) workflows using Informatica PowerCenter for large\-scale data integration from source systems (e.g., relational databases, flat files, APIs) into Cloudera Data Lake or data warehouse environments. * Implemen
Key Skills for This Role
Full Job Posting
Responsibilities
- Design, implement, and optimize data pipelines for batch and real-time data processing using Cloudera (Hadoop, Hive, Spark, Impala) and Informatica (PowerCenter, Cloud Data Integration)
- Build data extraction, transformation, and loading (ETL) workflows using Informatica PowerCenter for large-scale data integration from source systems (e.g., relational databases, flat files, APIs) into Cloudera Data Lake or data warehouse environments.
- Implement Spark jobs on Cloudera for distributed data processing and optimization of data workflows.
- Leverage Informatica for orchestrating ETL workflows, including data extraction, cleansing, transformation, and loading into data repositories (HDFS, Hive, SQL databases, etc.).
- Optimize the Informatica workflows to minimize runtime, ensure smooth data integration, and maintain high data quality.
- Utilize Hadoop and Spark on Cloudera to process large datasets and implement data transformations using MapReduce, Spark SQL, and PySpark.
- Leverage Impala for low-latency SQL queries on Hadoop, ensuring real-time access to processed data.
- Implement partitioning, bucketing, and indexing strategies in Hive and HBase to improve query performance on large datasets.
- Implement and enforce data quality rules within Informatica workflows, ensuring that all transformations meet the required standards for completeness, consistency, and accuracy.
- Ensure compliance with data governance and security protocols (e.g., encryption, masking, access control) in accordance with industry best practices.
- Automation and Scheduling: Automate ETL workflows using Informatica Server, integrating with Airflow, Nifi or other workflow orchestration tools for scheduling and monitoring jobs.
- Utilize Cloudera Navigator for monitoring and auditing data processes within the Hadoop ecosystem.
- Perform regular tuning of the ETL pipelines, data flows, and SQL queries to ensure optimal performance.
Qualifications
- Bachelor’s degree in Computer Science, Engineering, or related field.
- 6+ years of experience in the same field.
- Proven experience with the Cloudera Distribution of Hadoop (CDH), including expertise in HDFS, Hive, Impala, Spark, and HBase.
- Strong hands-on experience with Informatica PowerCenter (ETL), EDC, IDQ, B2B, and Axon.
- Deep understanding of ETL best practices, data pipelines, and distributed computing technologies such as Spark, MapReduce, PySpark, and Hadoop ecosystem components.
- Advanced SQL skills for data manipulation, aggregation, optimization, and reporting across relational and non-relational data stores (e.g., SQL Server, MySQL, PostgreSQL, Hive, Impala).
- Experience in Python and SQL.
- Strong background in data warehousing principles and data modeling, including dimensional modeling (star schema, snowflake schema) and OLAP/OLTP considerations.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at BBI
Project Manager
الرياض, KSA
BBI is seeking an experienced Project Manager to lead AI-driven initiatives. The role requires 5+ years of project management experience with Agile and Scrum methodologies. The position is remote and involves managing sc
Data Strategy Consultant
Dubai, UAE
BBI is seeking a Senior Data Strategy Consultant to lead data strategy, governance, and analytics projects for clients. Requires 3+ years of experience in data management, DAMA or NDMO knowledge, and bilingual communicat
Senior Data Modeler
الرياض, KSA
Responsibilities: Develop and maintain conceptual, logical, and physical data models using data modeling tools. Collaborate with stakeholders, business analysts, and Data Engineer teams to gather business requirements
Senior Data Engineer
الرياض, KSA
Design, develop, and maintain scalable data pipelines for the efficient collection, storage, and processing of large datasets. Develops and design best practices around data collection and maintain orchestration of data
Oracle ADF/Application Support Engineer
Doha, QAT
Qualifications: eSTR software development and operational support. Minor development changes (Oracle ADF forms, APIs, etc.) Break-fixes and certificate renewals. Onsite Oracle ADF expert (permanent Qatar resident req
Sales - Account Lead
الرياض, KSA
Overview A strategic and results-driven sales professional responsible for leading business growth within a defined portfolio of named accounts. The Account Lead owns the end-to-end sales cycle, develops long-term client