Data Engineer Consultant
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
The Job Description 1- Enterprise Data Ingestion and Data Engineering: Design and build scalable, reusable ingestion pipelines (realtime and batch) on GCP (GCS, BigQuery, Datafl.
Key Skills for This Role
Responsibilities
- Design and build scalable, reusable ingestion pipelines (real time and batch) on GCP (GCS, BigQuery, Dataflow)
- Develop parameterized pipelines using Google native services and/or Informatica IDMC mappings and taskflows
- Implement CDC patterns, idempotent loads, late arriving data handling, and schema evolution
- Optimize BigQuery ingestion strategies (batch vs. streaming, partitioning, clustering)
- Establish version control, CI/CD, and environment promotion (dev/test/prod)
- Profile source data and analyze data quality and patterns
- Design field level mappings, business rules, joins, aggregations, and derivations
- Implement transformations using BigQuery SQL, Dataflow, or IDMC transformation logic
- Build and parameterize end to end workflows using Cloud Composer (Airflow) and/or IDMC taskflows
- Define job dependencies, schedules, SLAs, and failure handling strategies
- Implement retries, backoff strategies, checkpoints, and restartability
- Integrate monitoring, logging, and alerting using Cloud Monitoring, Cloud Logging, and ChatOps tools
Requirements
- 5+ years of experience as a Data Engineer or ETL Developer
- Strong experience with Google Cloud Platform: BigQuery, GCS, Data Stream, Dataflow, Cloud Composer, IAM, Cloud Monitoring
- Proficiency in SQL (BigQuery optimized) and data modeling for analytics
- Hands on experience building production grade, scalable data pipelines
- Experience with CI/CD, Git based version control, and DevOps practices
- Bachelor's degree in Computer Science, Information Technology, Information Systems, or a related field
- Experience with Informatica Intelligent Data Management Cloud (IDMC) CDI, Mass Ingestion, Taskflows (optional)
Full Job Posting
Enterprise Data Ingestion and Data Engineering
- Design and build scalable, reusable ingestion pipelines (realtime and batch) on GCP (GCS, BigQuery, Dataflow).
- Develop parameterized pipelines using Google native services and/or Informatica IDMC mappings and taskflows.
- Implement CDC patterns, idempotent loads, late arriving data handling, and schema evolution.
- Optimize BigQuery ingestion strategies (batch vs. streaming, partitioning, clustering).
- Establish version control, CI/CD, and environment promotion (dev/test/prod).
- Results: GCP native pipelines (Dataflow, Composer, Cloud Run) and/or IDMC mappings/taskflows; BigQuery schema designs (raw / stage / curated zones with partitioning & clustering); CI/CD pipelines, deployment artifacts, and operational runbooks.
Data Mapping & Transformation Design
- Profile source data and analyze data quality and patterns.
- Design field level mappings, business rules, joins, aggregations, and derivations.
- Implement transformations using BigQuery SQL, Dataflow, or IDMC transformation logic.
- Results: Source to Target Mapping (STM) document; Transformation mappings.
Orchestration, Automation & Reliability Engineering
- Build and parameterize end to end workflows using Cloud Composer (Airflow) and/or IDMC taskflows.
- Define job dependencies, schedules, SLAs, and failure handling strategies.
- Implement retries, backoff strategies, checkpoints, and restartability.
- Integrate monitoring, logging, and alerting using Cloud Monitoring, Cloud Logging, and ChatOps tools.
- Results: Production ready orchestration workflows with environment aware parameters; Scheduling calendar, dependency diagrams, and SLA matrix; Alerting, monitoring dashboards, and operational SOPs.
Data Pipelines Monitoring, Performance & Cost Optimization
- Monitor pipeline health, data freshness, volumes, and anomaly patterns.
- Support and monitor data pipelines during off hours and weekends, troubleshooting and resolving issues to ensure SLA compliance.
- Track and manage SLAs related to runtime, failures, cost, and data latency.
- Optimize BigQuery performance (query refactoring, partition pruning, materialized views).
- Manage and optimize GCP costs (storage lifecycle, slot usage, query optimization, caching).
- Results: Daily/weekly operational and SLA reports; Performance and cost optimization plans with before/after metrics; BigQuery tuning guidelines and data materialization strategy.
Requirements
- 5+ years of experience as a Data Engineer or ETL Developer.
- Strong experience with Google Cloud Platform: BigQuery, GCS, Data Stream, Dataflow, Cloud Composer, IAM, Cloud Monitoring.
- Proficiency in SQL (BigQuery optimized) and data modeling for analytics.
- Hands on experience building production grade, scalable data pipelines.
- Experience with CI/CD, Git based version control, and DevOps practices.
- [Optional] Experience with Informatica Intelligent Data Management Cloud (IDMC) CDI, Mass Ingestion, Taskflows.
- Bachelor's degree in Computer Science, Information Technology, Information Systems, or a related.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Total-TECH Co
Ecommerce CX Excellence Manager
Jeddah, KSA
The Job Description 1. Customer Journey & Experience Governance: Define and maintain the customer journey framework across digital sales and support (exploration, decision, onbo...
Creative Copywriter
Jeddah, KSA
Develop compelling content, manage brand copy library, ensure alignment with brand guidelines, and collaborate on cross-functional projects with strong copywriting skills.
Creative Copywriter
Jeddah, KSA
Total-TECH Co in Jeddah seeks a Creative Copywriter to craft impactful, on-brand copy across all touchpoints. You will interpret creative briefs, develop concepts, and maintain brand voice. Requires 2-4 years of copywrit
Project Manager HIS
Jeddah, KSA
Lead healthcare information system implementation, manage stakeholders and vendors, ensure system integration, and oversee project governance and quality assurance.
BI Engineer Consultant
Jeddah, KSA
Design and maintain LookML models, develop dashboards, enable self-service BI, optimize SQL queries, and monitor performance with strong SQL and Looker expertise.
Sr. Executive Manager
Jeddah, KSA
Design and enhance secure, scalable Health Information System architecture, manage development activities, ensure interoperability, and deliver innovative solutions.
HIS Product Manager
Jeddah, KSA
Lead Health Information Systems product strategy, manage stakeholder communication, optimize performance, and ensure user-centric design in healthcare technology.
Oracle R12 EBS Finance & SCM Technical
Jeddah, KSA
Seeking an experienced Oracle R12 EBS Technical Consultant with strong SQL, PLSQL, and integration skills to implement finance and SCM modules. Requires expertise in AP, AR, GL, FA, Inventory, Purchasing, and custom deve
Ecommerce CX Excellence Manager
Jeddah, KSA
Creative Copywriter
Jeddah, KSA
Creative Copywriter
Jeddah, KSA
Project Manager HIS
Jeddah, KSA
BI Engineer Consultant
Jeddah, KSA
Sr. Executive Manager
Jeddah, KSA
HIS Product Manager
Jeddah, KSA
Oracle R12 EBS Finance & SCM Technical
Jeddah, KSA