Data Engineering (TPA)

Role Purpose

Enterprise Data Ingestion

• Build reusable, parameterized Informatica IDMC mappings and task flows for standardized ingestion.
• Implement change data capture, idempotent loads, schema evolution, and data-quality gates.
• Optimize BigQuery loads using partitioning, clustering, and load-versus-stream decisions.
• Set up version control and CI/CD pipelines for data engineering assets.

Mapping and Transformation

• Profile source systems and define field-level mappings and transformation rules.
• Specify joins, lookups, derivations, data-quality rules, exception handling, and reject criteria.

Orchestration and Reliability

• Parameterize task flows and configure schedules and dependencies.
• Implement retries, backoff, and checkpointing for reliable processing.
• Integrate monitoring and alerting through the operations console and ChatOps.

Privacy and Security

• Define least-privilege IAM roles and service accounts for data access.
• Apply dataset, table, row, and column-level security and data masking.
• Enable audit logging and retention policies and classify PHI and PII data.

Architecture and Modelling

• Implement Medallion Architecture across Bronze, Silver, and Gold layers.
• Develop scalable curated data models for analytical and reporting needs.
• Define data-quality, lineage, and governance standards for curated data.
• Collaborate with business and analytics teams to create trusted datasets.

Monitoring and Optimization

• Track data freshness, job health, volumes, and anomalies.
• Monitor SLAs for job duration, errors, cost per terabyte, and slot usage.
• Tune BigQuery performance and optimize costs through storage lifecycle management, query tuning, and caching.

Technical Skills

• Data engineering tools include Informatica IDMC, Airflow, and Apache Spark.
• Programming languages include SQL and Python.
• Cloud technologies include Google Cloud Platform and BigQuery; Azure and AWS are optional.
• BI platforms include Power BI, Looker, Looker Studio, and Tableau.
• The role requires BigQuery, Informatica IDMC or IICS, enterprise data pipelines, Medallion Architecture, data marts, and BI datasets.
• Healthcare data knowledge and cloud security understanding are required.

Education

• Bachelor’s degree in Computer Science, Information Technology, Information Systems, or a related field.