Associate - Project Manager
About the Role
Build and operate the data pipelines that feed the Entity Hub. This role lands all six in-scope sources into Fabric, implements standardization and transformation logic, and maintains the data quality checks and monitoring that the entity resolution engine depends on. Reliable, observable ingestion is the foundation the entire programme rests on.
Key Skills for This Role
Full Job Posting
Build and operate the data pipelines that feed the Entity Hub. This role lands all six in-scope sources into Fabric, implements standardization and transformation logic, and maintains the data quality checks and monitoring that the entity resolution engine depends on. Reliable, observable ingestion is the foundation the entire programme rests on.
Qualifications
- years hands-on data engineering with strong PySpark and SQL
- Production experience building ingestion pipelines from multiple heterogeneous sources
- Working knowledge of Delta Lake and medallion/lakehouse architecture
- Experience implementing incremental loads and CDC-style processing
- Experience implementing data quality checks and troubleshooting pipeline failures
Nice-to-Have
- Microsoft Fabric hands-on experience (Mirroring, Copy Jobs, Environments)
- Exposure to entity/master data standardization (name and address parsing)
- Familiarity with libraries such as Great Expectations for data quality
- Experience optimising for Fabric capacity/CU consumption
Key Deliverables Owned
- Operational ingestion pipelines for all agreed sources
- Bronze/raw layer with one Delta table per source and CDC retained
- Standardization and parsing transformation logic
- Data quality checks, monitoring and exception handling
- Source-to-target mapping and run documentation
Responsibilities
- Ingestion development — build and maintain pipelines to land the six in-scope sources (Secretary of State, D&B, ARROW, E1, hCue, DocCentral) into the Fabric Bronze/raw layer.
- Mirroring & CDC — implement Fabric Mirroring for supported structured sources and establish change-data-capture patterns; implement watermark/incremental load logic where mirroring is unavailable.
- Raw layer management — maintain one Delta table per source on an append-only basis, retaining evidence records and full source provenance.
- Standardization & transformation — implement name normalization, address parsing and attribute standardization logic in Spark notebooks; support identifier-spine construction.
- Data quality — implement data quality checks, validation rules, threshold alerts and exception handling; support reconciliation against source.
- Pipeline operations — schedule, monitor and troubleshoot pipeline runs; investigate failures and performance issues; maintain run documentation.
- Performance tuning — optimise Spark jobs, Delta file sizes, partitioning and pipeline efficiency to manage Fabric capacity consumption.
- Documentation — produce and maintain source-to-target mappings, transformation logic documentation and lineage records.
About EXL
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at EXL
Senior Executive
Pune, IND
The candidate should know how BDX Processing and LPAN Creation is processed and validated Bordereaux data and creating LPANs for accurate premium processing within the London Market. This role involves coordinating with
Diagnosis-Related Group (DRG) Manager
, USA
EXL Health is seeking a DRG Manager. This is a work from home opportunity.
Senior Executive - Operation
Chennai, IND
Review, Call, Analyze and Manage assigned outstanding receivables portfolio by ensuring outstanding/denied claims are resolved, follow up effectively for additional information as needed with insurance companies for clai
Technical Consultant
Jersey City, USA
We are looking for a Management Consultant with 10+ years of experience to join our growing practice as a client-facing advisor and solution leader. You will shape and sell data and cloud transformation programs, lead ex
Databricks Platform Engineer
, USA
Base Compensation Range: USD $100,000 – $140,000 The posted range is the hiring range for this role — a subset of the broader range available to employees over time — and reflects base salary across our national hiring
Assistant Manager
, IND
Process and validate employee travel and expense claims in line with company policies. Review expense reports, supporting documents, and approvals to ensure accuracy and compliance. Manage employee reimbursements and ens
AVP - MDM Informatica Lead
, USA
Base Compensation Range: USD $150,000 – $200,000 The posted range is the hiring range for this role — a subset of the broader range available to employees over time — and reflects base salary across our national hiring
Senior Executive
Pune, IND
Diagnosis-Related Group (DRG) Manager
, USA
Senior Executive - Operation
Chennai, IND
AWS Data Engineer
Gurugram, IND
Technical Consultant
Jersey City, USA
Databricks Platform Engineer
, USA
Assistant Manager
, IND
AVP - MDM Informatica Lead
, USA
