Senior Data Engineer- (Government entity)
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
We are seeking an experienced data engineer to design, build, and optimize data pipelines for a scalable and secure health data platform. You will work with big data systems, implement ETL processes, and ensure compliance with healthcare data regulations.
Key Skills for This Role
Responsibilities
- Design and implement data pipelines, ETL processes, schemas, and data models for multi petabyte scale datasets
- Build and optimize data processing systems on modern platforms like Spark, Delta Lake, Kafka
- Implement data quality, validation, and monitoring measures using tools such as Great Expectations
- Ensure compliance with security, access control, and regulatory requirements related to PHI and other sensitive data types
- Support adoption of emerging standards like FHIR for healthcare data exchange
- Collaborate with data scientists, analysts, and engineers to deliver performant, reliable data products
- Keep track of emerging technologies and trends in Data Engineering
Requirements
- 5+ years experience building and operating production big data platforms and pipelines
- Strong experience with SQL, Spark, workflow orchestrators, distributed message bus, Python, Presto, Deltalake, Apache big data tool suites, Docker, Kubernetes, MPP
- Hands on with cloud based data solutions (AWS, Azure, or GCP)
- Experience with real time data pipelines using streaming technologies like Apache Kafka or AWS Event hub
- Strong track record of designing and implementing scalable data models, schemas, ETL logic
- Experience with data governance, master data management, data pseudonymization and anonymization, and data catalog solutions
Full Job Posting
Role Summary
- We are seeking an experienced data engineer to join our team building a scalable and secure health data platform.
- You will design, build and optimize data pipelines (batch & streaming) for big data systems.
- Extracting, Analyzing and Modelling of rich & diverse health data sets.
Responsibilities
- Design and implement data pipelines, ETL processes, schemas, and data models to ingest, process, and prepare multi petabyte scale datasets for downstream analytics and machine learning.
- Build and optimize data processing systems on modern platforms like Spark, Delta Lake, Kafka, etc.
- Implement data quality, validation, and monitoring measures by leveraging tools such as Great Expectations.
- Ensure compliance with security, access control, and regulatory requirements related to PHI and other sensitive data types.
- Support adoption of emerging standards like FHIR for healthcare data exchange.
- Collaborate with data scientists, analysts, and engineers to understand data needs and deliver performant, reliable data products.
- Keep track of emerging technologies & trends in the Data Engineering world, incorporating modern tooling and best practices at Craft.
Qualifications and Experience
- 5+ years experience building and operating production big data platforms and pipelines.
- Strong experience with SQL, Spark, workflow orchestrators, distributed message bus, Python, Presto, Deltalake, Apache big data tool suites, Docker, Kubernetes, MPP.
- Hands on with the design and implementation of cloud based data solutions using platforms like AWS, Azure, or GCP, optimizing for scalability, cost efficiency, and performance.
- Develop real time data pipelines using streaming technologies like Apache Kafka or AWS Event hub.
- Manage and enhance distributed data systems (e.g., Hadoop, Spark) to efficiently process large scale datasets.
- Previous experience of working on health data and Azure cloud is a strong plus.
- Strong track record of designing and implementing scalable data models, schemas, ETL logic.
- Experience with data governance, master data management, data pseudonimization and anonymization, and data catalog solutions.
- A strong interest in learning new things and team player ethics.
- Strong analytical skills and good understanding of data structures and algorithms.
Preferred Qualifications
- Experience building data pipelines for machine learning.
- Knowledge of genomics, medical imaging, and/or EHR data domains.
- Knowledge of HIPAA, HL7 and other healthcare data privacy requirements.
- Hands on experience with fully managed data warehousing solutions Azure Synapse, AWS Redshift, Bigquery, Snowflake etc.
- Azure Batch & Blob Storage.
- Nextflow, Nextflow Tower.
Work Location
- In person
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Healthsoft Middle East
Quality Assurance Engineer - (Government entity)
Abu Dhabi, UAE
Healthsoft Middle East is looking for a QA Engineer to ensure software quality through manual and automated testing. The role involves designing test strategies, developing automated scripts with Playwright, performing A
Job Title: API Developer (Government entity)
Abu Dhabi, UAE
Healthsoft Middle East seeks an API Developer to design and implement FHIR-compliant APIs for a government entity. The role requires 5-7 years of API development experience, including FHIR and REST API expertise, and inv
Quality Assurance Engineer
Abu Dhabi, UAE
We are seeking a QA Engineer to ensure the quality and integrity of the onboarding process for healthcare facilities integrating with our Health Information Exchange platform. The role involves end-to-end testing of data
Fullstack Developer
Abu Dhabi, UAE
Healthsoft Middle East is seeking a Fullstack Developer to design, develop, and maintain scalable mobile and web applications using React Native, React JS, .NET, and Azure. The role requires 10+ years of experience, expe
Quality Assurance Engineer - (Government entity)
Abu Dhabi, UAE
Healthsoft Middle East is looking for a QA Engineer to ensure software quality through manual and automated testing for a government entity. The role involves designing test strategies, executing functional and performan
Procurement Coordinator
Abu Dhabi, UAE
JOB PURPOSE:- The Procurement Coordinator supports the Procurement Lead in managing day-to-day procurement activities, ensuring timely execution, compliance, and effective coordination with internal stakeholders and vend
Quality Assurance Engineer - (Government entity)
Abu Dhabi, UAE
Job Title: API Developer (Government entity)
Abu Dhabi, UAE
Quality Assurance Engineer
Abu Dhabi, UAE
Fullstack Developer
Abu Dhabi, UAE
Quality Assurance Engineer - (Government entity)
Abu Dhabi, UAE
Procurement Coordinator
Abu Dhabi, UAE