Senior Databricks Data Engineer

Responsibilities

• Design and lead Databricks Lakehouse architecture for large-scale healthcare data platforms.
• Define scalable data architectures for batch and real-time healthcare data processing.
• Design data ingestion, transformation, storage, and consumption patterns using Databricks.
• Develop architecture using Delta Lake, Unity Catalog, Databricks Workflows, and Delta Live Tables/Lakeflow.
• Integrate data from multiple healthcare sources, including clinical, claims, eligibility, provider, and patient systems.
• Design robust ETL/ELT pipelines using PySpark, SQL, and Databricks.
• Establish data quality, validation, reconciliation, lineage, and governance frameworks.
• Design secure healthcare data solutions while considering HIPAA and PHI/PII protection requirements.
• Work with cloud platforms such as Azure, AWS, or GCP to build scalable data solutions.
• Collaborate with data engineers, data scientists, analysts, product owners, and business stakeholders.
• Define technical standards, architectural patterns, and best practices for Databricks development.
• Lead performance optimization and cost-management initiatives across the Databricks platform.
• Provide technical leadership and mentoring to data engineering teams.
• Participate in Agile ceremonies and contribute to technical planning and roadmap discussions.
• - Design and lead Databricks Lakehouse architecture for large-scale healthcare data platforms. - Define scalable data architectures for batch and real-time healthcare data processing. - Design data ingestion, transformation, storage, and consumption patterns using Databricks. - Develop architecture using Delta Lake, Unity Catalog, Databricks Workflows, and Delta Live Tables/Lakeflow. - Integrate data from multiple healthcare sources, including clinical, claims, eligibility, provider, and patient systems. - Design robust ETL/ELT pipelines using PySpark, SQL, and Databricks. - Establish data quality, validation, reconciliation, lineage, and governance frameworks. - Design secure healthcare data solutions while considering HIPAA and PHI/PII protection requirements. - Work with cloud platforms such as Azure, AWS, or GCP to build scalable data solutions. - Collaborate with data engineers, data scientists, analysts, product owners, and business stakeholders. - Define technical standards, architectural patterns, and best practices for Databricks development. - Lead performance optimization and cost-management initiatives across the Databricks platform. - Provide technical leadership and mentoring to data engineering teams. - Participate in Agile ceremonies and contribute to technical planning and roadmap discussions.