Architect - Data

Key Responsibilities

• Own the end-to-end architecture across ingestion (Flink + PySpark), storage (Iceberg on GCS with BigLake Metastore), transformation (dbt over Starburst), FHIR serialization (flat FHIR Iceberg → bundles → FHIR-Repository), and consumption (Starburst, FHIR API, data products). Maintain the architecture specification document and its companion design docs.
• Lead the design and evolution of the Common Data Model (CDM): the dimensional, fact, bridge, and reference table classes; SCD2 semantics; hash key conventions; the write-authority matrix that governs cross-source field precedence. CDM is the project's central design artifact and the architect owns it.
• Define the bidirectional FHIR flow with origin-tag-based loop prevention. Specify the FHIR repository egress interceptor contract, the loopback Flink consumer pipeline, and the boundary semantics between FHIR-repo-owned and CDM-owned fields.
• Define the patient identity resolution architecture using Informatica MDM, including the synchronous-call pattern at ingestion, the asynchronous ECI change event pipeline, and the DLQ taxonomy for MDM failures.
• Set the standards every other role implements against — naming conventions, hash algorithms (SHA-256 BINARY 32, 0x1F separator, NULL handling), Iceberg table properties (CoW vs MoR, partitioning, compaction cadence), DLQ taxonomy, observability (metric naming, structured logging, correlation ID propagation), and security (PHI handling, encryption, audit logging).
• Own the spec-driven development framework's program-level and component-level specs. Approve significant changes to platform-wide rules. Architect specs are the constitution every implementing engineer references.
• Drive the high-volume capacity design — sustained 50K msg/sec ingest target with 150K peak, billions of rows in CDM facts, thousand-concurrent Starburst workloads, thousands per second FHIR API. Lead capacity planning, validation, and the parallel-run cutover from the existing Health Data Engine.
• Evaluate and recommend technology choices that are still open — Confluent Cloud tier, Starburst Galaxy vs Enterprise, FHIR-Repository sizing, multi-region failover topology. Build option analyses; defend recommendations with concrete trade-off matrices.
• Provide architectural review for engineering work. Review specs at the component and unit level when they touch architectural concerns. Mentor data engineers and data modelers on design principles.
• Lead architectural review meetings with customer technical stakeholders — including the Health Data Engine retirement team, governance, security, and clinical informatics. Translate complex trade-offs into language non-architect audiences can engage with.
• Establish and evolve the disaster recovery, backup, and reprocessing strategy across all platform layers. RPO under 5 minutes for streaming, RTO under 4 hours for full platform recovery.