Data Architect

• Design and maintain the organisation's lakehouse architecture using Databricks and Delta Lake.

• Define data modelling standards (dimensional, Data Vault 2.0, or medallion architecture) across Bronze, Silver, and Gold layers.

• Architect scalable ingestion frameworks using structured and unstructured data sources (Kafka, JDBC, REST APIs, cloud storage).

• Own schema evolution strategy and ensure backward-compatibility across data assets.

• Build and maintain production-grade ETL/ELT pipelines using PySpark, Spark SQL, and Databricks Workflows.

• Implement Delta Live Tables (DLT) for declarative, auto-scaling pipeline development.

• Optimise Spark jobs for performance — partitioning, Z-ordering, caching, and cluster right-sizing.

• Establish CI/CD practices for data pipelines using tools such as GitHub Actions, Azure DevOps, or Databricks Asset Bundles.

• Implement Unity Catalog for data discovery, lineage tracking, fine-grained access control, and compliance.

• Define and enforce data quality rules using Great Expectations, DLT expectations, or equivalent frameworks.

• Work with data governance teams to document metadata, business glossary, and data contracts.

• Manage Databricks workspace configuration: clusters, pools, secrets, and access policies.

• Collaborate with cloud and DevOps teams on infrastructure-as-code (Terraform) for Databricks on Azure / AWS / GCP.

• Monitor platform health, SLAs, and cost using Databricks system tables and cloud-native monitoring tools.

• Partner with data consumers (analysts, data scientists, ML engineers) to define SLAs and publish clean, well-documented data products.

• Review code and provide architectural guidance to junior engineers.

• Contribute to and champion internal data engineering best practices, runbooks, and documentation.

• 4+ years of hands-on experience with Databricks (Unified Data Analytics Platform).

• Strong proficiency in PySpark and Spark SQL for large-scale data transformation.

• Deep knowledge of Delta Lake — ACID transactions, time travel, OPTIMIZE, VACUUM.

• Experience with Databricks Workflows, Jobs, and Delta Live Tables (DLT).

• Familiarity with Unity Catalog and Databricks governance features.

• Solid understanding of data modelling paradigms: dimensional modelling, Data Vault, or medallion architecture.

• Experience designing and operating streaming pipelines (Structured Streaming, Kafka, Event Hubs, or Kinesis).

• Proficiency in SQL; experience with dbt is a strong plus.

• Hands-on experience with cloud platforms: Azure (ADLS, ADF), AWS (S3, Glue), or GCP (BigQuery, GCS).

• Version control with Git; experience with branching strategies and code review workflows.

• Ability to write testable, modular pipeline code with unit and integration tests.

• Familiarity with CI/CD pipelines and infrastructure-as-code (Terraform preferred).

• Databricks Certified Data Engineer Associate or Professional certification.

• Experience with data mesh or data product frameworks.

• Exposure to ML pipelines, MLflow, or Feature Store on Databricks.

• Knowledge of data cataloguing tools (Alation, Collibra, or Databricks Unity Catalog).

• Experience with Apache Iceberg or Apache Hudi as alternative table formats.

• Familiarity with real-time analytics or OLAP systems (Druid, ClickHouse, Redshift).