Senior Developer, Data Engineer

Apache Airflow: Expert level, with at least three years running Airflow in production. You should know the scheduler internals, executor types, XComs, Connections, Variables, pools, and the TaskFlow API well enough to make trade-off decisions and explain them clearly.

Airflow on Kubernetes: Real experience deploying and operating Airflow on Kubernetes via KubernetesExecutor or CeleryKubernetesExecutor: pod templates, resource tuning, persistent volume claims, Helm chart management, and upgrades that do not take the scheduler offline.

DAG Development: Strong Python applied to DAG authoring: dynamic task mapping, custom operators and sensors, cross-DAG triggers, parameterised pipelines, and proper test coverage. Writing testable DAGs should feel natural, not optional.

ETL and ELT Pipelines: Demonstrated experience building batch and incremental pipelines at scale. You understand transformation logic, data quality validation, and lineage capture, and you have shipped pipelines that other people maintain without you.

Data Lakehouse Architecture: A solid grasp of lakehouse design in practice: how to implement a medallion model, when to use which table format, and how to make data reliably available to downstream consumers without constant hand-holding.

Apache Iceberg: Hands-on knowledge of the Iceberg table spec: snapshot isolation, partition evolution, hidden partitioning, and time-travel queries. You have managed Iceberg tables in a real catalog environment, whether Hive metastore, a REST catalog, or otherwise.

Databricks: Practical Databricks experience covering cluster management, Delta Lake, Workflows or Jobs, and Unity Catalog. Experience with Databricks Asset Bundles or MLflow is a welcome bonus.

dbt: Confident building dbt projects from the ground up: models, sources, tests, snapshots, seeds, and macros. You know how to wire dbt into Airflow, write documentation that people actually use, and make freshness checks meaningful.

Python: Strong engineering-grade Python: well-structured, tested, and packaged. Comfortable with pandas, PySpark, SQLAlchemy, and Pydantic. You write code other engineers want to read.

Apache Kafka and Streaming: Experience with near-real-time ingestion using Kafka, Kafka Connect, or Flink, including landing data into lakehouse storage.

Apache Spark: PySpark proficiency for large-scale distributed processing, with a feel for memory management, shuffle optimisation, and partition sizing.

Kubernetes and OpenShift: Platform knowledge beyond Airflow: Helm, RBAC, and namespace management, enough to self-serve on infrastructure without leaning on platform teams.

Data Governance and Contracts: Familiarity with data contract approaches, catalogue tools such as DataHub or Collibra, and quality frameworks like Great Expectations or Soda.

Cloud Platforms: Experience with managed data services on AWS, Azure, or GCP: S3, Glue, ADLS, Synapse, GCS, Dataproc, and similar.

Infrastructure as Code: Ability to provision data infrastructure using Terraform or Pulumi.

ML Feature Engineering: An understanding of feature stores and experience building pipelines that serve model training and inference reliably.