Senior Data Platform Engineer

Design and implement Bronze, canonical Silver, and Gold Delta table schemas with PAN-safe physical layout — statistics allowlist configuration, liquid clustering, deliberate retention settings, V-Order optimization.

Build Fabric Spark notebooks for the feature pipeline (rolling velocity features across 24h / 7d / 30d windows, merchant-level rolling features, hierarchical rollups, cross-merchant features via pan_token joins).

Author per-flavor-version Silver adapters — one function per flavor, unioned by name, tested against golden fixtures.

Build the canonical MERGE DAG (Bronze → canonical Silver, idempotent on _natural_key per Aperia's canonical Fabric internal spec v1.4).

Build the Gold materialization pipeline for both grains (fact_transaction_risk, fact_merchant_risk) with correct score-source stamping and idempotent merge semantics.

Design and implement Direct Lake semantic models per reporting surface; performance-test against representative data volumes.

Configure OneLake namespace layouts, workspace catalogs, and cross-workspace shortcut relationships with the residuals programme.

Establish idempotent Delta write patterns using delta-rs from Python containers (segmented parallel writes with single atomic commit).

Fabric capacity sizing (F-SKU selection, load testing, Reservation vs PAYG decisions).

Partner with the DevOps engineer on Fabric IaC (Terraform capacity provisioning plus Fabric REST workspace bootstrap).

Consult on ClickHouse contingency if Direct Lake latency proves insufficient for client-tier UIs.

5+ years data engineering experience, with at least 2 years hands-on Spark development (PySpark preferred).

Deep Delta Lake internals knowledge — transaction log format, checkpoints, retention settings (logRetentionDuration, deletedFileRetentionDuration), VACUUM, OPTIMIZE, deletion vectors, statistics collection.

Parquet format expertise — footers, statistics allowlists, row-group sizing, dictionary encoding, column indexes.

Microsoft Fabric hands-on experience — workspaces, capacities, notebooks, lakehouses, semantic models, Direct Lake, SQL analytics endpoint.

SQL performance tuning skills for both OLAP (Fabric) and OLTP (PostgreSQL) workloads.

Strong Python data engineering skills — pandas, PyArrow, delta-rs.

Ability to communicate architecture clearly in writing and to work autonomously on well-scoped features.

PCI-scoped data platform experience.

Financial services or payments domain background.

Power BI semantic model authoring (DAX).

Familiarity with published data-standard governance patterns — logical model vs physical profile, SCD2 dimensions, natural-key idempotency, extension attribute stores.

OneLake shortcuts and cross-workspace lakehouse patterns.

ClickHouse or similar OLAP engine familiarity.

Terraform for Azure data resources.

Prior experience with fluid or schema-flexible ingestion patterns.

Founding-member impact on a strategic platform serving a multi-hundred-million-dollar industry — your architectural decisions carry weight and ship to production.

Direct working relationship with the SVP of Technology; small team; minimal bureaucracy; senior peers.

Genuine engineering culture — technical correctness valued over slideware; extensive design documentation; real design reviews; ADR discipline.

Explicit team commitment to AI-assisted development (Claude Code, Cursor, GitHub Copilot) — treated as a first-class productivity multiplier, not a novelty. Team standards documented and enforced.

Modern tech stack: Microsoft Fabric, AKS, Azure, Airflow, Python, Delta Lake, Kogito/DMN, OpenBao, Kubernetes.

Full-time