Principal Data Engineer

We Support Flexible Working – Click here for more information on flexible working options

The Data Engineer will work alongside the Team Product Owner in a Feature Team managing delivery by leading others.

They will operate as co-leader with the Team PO with involvement in decision making and planning of Product roadmap.

In some cases, this role will be an Individual Contributor operating at Lab rather than feature team level.

• Job Title: Engineering Leader – Data (Lloyds Technology Centre)

Location: Hyderabad, India (Hybrid: at least 2 days/week in office)

Lloyds Technology Centre is the strategic technology hub for Lloyds Banking Group , enabling digital transformation and innovation across the banking ecosystem.

Our mission is to build secure, scalable, and data-driven solutions that empower millions of customers and ensure compliance with financial regulations.

Own the delivery and operation of data products and platforms across multiple squads, with end ‑ to ‑ end accountability for outcomes, quality, and reliability.

Establish reusable, metadata ‑ driven engineering patterns; elevate semantic models, KPI/metrics , and BI consumption ; and build an AI ‑ ready foundation (knowledge layers, knowledge graphs, semantic models) that accelerates analytics and machine learning use ‑ cases across the organisation.

• Delivery ownership: Define roadmaps, OKRs, and release plans; oversee scope, estimation, risk, and stakeholder communication; ensure on ‑ time, on ‑ budget, quality delivery across squads.

• Team ownership & leadership: Build and lead high ‑ performing teams (hiring, coaching, performance management); set coding standards, DoR /DoD, working agreements, and succession plans.

• Engineering patterns at scale: Establish factory ‑ mode delivery via inner ‑ sourced, reusable frameworks (ingestion, transform, quality, lineage, observability) and golden paths with strong documentation

• Semantic layer & metrics: Define and govern enterprise semantic models, KPI/metric definitions , conformed dimensions, metric stores, and query ‑ ready views to enable consistent BI and self ‑ serve analytics.

• AI readiness & knowledge layers: Shape data for ML/GenAI—ontologies, knowledge graphs , feature/embedding strategies, and patterns that make the platform AI ‑ ready by design.

• Data modelling & performance: Guide dimensional (star/snowflake) and Data Vault 2.0 modelling ; set standards for physical design on cloud warehouses (partitioning, clustering, caching, workload mgmt.)

• Streaming & real ‑ time: Oversee event pipelines (Kafka/Pub/Sub/Kinesis + Flink/Spark) with exactly ‑ once semantics, replay, SLAs/SLOs, and resiliency patterns.

• Quality, metadata & lineage: Make quality the default —data contracts, DQ rules/tests, reconciliation—and automate capture/propagation of technical/business metadata and end ‑ to ‑ end lineage

• Observability & FinOps: Implement platform and pipeline telemetry (logs, metrics, tracing) and proactive cost/performance guardrails for cloud DW/compute; continuously optimise jobs/queries.

• Security & compliance: Champion IAM, least ‑ privilege patterns, encryption, secrets mgmt., and auditability; embed privacy ‑ by ‑ design and regulatory controls into pipelines and platforms.

• Vendor & partner management: Govern partner delivery (e.g., TCS/HCL), enforce standards, and ensure reusable IP is contributed back to inner ‑ source repos.Required Skills

• Leadership & ownership: Proven record of owning complex data platform deliveries, leading multiple squads, and managing outcomes/SLAs in production.

• Semantic & BI expertise: Hands ‑ on experience defining semantic layers , KPI/metric logic, metric stores, and consumption models that scale across BI tools.

• AI ‑ ready design: Practical understanding of AI/ML data needs—feature engineering, data products for ML, knowledge graphs/ontologies , vector/embedding patterns—and how to operationalise them on cloud platforms.

• Modelling depth: Dimensional (star/snowflake), Data Vault 2.0, SCDs, schema evolution/versioning; ability to set and enforce modelling standards.

• Cloud data platforms: Strong experience with one or more of BigQuery , Snowflake, Redshift, Synapse/Databricks SQL; clarity on cloud DW vs traditional MPP trade ‑ offs and performance levers.

• Streaming systems: Kafka/Pub/Sub/Kinesis and Spark/Flink; event schema management (Avro/ Protobuf ), idempotency, back ‑ pressure, state mgmt.

• Software ‑ engineering mindset: Git ‑ based workflows, code reviews, automated testing, CI/CD, artifact versioning; inner ‑ sourcing culture and documentation excellence.

• IaC & runtime: Terraform/CloudFormation, Docker/Kubernetes; sizing, autoscaling, job concurrency, and reliability engineering.

• Data quality & governance: Data contracts; testing frameworks (e.g., Great Expectations/ dbt tests); catalogue /lineage tooling; policy enforcement via pipelines.

• FinOps & observability: Cost modelling for cloud DW/compute, workload tuning, budgets/alerts; platform and pipeline telemetry with actionable SLOs.