Principal Data Engineer

We Support Flexible Working – Click here for more information on flexible working options

Job Title: Engineering Leader – Data (Lloyds Technology Centre) Location: Hyderabad, India (Hybrid: at least 2 days/week in office) Experience: 1 5+ years About Lloyds Technology Centre Lloyds Technology Centre is the strategic technology hub for Lloyds Banking Group , enabling digital transformation and innovation across the banking ecosystem. Our mission is to build secure, scalable, and data-driven solutions that empower millions of customers and ensure compliance with financial regulations. Role Overview Own the delivery and operation of data products and platforms across multiple squads, with end ‑ to ‑ end accountability for outcomes, quality, and reliability. Establish reusable, metadata ‑ driven engineering patterns; elevate semantic models, KPI/metrics , and BI consumption ; and build an AI ‑ ready foundation (knowledge layers, knowledge graphs, semantic models) that accelerates analytics and machine learning use ‑ cases across the organisation. Key Responsibilities Delivery ownership: Define roadmaps, OKRs, and release plans; oversee scope, estimation, risk, and stakeholder communication; ensure on ‑ time, on ‑ budget, quality delivery across squads. Team ownership & leadership: Build and lead high ‑ performing teams (hiring, coaching, performance management); set coding standards, DoR /DoD, working agreements, and succession plans. Engineering patterns at scale: Establish factory ‑ mode delivery via inner ‑ sourced, reusable frameworks (ingestion, transform, quality, lineage, observability) and golden paths with strong documentation Semantic layer & metrics: Define and govern enterprise semantic models, KPI/metric definitions , conformed dimensions, metric stores, and query ‑ ready views to enable consistent BI and self ‑ serve analytics. AI readiness & knowledge layers: Shape data for ML/GenAI—ontologies, knowledge graphs , feature/embedding strategies, and patterns that make the platform AI ‑ ready by design. Data modelling & performance: Guide dimensional (star/snowflake) and Data Vault 2.0 modelling ; set standards for physical design on cloud warehouses (partitioning, clustering, caching, workload mgmt.) Streaming & real ‑ time: Oversee event pipelines (Kafka/Pub/Sub/Kinesis + Flink/Spark) with exactly ‑ once semantics, replay, SLAs/SLOs, and resiliency patterns. Quality, metadata & lineage: Make quality the default —data contracts, DQ rules/tests, reconciliation—and automate capture/propagation of technical/business metadata and end ‑ to ‑ end lineage Observability & FinOps: Implement platform and pipeline telemetry (logs, metrics, tracing) and proactive cost/performance guardrails for cloud DW/compute; continuously optimise jobs/queries. Security & compliance: Champion IAM, least ‑ privilege patterns, encryption, secrets mgmt., and auditability; embed privacy ‑ by ‑ design and regulatory controls into pipelines and platforms. Vendor & partner management: Govern partner delivery (e.g., TCS/HCL), enforce standards, and ensure reusable IP is contributed back to inner ‑ source repos. Required Skills Leadership & ownership: Proven record of owning complex data platform deliveries, leading multiple squads, and managing outcomes/SLAs in production. Semantic & BI expertise: Hands ‑ on experience defining semantic layers , KPI/metric logic, metric stores, and consumption models that scale across BI tools. AI ‑ ready design: Practical understanding of AI/ML data needs—feature engineering, data products for ML, knowledge graphs/ontologies , vector/embedding patterns—and how to operationalise them on cloud platforms. Modelling depth: Dimensional (star/snowflake), Data Vault 2.0, SCDs, schema evolution/versioning; ability to set and enforce modelling standards. Cloud data platforms: Strong experience with one or more of BigQuery , Snowflake, Redshift, Synapse/Databricks SQL; clarity on cloud DW vs traditional MPP trade ‑ offs and performance levers. Streaming systems: Kafka/Pub/Sub/Kinesis and Spark/Flink; event schema management (Avro/ Protobuf ), idempotency, back ‑ pressure, state mgmt. Software ‑ engineering mindset: Git ‑ based workflows, code reviews, automated testing, CI/CD, artifact versioning; inner ‑ sourcing culture and documentation excellence. IaC & runtime: Terraform/CloudFormation, Docker/Kubernetes; sizing, autoscaling, job concurrency, and reliability engineering. Data quality & governance: Data contracts; testing frameworks (e.g., Great Expectations/ dbt tests); catalogue /lineage tooling; policy enforcement via pipelines. FinOps & observability: Cost modelling for cloud DW/compute, workload tuning, budgets/alerts; platform and pipeline telemetry with actionable SLOs.