Lead Data Engineer

Key Responsibilities

• Leading the as-is analysis of a ~1,100-table SQL Server RDS estate — establishing what exists, how it's structured, and where the risk sits, in the absence of any existing documentation or stored procedure inventory.
• Working alongside Claude-driven codebase scanning to map integration points, trace data lineage, and identify likely dead code, then applying engineering judgement to validate and prioritise the findings.
• Investigating duplicated tables across schemas to determine which copies are live, which are redundant, and what depends on each — building the tooling and queries needed where none currently exist.
• Designing an approach to 15 years of unarchived trade data — assessing volume, growth, and usage patterns to inform archiving, retention, and future-state options.
• Producing clear, defensible documentation of the database estate — schemas, dependencies, stored procedures, and data flows — that the client can rely on long after the engagement ends.
• Advising on remediation and modernisation options once the estate is understood, from consolidation and archiving through to longer-term platform or cloud migration.
• Leading and mentoring a small team of data engineers on the engagement, setting standards for how findings are verified, tested, and documented.
• Acting as a credible, hands-on technical partner to client stakeholders — including DBAs, platform owners, and risk/compliance contacts — who will be understandably cautious about changes to a live trading data estate.

Skills & Experience

• You know your way around legacy SQL Server estates
• You're genuinely strong in SQL Server — schema design, query performance, stored procedures, indexing — and comfortable working in large, poorly documented production estates.
• You've reverse-engineered a database landscape before: worked out what's live vs. dead, resolved duplicated or conflicting schemas, and built confidence in an estate nobody fully understood.
• You understand the particular pressures of financial services data — long retention requirements, audit and regulatory expectations, and the need to prove data integrity for trade and transaction records.
• You're comfortable with ambiguity and incomplete information, and know how to build a reliable picture of a system methodically rather than guessing.
• You're comfortable using AI tooling as part of the analysis
• You're open to and effective at using AI-assisted tools (we use Claude) to scan codebases, map integration points, and surface candidate dead code at a scale manual review can't match.
• You know how to sanity-check and validate AI-generated findings against the real system — you treat the output as a strong starting hypothesis, not ground truth.
• You can translate what the tooling surfaces into a clear, prioritised narrative that both engineers and non-technical stakeholders can act on.
• You know your tools
• Deep, hands-on SQL Server experience — T-SQL, SSMS, execution plans, and diagnosing performance or integrity issues in large OLTP schemas.
• Comfortable with AWS RDS for SQL Server, including monitoring, performance insights, and operational management.
• Solid scripting ability (Python and/or PowerShell) for building the discovery, cataloguing, and dependency-mapping tooling this engagement needs from scratch.
• Experience with data cataloguing, lineage, or metadata management tooling is a strong plus, even if the client doesn't currently have any in place.
• Familiarity with data archiving and retention strategies for large historical datasets is beneficial.
• You thrive in a consultancy setting
• You've led or mentored data engineers and are comfortable owning technical direction on a complex, ambiguous engagement.
• You can navigate the caution and politics of a large FS operator — multiple stakeholders, risk-averse change processes, and legacy systems that can't just be switched off.
• You're happy being onsite in Glasgow two days a week, building trust face-to-face with the client team
• You balance hands-on delivery with client engagement, and know when to push back on scope and when to be pragmatic.
• Tech snapshot
• Primary platform Microsoft SQL Server (AWS RDS) — T-SQL, SSMS, stored procedures, execution plans
• Cloud AWS — RDS, S3, and supporting services for tooling and discovery outputs
• Estate ~1,100 tables, 15 years of trade data, no archiving strategy, no stored procedure inventory, duplicated schemas
• AI-assisted analysis Claude — codebase scanning, integration point mapping, dead code identification
• Languages SQL (T-SQL), Python and/or PowerShell for tooling
• Documentation & lineage - Data cataloguing / lineage tooling (implementation as part of the engagement)
• Ways of working Hybrid — client site in Glasgow, 2 days per week

What we’ll offer you:

Our hiring process

Inclusion at CreateFuture