Lead Databricks Engineer (Senior Software Engineer II)
Role Overview
Technical owner and steward of Nielsen's Databricks Lakehouse for Corporate functions.
Leads platform strategy, architecture, governance, reliability, and cost maturity.
Builds AI-native platform capabilities and uses AI-assisted engineering practices to scale platform delivery.
Key Skills for This Role
Full Job Posting
Role overview
Technical owner and steward of Nielsen's Databricks Lakehouse for Corporate functions.
Leads platform strategy, architecture, governance, reliability, and cost maturity.
Builds AI-native platform capabilities and uses AI-assisted engineering practices to scale platform delivery.
Platform ownership and governance
- Own Lakehouse architecture, workspace topology, cluster policies, compute strategy, and multi-cloud deployment patterns across AWS, Azure, or GCP.
- Design and enforce Unity Catalog governance, including catalogs, schemas, access controls, lineage, metadata management, and audit logging.
- Define standards for Delta Lake, Delta Live Tables, medallion architecture, and performance tuning.
- Industrialize platform reliability through infrastructure as code, CI/CD, and monitoring, targeting 99.99% availability.
- Manage cost transparency and optimization through right-sizing, workload isolation, chargeback/showback, and budget alerts.
- Partner with engineering, analytics, data science, and business stakeholders on platform capabilities and self-service patterns.
- Govern Databricks AI features and use cases within access-control and governance boundaries.
- Lead a developer community of practice, publish standards, run enablement sessions, and remove delivery friction.
Experience and technical qualifications
- 8–10+ years of experience in data engineering, data platform architecture, or distributed systems.
- 3+ years architecting and operating Databricks environments at scale.
- Hands-on experience building and operating production Databricks platforms, including Unity Catalog, Delta Lake, Delta Live Tables, Workflows, SQL, and cluster/compute policies.
- Experience architecting feature stores, vector search/indexes, and GenAI-ready pipelines.
- Experience with AI-assisted and agentic engineering practices.
- Deep hands-on Apache Spark expertise for batch and structured-streaming processing and performance tuning.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at The Nielsen Company
Senior Customer Success Manager
, USA
Customer Success at Nielsen As a part of our Customer Success team at Nielsen, you will play a critical role in ensuring customer satisfaction by helping our customers utilize Nielsen’s products and services to achieve t
Senior Controllership Associate II
Mumbai, IND
Role Summary Support global Record-to-Report (R2R) operations by delivering accurate, timely, and compliant financial reporting. The role goes beyond processing to enable stronger stakeholder collaboration and drive cont
Senior Process Improvement Professional II
Mumbai, IND
Overall objective The Senior Process Improvement Professional will be a key driver in transforming our operational capabilities. This role is responsible for leading, implementing, and sustaining a culture of continuous
Senior Technical Program Manager II (Bangalore)
Bengaluru, IND
Lead governance of growth initiatives with comprehensive weekly updates on progress, providing visibility into development timelines, milestone achievements, risks and any deviations from targets. Identify and communicat
Member Technical Staff 2 - C/C++
Bengaluru, IND
We are looking for a highly skilled Senior Member of Technical Staff with 5–8 years of hands-on experience in systems-level programming, cloud infrastructure, and database technologies. The ideal candidate will have deep
Customer Support Representative II
Mumbai, IND
This is role is for Gracenote - A Global entertainment data and technology company that provides music, video, and sports metadata, automatic content recognition (ACR), and content discovery solutions that power search a
Treasury Manager, Capital Markets
, USA
Nielsen is seeking a Treasury Manager, Capital Markets to lead key capital markets activities across the treasury function. This role is responsible for managing debt and credit facilities, driving cash flow forecasting,
Manager, Convening Event Experiences
New York City, USA
The Manager, Convening runs Nielsen's always-on engagement calendar, mainly in New York. Where the premium portfolio creates a handful of major moments each year, this role keeps Nielsen in front of clients continuously
Senior Customer Success Manager
, USA
Senior Controllership Associate II
Mumbai, IND
Senior Process Improvement Professional II
Mumbai, IND
Senior Technical Program Manager II (Bangalore)
Bengaluru, IND
Member Technical Staff 2 - C/C++
Bengaluru, IND
Customer Support Representative II
Mumbai, IND
Treasury Manager, Capital Markets
, USA
Manager, Convening Event Experiences
New York City, USA