company_site
Data Engineer - Tech Lead (Databricks, Pyspark)
EPAM Systems, Inc.
London, GBR
Full-time
Senior
Hybrid
Discovered 2 weeks ago
Azure DatabricksPySparkDelta Live TablesDelta LakeApache IcebergPython
Free
Job Fit Check
Base Career helps you apply smarter for this job.
?%
Ready to ScanKey skills for this role
Azure DatabricksPySparkDelta Live Tables
Key Skills for This Role
Azure DatabricksPySparkDelta Live TablesDelta LakeApache IcebergPython
Full Job Posting
Responsibilities
- Lead the architecture, design and build of large-scale data platforms using Azure Databricks and modern cloud technologies
- Implement and optimize ETL workflows and streaming pipelines with PySpark and Delta Live Tables following Lakehouse principles
- Enhance performance, manage cloud costs and ensure platform reliability for structured streaming workloads
- Define data governance, security and quality standards to maintain consistency across the platform
- Collaborate with stakeholders to translate complex business requirements into actionable technical solutions
- Develop integration approaches using Azure-native services such as Data Factory, Synapse and Blob Storage
- Mentor data engineers, promote modern engineering practices and perform technical reviews
- Drive adoption of CI/CD, Infrastructure as Code and automated testing in data engineering environments
- Implement observability and monitoring using tools like Databricks Workflows and related frameworks
- Contribute to AI-driven initiatives by leveraging Databricks ML/MosaicML to integrate Generative AI and LLM-based solutions
Requirements
- Bachelor’s or Master’s degree in Computer Science, Software Engineering or related field
- Extensive experience designing and implementing production-grade platforms using Azure Databricks
- Expertise in PySpark, including advanced optimization, data skew mitigation and query tuning
- Strong programming skills in Python with knowledge of modern software design principles
- Practical experience with structured streaming, Delta Lake and Delta Live Tables
- Proven experience in Lakehouse migration and modernization using open table formats such as Delta Lake or Apache Iceberg
- Proficiency with cloud-native services on Azure and knowledge of multi-cloud environments (AWS or GCP)
- Hands-on experience with CI/CD and Infrastructure as Code tools (Terraform, GitHub Actions, Jenkins)
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at EPAM Systems, Inc.
Director, Retail Media Consulting
, USA
Discovered 1 weeks agoFull-time
VP, Southeast Market Leader
, USA
Discovered 1 weeks agoFull-time
Senior Director, Health Care Client Partner
, USA
Discovered 1 weeks agoFull-time
Director, Client Partner
, USA
Discovered 1 weeks agoFull-time
Associate Content Strategy Manager - AI Strategy
, USA
Discovered 1 weeks agoFull-time
Senior PBAC Engineer
, USA
Discovered 1 weeks agoFull-time
Director, Digital Engagement
, USA
Discovered 1 weeks agoFull-time
Media & Entertainment — Data & AI Consultant
, USA
Discovered 1 weeks agoFull-time