Senior Databricks Data Engineer
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
Luxoft seeks a Senior Data Engineer with strong Databricks and PySpark expertise to build scalable data pipelines and lakehouse models for investment analytics. Requires 5+ years experience, production Databricks experience, and financial domain knowledge is a plus.
Key Skills for This Role
Responsibilities
- Build, optimize, and maintain end to end data pipelines using Databricks, PySpark, and SQL
- Develop and manage Declarative Pipelines (e.g., Lakeflow / DLT style pipelines) for scalable incremental processing
- Implement robust transformations and modelling patterns to deliver trusted datasets for downstream consumption
- Implement data quality validation, monitoring, reconciliation, and alerting
- Debug pipeline failures, resolve production incidents, and improve pipeline stability and performance
- Contribute to design and evolution of data models supporting investment footprint
- Partner with business stakeholders to translate requirements into implementable data solutions
- Follow and enhance engineering standards for version control, CI/CD, testing, documentation
- Optimize compute and storage usage through partitioning/clustering strategies and performance tuning
Requirements
- Bachelor's degree in Computer Science, Engineering, Information Systems, or related discipline
- 5+ years experience in data engineering roles
- Strong hands on experience with Databricks in production environments
- Strong programming experience with PySpark and SQL
- Proven experience with Declarative Pipelines / pipeline orchestration on Databricks
- Experience delivering production ready datasets with operational controls
- Experience with modern Lakehouse concepts (Delta tables, optimization strategies)
Full Job Posting
Project Description
- We are seeking a Senior Data Engineer with strong hands on expertise in Databricks, PySpark, and cloud based data platforms to support the development, migration, and optimization of our enterprise data platform within the investment domain.
- This role will focus on building and maintaining scalable data pipelines and lakehouse data models that support investment analytics, portfolio management, risk analysis, and trading data workflows.
Responsibilities
- Build, optimize, and maintain end to end data pipelines using Databricks, PySpark, and SQL across ingestion, curation, and consumption layers.
- Develop and manage Declarative Pipelines (e.g., Lakeflow / DLT style pipelines) to support scalable incremental processing and operational reliability.
- Implement robust transformations and modelling patterns to deliver trusted datasets for downstream consumption.
- Implement data quality validation, monitoring, reconciliation, and alerting to ensure datasets meet required standards.
- Debug pipeline failures, resolve production incidents, and continuously improve pipeline stability, performance, and cost efficiency.
- Apply best practices around auditability, lineage, and data correctness.
- Contribute to the design and evolution of data models supporting the organization’s investment footprint.
- Partner with business stakeholders to translate requirements into implementable data solutions.
- Support integration of multi vendor and internal data sources into curated datasets.
- Follow and enhance engineering standards for version control, CI/CD, testing, documentation, and secure development practices.
- Optimize compute and storage usage through partitioning/clustering strategies, incremental patterns, and performance tuning.
- Contribute reusable libraries, patterns, templates, and approaches that improve delivery speed and consistency.
Mandatory Skills Description
- Bachelor’s degree (Computer Science, Engineering, Information Systems, or related discipline).
- 5+ years experience in data engineering roles.
- Strong hands on experience with Databricks in production environments.
- Strong programming experience with PySpark and strong SQL.
- Proven experience with Declarative Pipelines / pipeline orchestration on Databricks.
- Strong understanding of data engineering fundamentals: ingestion patterns, transformation design, incremental processing, testing, performance tuning.
- Experience delivering production ready datasets with appropriate operational controls.
- Experience with modern Lakehouse concepts (Delta tables, optimization strategies, file skipping, metadata/statistics awareness).
- Exposure to data governance practices: cataloguing, documentation, business glossary/terms, lineage.
- Experience working in enterprise environments with CI/CD pipelines and structured release processes.
- Familiarity with vendor market data feeds (e.g., Bloomberg, Refinitiv, MSCI, FactSet) or similar multi source mastering patterns.
Nice to Have Skills
- Strong Hands on Expertise in Palantir Foundry.
- Proven Migration Experience from Palantir to Databricks.
- Familiarity with Dynatrace or Datadog for system observability and monitoring.
- Databricks certification, cloud certifications (Azure/AWS), or enterprise data architecture.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Luxoft
Treasury Front Office Consultant
Dubai, UAE
Luxoft is seeking a Treasury Front Office Consultant to upgrade treasury applications and provide trading floor support. The role requires 7-10 years of experience in front office support with expertise in exotic derivat
Technical Architect & Engineering Lead - Abu Dhabi, United Arab Emirates
Abu Dhabi Emirate, UAE
Luxoft seeks a hands-on Technical Architect & Engineering Lead to join IBM Technology feature teams in Abu Dhabi. You will build cloud-native solutions from scratch, drive engineering excellence with DDD, TDD, and Hexago
DataOps Consultant, Lakehouse Support - Abu Dhabi, United Arab Emirates
Abu Dhabi, UAE
Luxoft is seeking a DataOps Consultant for L2 Lakehouse Support in an investment management environment. The role involves incident management, operational diagnostics, and root cause analysis for batch and streaming dat
Murex FO Support Consultant
Abu Dhabi, UAE
Luxoft is looking for a Murex FO Support Analyst to join the GM IT FO Squad in Abu Dhabi. You will provide level 1/2 trading floor support to traders and control groups, configure Murex static data, and work on patch upg
DataOps Consultant, Lakehouse Support
Abu Dhabi Emirate, UAE
Luxoft seeks a DataOps Consultant for L2 Lakehouse Support in an investment management environment. The role involves incident management, diagnostics across data pipelines, and coordination with engineering teams. Requi
Murex FO Support Consultant
Abu Dhabi, UAE
Luxoft is seeking a Murex FO Support Consultant to provide Level 1/Level 2 trading floor support to traders and control groups in Abu Dhabi. The role requires 7-10 years of Murex front office support experience, expertis
Senior Solution Architect
Abu Dhabi, UAE
A senior architect responsible for end-to-end architecture of all IBM Technology projects spanning vendor-based, in-house, business-led, and data/AI initiatives. The role owns a...
Principal Full-Stack Architect (.NET & Cloud)
Abu Dhabi, UAE
Project description The Principal Full-Stack Architect (.NET & Cloud) is a senior technical leader responsible for defining and driving the architecture of enterprise-scale appl...
Treasury Front Office Consultant
Dubai, UAE
Technical Architect & Engineering Lead - Abu Dhabi, United Arab Emirates
Abu Dhabi Emirate, UAE
DataOps Consultant, Lakehouse Support - Abu Dhabi, United Arab Emirates
Abu Dhabi, UAE
Murex FO Support Consultant
Abu Dhabi, UAE
DataOps Consultant, Lakehouse Support
Abu Dhabi Emirate, UAE
Murex FO Support Consultant
Abu Dhabi, UAE
Senior Solution Architect
Abu Dhabi, UAE
Principal Full-Stack Architect (.NET & Cloud)
Abu Dhabi, UAE