Senior Databricks Data Engineer
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
Develop and maintain scalable data pipelines and transformation workflows using Databricks, PySpark, and SQL. Support the migration of datasets, pipelines, and transformation lo.
Key Skills for This Role
Responsibilities
- Develop and maintain scalable data pipelines and transformation workflows using Databricks, PySpark, and SQL.
- Support migration of datasets, pipelines, and transformation logic from Palantir Foundry to Databricks Delta Lake.
- Work with investment and financial datasets, including market data, portfolio positions, transactions, pricing data, and risk metrics.
- Build and maintain ETL/ELT pipelines for investment analytics, reporting, and portfolio management systems.
- Implement data validation, reconciliation, and quality frameworks to ensure financial data accuracy and consistency.
- Optimize Spark jobs, cluster configurations, and storage formats to improve performance and cost efficiency.
- Maintain data lineage, documentation, and governance practices to meet financial industry standards.
- Support monitoring, troubleshooting, and performance tuning of production data pipelines.
Requirements
- Advanced Databricks experience (architecture, Delta Lake, job orchestration, cluster management, performance tuning).
- Experience working with investment data (market data, portfolio holdings, transactions, pricing data, risk metrics, financial instruments).
- Expert level PySpark and Python skills.
- Advanced SQL and data modeling expertise (dimensional modeling, lakehouse architecture, query optimization).
- Cloud platform experience (Azure preferred).
- Nice to have: Strong hands on expertise in Palantir Foundry.
- Nice to have: Proven migration experience from Palantir to Databricks.
- Nice to have: Familiarity with Dynatrace or Datadog.
- Nice to have: Databricks certification, cloud certifications (Azure/AWS), or enterprise data architecture certifications.
Full Job Posting
Responsibilities
- Develop and maintain scalable data pipelines and transformation workflows using Databricks, PySpark, and SQL.
- Support the migration of datasets, pipelines, and transformation logic from Palantir Foundry to Databricks Delta Lake.
- Work with investment and financial datasets, including market data, portfolio positions, transactions, pricing data, and risk metrics.
- Build and maintain ETL/ELT pipelines that enable data availability for investment analytics, reporting, and portfolio management systems.
- Implement data validation, reconciliation, and quality frameworks to ensure financial data accuracy and consistency.
- Optimize Spark jobs, cluster configurations, and storage formats to improve performance and cost efficiency.
- Maintain data lineage, documentation, and governance practices to meet financial industry standards.
- Support monitoring, troubleshooting, and performance tuning of production data pipelines.
Must have
- Advanced Databricks Experience. Deep knowledge of Databricks architecture, Delta Lake, job orchestration, cluster management, and performance tuning.
- Experience working with investment data, such as market data, portfolio holdings, transactions, pricing data, risk metrics, and financial instruments
- Expert Level PySpark & Python Skills. Strong ability to design, optimize, and refactor distributed data processing workflows.
- Advanced SQL & Data Modelling Expertise. Experience in dimensional modeling, lakehouse architecture patterns, and query optimization.
- Cloud Platform Experience (Azure preferred). Hands on experience deploying and managing data platforms in cloud environments, including storage, security, and networking considerations.
Nice to have
- Strong Hands on Expertise in Palantir Foundry. Proven experience with Foundry pipelines, ontologies, data lineage, transformations, and platform governance.
- Proven Migration Experience from Palantir / to Databricks. Demonstrated experience leading or executing platform migrations, including pipeline conversion, data model redesign, and production cutover.
- Familiarity with Dynatrace or Datadog for system observability and monitoring.
- Databricks certification, cloud certifications (Azure/AWS), or enterprise data architecture certifications.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Luxoft
Technical Architect & Engineering Lead - Abu Dhabi, United Arab Emirates
Abu Dhabi Emirate, UAE
Luxoft seeks a hands-on Technical Architect & Engineering Lead to join IBM Technology feature teams in Abu Dhabi. You will build cloud-native solutions from scratch, drive engineering excellence with DDD, TDD, and Hexago
DataOps Consultant, Lakehouse Support - Abu Dhabi, United Arab Emirates
Abu Dhabi, UAE
Luxoft is seeking a DataOps Consultant for L2 Lakehouse Support in an investment management environment. The role involves incident management, operational diagnostics, and root cause analysis for batch and streaming dat
Murex FO Support Consultant
Abu Dhabi, UAE
The Murex FO support analyst will be required to be part of GM IT FO Squad responsible to provide IT services to Global Markets team. The squad is responsible for support, chang...
DataOps Consultant, Lakehouse Support
Abu Dhabi Emirate, UAE
Luxoft seeks a DataOps Consultant for L2 Lakehouse Support in an investment management environment. The role involves incident management, diagnostics across data pipelines, and coordination with engineering teams. Requi
Murex FO Support Consultant
Abu Dhabi, UAE
Luxoft is seeking a Murex FO Support Consultant to provide Level 1/Level 2 trading floor support to traders and control groups in Abu Dhabi. The role requires 7-10 years of Murex front office support experience, expertis
Senior Solution Architect
Abu Dhabi, UAE
A senior architect responsible for end-to-end architecture of all IBM Technology projects spanning vendor-based, in-house, business-led, and data/AI initiatives. The role owns a...
Principal Full-Stack Architect (.NET & Cloud)
Abu Dhabi, UAE
Project description The Principal Full-Stack Architect (.NET & Cloud) is a senior technical leader responsible for defining and driving the architecture of enterprise-scale appl...
Murex Credit Risk/MLC Business Analyst
Saudi Arabia, KSA
Project description Large bank in Saudi is looking for Credit risk BA for murex greenfield implementation. We require Credit Risk BA to help implement the changes and projects a...
Technical Architect & Engineering Lead - Abu Dhabi, United Arab Emirates
Abu Dhabi Emirate, UAE
DataOps Consultant, Lakehouse Support - Abu Dhabi, United Arab Emirates
Abu Dhabi, UAE
Murex FO Support Consultant
Abu Dhabi, UAE
DataOps Consultant, Lakehouse Support
Abu Dhabi Emirate, UAE
Murex FO Support Consultant
Abu Dhabi, UAE
Senior Solution Architect
Abu Dhabi, UAE
Principal Full-Stack Architect (.NET & Cloud)
Abu Dhabi, UAE
Murex Credit Risk/MLC Business Analyst
Saudi Arabia, KSA