company_site

Senior Data Engineer (Investment Data)

Luxoft

Abu Dhabi, UAE

Full-time

Senior · 5+ years experience

Discovered Today

• Bachelor’s degree (Computer Science, Engineering, Information Systems, or related discipline). • 5+ years experience in data engineering roles (flexible based on depth of capability). • Strong hands-on experience with Databricks in production environments (prerequisite). • Strong programming experience with PySpark (must) and strong SQL (must). • Proven experience with Declarative Pipelines / pipeline orchestration on Databricks (prerequisite). • Strong understanding of data engineering fundamentals: ingestion patterns, transformation design, incremental processing, testing, performance tuning. • Experience delivering production-ready datasets with appropriate operational controls (monitoring, troubleshooting, reliability patterns). • Experience with modern Lakehouse concepts (Delta tables, optimization strategies, file skipping, metadata/statistics awareness). • Exposure to data governance practices: cataloguing, documentation, business glossary/terms, lineage. • Experience working in enterprise environments with CI/CD pipelines and structured release processes. • Familiarity with vendor market data feeds (e.g., Bloomberg, Refinitiv, MSCI, FactSet) or similar multi-source mastering patterns.

Apply

Free

Job Fit Check

Base Career helps you apply smarter for this job.

Ready to Scan

Key skills for this role

Smart Apply

Full Job Posting

Responsibilities

Participate in requirements clarification and sprint planning sessions. • Design technical solutions and implement them, inc ETL Pipelines – Build robust data pipelines in PySpark to extract, transform, using PySpark • Optimize ETL Processes – Enhance and tune existing ETL processes for better performance, scalability, and reliability • Writing unit and integration tests. • Support QA teammates in the acceptance process. • Resolving PROD incidents as a 3rd line engineer.

Skills

Bachelor’s degree (Computer Science, Engineering, Information Systems, or related discipline). • 5+ years experience in data engineering roles (flexible based on depth of capability). • Strong hands-on experience with Databricks in production environments (prerequisite). • Strong programming experience with PySpark (must) and strong SQL (must). • Proven experience with Declarative Pipelines / pipeline orchestration on Databricks (prerequisite). • Strong understanding of data engineering fundamentals: ingestion patterns, transformation design, incremental processing, testing, performance tuning. • Experience delivering production-ready datasets with appropriate operational controls (monitoring, troubleshooting, reliability patterns). • Experience with modern Lakehouse concepts (Delta tables, optimization strategies, file skipping, metadata/statistics awareness). • Exposure to data governance practices: cataloguing, documentation, business glossary/terms, lineage. • Experience working in enterprise environments with CI/CD pipelines and structured release processes. • Familiarity with vendor market data feeds (e.g., Bloomberg, Refinitiv, MSCI, FactSet) or similar multi-source mastering patterns.