Data Engineer - Lead

a) Mandatory Skills

o Databricks

o Apache Spark

o Python and SQL

8+ years strong hands-on experience with:

Experience building scalable ETL/ELT data ingestion, Transformation and curation pipelines

• Strong understanding of Delta Lake (ACID, schema evolution, versioning)

• Experience with Lakehouse architecture (ingestion + curated layers)

• Ability to implement source-to-target mappings and physical schemas

• Experience implementing data quality, validation, and monitoring frameworks

• Strong understanding of distributed data processing

• Experience with CI/CD, Git, and pipeline automation

• Strong banking/financial services domain experience

• Ability to deliver production-grade, scalable data products end-to-end

• Experience working in Agile, cross-functional teams

• Great interpersonal and communication skills

• Can work independently with minimum supervision

• Flexible and adaptable working style to work with multiple stakeholders

• Open to work in UK Shift (10 AM to 9 PM IST with 8 hours of productive work)

b) Good to Have Skills

• Exposure to Snowflake (hybrid environments)

• Knowledge of Scala

• Experience with data governance and catalog tools

• Exposure to workflow orchestration tools (Airflow, etc.)

• Experience in regulated environments

• Familiarity with financial domains (Payments, FX, Accounts)

• Experience in data product-oriented delivery approaches

• AI-assisted data product development experience