eightfold
Data Engineer - Assistant Vice President
Citi
Pune, IND
Full-time
Mid · 3+ years experience
Hybrid
Discovered 1 weeks ago
PythonPySparkSpark SQLDatabricksDelta LakeDatabricks Unity Catalog
Free
Job Fit Check
Base Career helps you apply smarter for this job.
?%
Ready to ScanKey skills for this role
PythonPySparkSpark SQL
Key Skills for This Role
PythonPySparkSpark SQLDatabricksDelta LakeDatabricks Unity Catalog
Full Job Posting
Key Responsibilities
- Data Pipeline Architecture & Development: Design, develop, and deploy scalable batch and real-time end-to-end data pipelines (ETL/ELT) using Python, PySpark, Spark SQL, and Databricks(Must have).
- Cloud Infrastructure Integration: Deploy and maintain Databricks workspaces on cloud environments (AWS or GCP). Manage secure integrations with cloud storage (S3/GCS), access controls (IAM), secrets management (Vault/KMS), and serverless query engines.
- Performance Optimization & Tuning: Diagnose and resolve performance bottlenecks in Spark clusters, SQL queries, and Databricks jobs. Optimize storage layouts using Delta Lake properties (e.g., Z-Ordering, partitioning, and vacuuming).
- Data Quality & Governance: Implement automated data validation frameworks, data quality monitoring, and metadata management solutions utilizing Databricks Unity Catalog to ensure strict compliance with internal data governance policies and external financial regulations (such as BCBS 239).
- Technical Leadership & Mentorship: Act as a technical lead within an Agile/Scrum environment. Lead peer code reviews, enforce coding standards, and mentor junior and mid-level data engineers (C10/C11).
- DevOps & CI/CD: Establish and maintain automated CI/CD pipelines (using Jenkins, GitLab, or GitHub Actions) for packaging and deploying data engineering artifacts (dbt, Spark jobs, Databricks workflows).
- Collaboration: Partner with Data Science teams to operationalize machine learning models, and work with business intelligence developers to build efficient semantic layers for reporting.
Technical Qualifications (Must-Haves)
- Programming Languages: Strong, production-grade proficiency in Python (including standard libraries, pandas, and testing frameworks like pytest) and advanced SQL (including window functions, CTEs, and query optimization).
- Distributed Computing: Deep hands-on experience with Apache Spark (PySpark) for processing multi-terabyte datasets in a distributed cluster environment.
- Unified Lakehouse Platforms: Minimum 3 years of hands-on experience developing within Databricks . Expert knowledge of Delta Lake ACID transactions, Delta Live Tables (DLT), Unity Catalog, and Databricks Workflows is required.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Citi
Senior QA Automation Engineer - Assistant Vice President
Pune, IND
SeniorFull-time
Discovered TodayView →
Project Manager -Scrum Master - Assistant Vice President
Pune, IND
SeniorFull-time
Discovered TodayView →
Oracle EBS P2P Senior App Developer - Assistant Vice President
Pune, IND
SeniorFull-time
Discovered TodayView →
IT Business Lead Analyst
London, GBR
Discovered TodayFull-time
Senior QA Automation Engineer - Assistant Vice President
Pune, IND
Discovered TodayFull-time
Project Manager -Scrum Master - Assistant Vice President
Pune, IND
Discovered TodayFull-time
Oracle EBS P2P Tech Lead - Vice President
Pune, IND
Discovered TodayFull-time
Mainframe Tech Lead Analyst
Chennai, IND
Discovered TodayFull-time
Lead Full Stack Java Engineer - Vice President
Chennai, IND
Discovered TodayFull-time
Applications Support Intermediate Analyst
Pune, IND
Discovered TodayFull-time
Oracle EBS P2P Senior App Developer - Assistant Vice President
Pune, IND
Discovered TodayFull-time