The Lead Data Engineer, Specialist is responsible for designing and delivering scalable data platforms, cloud-native data pipelines, and analytics solutions. This is a hands-on technical leadership role requiring deep data engineering expertise, strong problem-solving skills, and the ability to lead a team of Data and AI Engineers.
The role requires a strong product mindset to understand business needs, translate them into sustainable technical solutions, and take end-to-end ownership of delivery, quality, reliability, and continuous improvement. The individual will collaborate with business and technology stakeholders to deliver secure, high-quality, and future-ready data solutions while contributing to modern AI-assisted engineering practices.
Key Skills for This Role
AWSDatabricksdbtPythonPySparkPandas
Full Job Posting
Role Summary
The Lead Data Engineer, Specialist is responsible for designing and delivering scalable data platforms, cloud-native data pipelines, and analytics solutions. This is a hands-on technical leadership role requiring deep data engineering expertise, strong problem-solving skills, and the ability to lead a team of Data and AI Engineers.
The role requires a strong product mindset to understand business needs, translate them into sustainable technical solutions, and take end-to-end ownership of delivery, quality, reliability, and continuous improvement. The individual will collaborate with business and technology stakeholders to deliver secure, high-quality, and future-ready data solutions while contributing to modern AI-assisted engineering practices.
Responsibilities
Lead and mentor a team of Data and AI Engineers, providing technical direction, coaching, and support.
Lead solution architecture and technical design, ensuring alignment with business objectives and product priorities.
Provide technical leadership through architecture reviews, design reviews, code reviews, and engineering best practices.
Design, develop, and optimize scalable data pipelines that ingest, process, and transform large datasets from diverse sources.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
Build cloud-based data platforms and storage solutions to support analytical and operational workloads.
Design and optimize data models, schemas, data quality controls, and governance frameworks.
Implement batch and real-time data processing solutions using modern orchestration and streaming technologies.
Partner with product owners, business stakeholders, analysts, architects, and engineering teams to deliver data-driven solutions.
Take end-to-end ownership of solution design, development, deployment, production reliability, and continuous improvement.
Evaluate technical options and make informed decisions considering performance, scalability, security, cost, and maintainability.
Implement CI/CD, automation, testing, observability, and DevOps practices to improve engineering productivity and platform reliability.
Apply AI-assisted development practices and modern engineering approaches to improve quality, efficiency, and innovation.
Support production operations, troubleshoot complex issues, perform root-cause analysis, and implement preventive actions.
Participate in application-support rotations and work in shifts or outside regular business hours when required during critical incidents, major releases, or other business-critical situations.
Qualifications and Skills
Bachelor’s degree in Computer Science, Information Technology, Engineering, Computer Applications, or a related discipline from a recognized institution.
Minimum of 8 years of relevant technology experience.
At least 4 years of hands-on experience designing and developing scalable data pipelines, ETL or ELT frameworks, and database or data platform solutions.
Demonstrated experience in leading technical solutions and guiding engineering teams through design, development, deployment, and production support.
Experience working in Agile product environments with strong ownership and accountability.
Must have Skills
Cloud Platforms: AWS, including S3, Glue, Lambda, Kinesis, Redshift, Athena, Step Functions, SNS/SQS, IAM, CloudWatch, and EMR
Data Engineering: Redshift, Databricks, dbt, Spark
Programming: Python, PySpark, Pandas, and Advanced SQL
Orchestration and Streaming: Airflow and Kafka
DevOps and Infrastructure: Docker, Terraform, GitHub Actions, CodePipeline, ECS, ECR, and CI/CD
Analytics and Integration: Power BI and REST APIs
Engineering Practices: Git, automated testing, monitoring, observability, data quality, and governance
Additional Requirements
Strong product mindset with the ability to connect technical decisions to business and customer outcomes.
Ability to lead technical solution design and drive delivery from concept through production.
Strong end-to-end ownership of deliverables, technical decisions, quality, reliability, and support.
Proven ability to lead, mentor, and develop engineers while remaining technically hands-on.
Strong analytical, problem-solving, troubleshooting, and debugging skills.
Effective communication and stakeholder management across technical and non-technical teams.
Ability to manage ambiguity, balance competing priorities, and proactively address risks and dependencies.
Willingness to support critical production incidents and application-support requirements outside regular working hours when needed.
Location
This role is based in Hyderabad, Telangana at Vanguard India. Only qualified external applicants will be considered.
Our mission
Vanguard adheres to a simple purpose: To take a stand for all investors, to treat them fairly, and to give them the best chance for investment success.
Our commitment to you
Vanguard takes the same long-term view of your success—at work and in life—with Benefits and Rewards packages that reflect what you care about, throughout all the phases and stages of your life. Our Total Rewards programs provide you and your loved ones with wellness support for key areas in your life:
Financial wellness We're committed to enabling your financial success and provide competitive offers and programs.
Physical wellness We're committed to providing benefits that support your physical health and wellness.
Personal wellness We're committed to providing resources that help support the full scope of your life.
How we work
Vanguard has implemented a hybrid working model for most of our employees (crew members), designed to capture the benefits of enhanced flexibility while enabling in-person learning, collaboration, and connection. We believe our mission-driven and highly collaborative culture is a critical enabler to support long-term client outcomes and enrich the employee experience.
