{bc}
Apply
workday

Consultant/Sr. Consultant, Consumer Analytics Data Engineering

Lilly
Bengaluru, IND
Full-time
Senior · 3+ years experience
Onsite
Discovered 2 days ago
DatabricksAWSUnity CatalogDelta Live TablesDatabricks WorkflowsDatabricks SQL
Free

Job Fit Check

Base Career helps you apply smarter for this job.

?%
Ready to Scan

Key skills for this role

DatabricksAWSUnity Catalog
Smart Apply

Full Job Posting

Job Responsibilities

  • Design, develop, and implement scalable ETL/ELT pipelines for extracting, transforming, and loading consumer data from various sources (e.g., CRM, marketing platforms, DCM, GA4, digital channels), with Databricks/AWS as the primary execution platform.
  • Develop and manage end-to-end solutions on Databricks including Unity Catalog, Delta Live Tables, Databricks Workflows, Databricks SQL, etc.; own platform governance covering schemas, permissions, and data lineage.
  • Design multi-hop lakehouse architectures (Bronze / Silver / Gold) using Delta Lake; optimize Spark compute, cluster configurations, and Auto Loader for performance and cost efficiency.
  • Leverage AWS data services — S3, Glue, Lambda and Redshift — in conjunction with Databricks to build reliable, end-to-end consumer data flows.
  • Architect and optimize data models and schemas to support complex analytical queries and reporting requirements related to consumer behaviour, preferences, and engagement.
  • Publish semantic layers (metrics definitions, certified datasets, business logic) consumed by downstream BI tools and AI agents; build and deploy agentic workflows using Databricks AI Functions or similar frameworks.
  • Ensure data quality, integrity, and governance across all consumer data assets by implementing validation rules, schema evolution controls, and monitoring processes through Unity Catalog.
  • Collaborate with data scientists, business analysts, and marketing teams to understand data needs and translate them into technical data engineering solutions; partner to productionize ML models and feature stores on Databricks.
  • Implement automation for data ingestion, processing, and delivery with a focus on efficiency, reliability, and SLA adherence.
  • Troubleshoot and resolve data-related issues, performing root cause analysis and implementing corrective actions.
  • Stay current with Databricks platform updates, AWS data services, and emerging best practices in data engineering and AI-driven analytics.

Job Qualifications

  • Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or a related quantitative field.
  • 3-8 years of data engineering experience with hands-on production experience on Databricks.
  • Deep knowledge of Databricks platform architecture — Unity Catalog, Delta Lake, Databricks Workflows, Databricks SQL, and cluster/compute management. (Preferred)
  • Experience designing lakehouse architectures (medallion/multi-hop patterns) at scale and with ETL/ELT orchestration tools. (Preferred)
  • Experience with AWS data stack: S3, Glue, Redshift, Lambda, and IAM.
  • Strong proficiency in Python and PySpark; advanced SQL for complex analytical workloads.
  • Familiarity with Software Development Life Cycle (SDLC) practices — including version control (Git), CI/CD pipelines, code reviews, and agile development methodologies. (Preferred)
  • Strong understanding of data warehousing concepts, dimensional modeling, and data governance principles.
  • Hands-on experience building semantic layers (e.g., dbt metrics, Databricks AI/BI semantic layer) and creating AI agents or agentic pipelines (Databricks AI Functions etc.) .
  • Excellent problem-solving, communication, and stakeholder collaboration skills.
  • Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form ( https://careers.lilly.com/us/en/workplace-accommodation ) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.
  • Lilly does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.

Apply for this job in 1 click

Skip the repetitive application forms

Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.

Sarah M.James T.Maya R.

Trusted by over 500,000 job seekers on Base Career

Start Free Today

More from this employer

More jobs at Lilly

AI Solution Architect - MQ DIA

Bengaluru, IND

Discovered 2 days agoFull-time

Technical Lead Architect - CTI MD Tech (GRA/GSC)

Bengaluru, IND

Discovered 6 days agoFull-time

DSM-Biomeds

Delhi, IND

Discovered 1 weeks agoFull-time

Advisor - Software Engineer - SPE

Hyderabad, IND

Discovered 1 weeks agoFull-time

Data Analyst – Consultant/Senior Consultant

Bengaluru, IND

Discovered 1 weeks agoFull-time

Sales Consultant Cardio Metabolic Health- Moncton/Northern NB

Moncton, CAN

Discovered 2 weeks agoFull-time

PR&D – Technical Lead Architect – Semantic Layer & Agentic AI

Bengaluru, IND

Discovered 2 weeks agoFull-time

CS&D Technical Lead Architect - CTI MD Tech (LRL)

Bengaluru, IND

Discovered 2 weeks agoFull-time