Data Engineer
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
We are seeking an ambitious Data and AI Engineer with 5-8 years of experience to play a pivotal role in modernizing client’s core data infrastructure and scaling advanced AI capabilities. This role bridges legacy big data environment and future cloud platform. You will actively maintain and optimize our existing data pipelines while architecting, developing, and transitioning workflows to an AI-ready cloud ecosystem. Beyond traditional data pipelines, you will play an active role in building and deploying intelligent AI agents and leveraging advanced Large Language Models (LLMs) like Anthropic Claude, utilizing the cutting-edge Databricks AI suite to deliver immediate business value.
Key Skills for This Role
Full Job Posting
Responsibilities
- ROLES AND RESPONSIBILITIES: • Cloud Modernization & Migration: Deconstruct legacy Apache Spark and Hadoop MapReduce workflows to re-architect and rebuild them as optimized, production-ready pipelines within AWS and Databricks. • AI Agent & LLM Development: Design, build, and deploy intelligent AI agents and workflow automation tools leveraging leading Large Language Models (LLMs) such as Anthropic Claude. • AI Data Pipeline Engineering: Build and optimize pipeline architectures explicitly tailored for AI use cases, including unstructured data ingestion, real-time feature tokenization, and metadata tagging for vector databases. • Legacy Infrastructure Maintenance: Monitor, maintain, and troubleshoot existing big data workloads running on our Hadoop cluster to guarantee data availability for business operations during the multi-phase migration, resolving bottlenecks and Out Of-Memory (OOM) errors. • BI Engineering & Support: Act as the primary engineering liaison for downstream business stakeholders utilizing BI tools (e.g. Tableau, Looker, etc.) by performing minor functional enhancements, bug fixes, and data extract optimizations to resolve report dashboard latency. • Cloud Optimization: Utilize Databricks and Delta Lake features (e.g., ACID transactions, Z-Ordering, caching) to significantly improve pipeline performance, reliability, and cost efficiency. • Databricks AI Suite Implementation: Leverage Databricks tools (such as Databricks Vector Search, Mosaic AI, and Lakeflow) to orchestrate, track, and serve productio
Qualifications
- AI & Agentic Frameworks: Hands-on experience or deep technical familiarity building functional AI agents, integrating LLM APIs (specifically Anthropic Claude), and utilizing orchestration frameworks (e.g., LangChain, Databricks Mosaic AI Agent Framework or any other tool ). • Distributed Computing: Foundational understanding of distributed storage and computing concepts—specifically partitioning, shuffling, caching, and broadcast joins. Solid hands-on experience with Apache Spark is required. • Programming & SQL: Strong proficiency in Python (PySpark) or Scala, alongside intermediate-to-advanced SQL querying capabilities (window functions, query tuning, and complex joins). • Cloud & Databricks Exposure: Direct experience or deep theoretical knowledge of the AWS ecosystem (S3, IAM) and Databricks environments. • Visualization Layer: Practical experience working with any BI tool (e.g. Tableau, Looker, Power BI, etc.) with the capability to debug calculated fields, modify parameters, and troubleshoot slow-loading reports.
About EXL
EXL is a global data and AI company that provides analytics, digital operations, and industry-specific services to enterprises in insurance, healthcare, banking, retail, media, and energy.
Visit company websiteApply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at EXL
CRM Manager (Planning), 1:1 Fan Engagement Marketing
, CAN
Position: Production Manager, 1:1 Fan Engagement Marketing Location: Remote in US and Canada Salary: $105,000-$125,000 + bonus + incentives The posted range is the hiring range for this role — a subset of the broader ran
Lead Assistant Manager_Premium Accounting Services
Noida, IND
Lead Assistant manager for Order to Cash role
Product Coordinator - MarTech Platform and Governance
, CAN
• The Product Coordinator - MarTech Platforms & Governance plays a critical role in supporting the operational scale and governance of the organization's multi-tenant MarTech ecosystem. This role is responsible for manag
Senior Specialist - UI/UX Front End Developer
, CAN
We are currently seeking an experienced Senior Specialist with 6 to 9 years of professional experience bridging modern UI/UX engineering with Adobe Experience Manager (AEM) and Adobe Edge Delivery Services (EDS). The ide
Executives
, IND
The employer is looking for an entry-level operations professional to manage customer orders, delivery enquiries, returns, and account administration. The role supports store and international customer queries while main
CRM Manager (Planning), 1:1 Fan Engagement Marketing
, CAN
Lead Data Engineer
London, GBR
Lead Assistant Manager_Premium Accounting Services
Noida, IND
Product Coordinator - MarTech Platform and Governance
, CAN
Senior Specialist - UI/UX Front End Developer
, CAN
Executive
Noida, IND
Lead Data Engineer
, GBR
Executives
, IND