AWS Data Engineer
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
We are seeking a highly skilled AWS Data Engineer with deep expertise in AWS cloud architecture, big data processing, real-time streaming, and modern data lake technologies. The ideal candidate will have strong hands-on experience in Spark (PySpark), Iceberg, EMR, Starburst/Trino, and event-driven architectures, along with experience building real-time and API-driven data applications who can design and build generic solutions for one of our Fortune 500 Client programs in the realm of Financial Master & Reference Data Management. This is high visibility, fast-paced key initiative will integrate data across internal and external sources, provide analytical insights, and integrate with the customer’s critical systems.
Key Skills for This Role
Full Job Posting
Responsibilities
- Key Responsibilities
- Design and implement scalable, secure, and cost-optimized AWS data architectures.
- Develop and maintain ETL pipelines using AWS Lambda and AWS Glue ETL.
- Configure and manage AWS Glue Crawlers, Glue Data Catalog, and schema evolution.
- Build, optimize, and unit test applications on the Apache Spark framework using PySpark.
- Design and optimize data lakes using Apache Iceberg on AWS, including table compaction and Iceberg performance tuning.
- Work extensively with data formats such as Avro, Parquet, JSON, XML, and CSV.
- Orchestrate event-driven workflows using AWS Step Functions and Amazon EventBridge.
- Connect and integrate Starburst from Lambda and Glue ETL jobs for federated querying.
- Implement CI/CD pipelines for automated testing and deployment.
- Perform unit testing using PyTest, and performance tuning of Spark and Python applications
- Key Responsibilities - Design and implement scalable, secure, and cost-optimized AWS data architectures. - Develop and maintain ETL pipelines using AWS Lambda and AWS Glue ETL. - Configure and manage AWS Glue Crawlers, Glue Data Catalog, and schema evolution. - Build, optimize, and unit test applications on the Apache Spark framework using PySpark. - Design and optimize data lakes using Apache Iceberg on AWS, including table compaction and Iceberg performance tuning. - Work extensively with data formats such as Avro, Parquet, JSON, XML, and CSV. - Orchestrate event-driven workflows using AWS Step Functions and Amazon EventBridge. - Connect and integrate Starburst from Lambda and Glue ETL jobs for federated querying. - Implement CI/CD pipelines for automated testing and deployment. - Perform unit testing using PyTest, and performance tuning of Spark and Python applications
Qualifications
- Strong understanding of AWS architecture best practices, scalability, security, and cost optimization strategies.
- Strong hands-on experience with AWS services including Lambda, Glue ETL, Athena, S3, DynamoDB, Step Functions, EventBridge, SNS, and SQS.
- Deep experience in Apache Spark (PySpark/Scala) development, unit testing, and performance optimization.
- Strong Python programming skills using libraries such as pandas, requests, json, and awswrangler.
- Experience on Apache Kafka and Confluent Kafka.
- Experience designing and optimizing data lakes using Apache Iceberg, including compaction and Iceberg optimization techniques.
- - Strong understanding of AWS architecture best practices, scalability, security, and cost optimization strategies. - Strong hands-on experience with AWS services including Lambda, Glue ETL, Athena, S3, DynamoDB, Step Functions, EventBridge, SNS, and SQS. - Deep experience in Apache Spark (PySpark/Scala) development, unit testing, and performance optimization. - Strong Python programming skills using libraries such as pandas, requests, json, and awswrangler. - Experience on Apache Kafka and Confluent Kafka. - Experience designing and optimizing data lakes using Apache Iceberg, including compaction and Iceberg optimization techniques.
About EXL
EXL is a global data and AI company that provides analytics, digital operations, and industry-specific services to enterprises in insurance, healthcare, banking, retail, media, and energy.
Visit company websiteApply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at EXL
Lead Assistant Manager-Procure to Pay-Functional Lead-AP
Bengaluru, IND
Lead Assistant Manager-Auditors-Surgery
Chennai, IND
Experienced Quality Auditor – Lead and manage medical Coding professional with CPC certification and a minimum of 3+ years of post-certification coding experience. The ideal candidate must have strong expertise in IVR co
Senior Manager
Chennai, IND
Professionals in this group are responsible for monitoring the scrum processes and scrum meetings. They increase team efficiency, motivate their team, and argue for changes that ensure quality and timeliness. The project
Lead Assistant Manager
Noida, IND
• Should have very good understanding of US insurance and Insurance Finance • Should have worked for various LOB’s such as Auto, GL, Property, WC etc. • Should have fair amount of exposure on both Commercial and Personal
Lead Assistant Manager
Noida, IND
Perform monthly, quarterly, and annual close activities for External Reinsurance operations across multiple Property & Casualty (P&C) entities. Prepare and review GL, subledger, payment tracker, and profit share reconcil
Network Operation Specialist
Noida, IND
EXL is seeking a telecom network operations professional to manage enterprise capacity planning, demand forecasting, circuit commissioning, provider governance, cost optimization, and service delivery. The role supports
2868798-Assistant Manager
Bengaluru, IND
Lead Assistant Manager-Procure to Pay-Functional Lead-AP
Bengaluru, IND
4702090-Assistant Manager
Bengaluru, IND
Lead Assistant Manager-Auditors-Surgery
Chennai, IND
Senior Manager
Chennai, IND
Lead Assistant Manager
Noida, IND
Lead Assistant Manager
Noida, IND
Network Operation Specialist
Noida, IND