Python with Spark Developer (5.1-7 years)-Chennai
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
Role Overview
Capco is seeking a Python and PySpark Developer with 5–7 years of experience.
The role designs, develops, tests, and maintains high-performance data-processing pipelines using Python, PySpark, and SQL.
The developer transforms raw market, trade, and client data into trusted datasets supporting analytics, reporting, and machine-learning models for the CEFS franchise.
The role requires sound knowledge of Agile practices such as Scrum or Kanban.
Key Skills for This Role
Full Job Posting
About Capco
Capco is a global technology and management consulting firm serving clients across banking, financial services, insurance, payments, and energy.
The company describes its culture as innovative, inclusive, collaborative, and non-hierarchical, with ongoing learning opportunities.
Position Purpose
Capco is seeking a Python and PySpark Developer with 5–7 years of experience.
The role designs, develops, tests, and maintains high-performance data-processing pipelines using Python, PySpark, and SQL.
The developer transforms raw market, trade, and client data into trusted datasets supporting analytics, reporting, and machine-learning models for the CEFS franchise.
The role requires sound knowledge of Agile practices such as Scrum or Kanban.
Direct Responsibilities
- Design and develop ingestion, transformation, and enrichment pipelines with Python, PySpark, and SQL.
- Write and optimize complex SQL queries, analytical UDFs, and window functions for data aggregation and reporting.
- Translate functional requirements into technical specifications with data architects, data scientists, and business analysts.
- Unit-test, integration-test, and review code.
- Maintain Git, Jenkins, and Docker CI/CD pipelines for automated build, testing, and deployment.
- Monitor production workloads, troubleshoot performance and memory issues, investigate job failures, and document data lineage and operational runbooks.
Contributing Responsibilities
- Contribute to innovation, including AI and machine-learning initiatives, and recommend technical practices for efficiency improvements.
- Participate in sprint planning, daily standups, retrospectives, and backlog grooming.
- Mentor junior engineers and promote Python, Spark optimization, and data engineering best practices.
- Evaluate emerging technologies and deliver proof-of-concepts for CEFS.
Technical Competencies
- Strong experience with Python, including NumPy, Pandas, Python frameworks, RESTful APIs, and MS-SQL or Oracle.
- Strong PySpark experience with DataFrames, Spark SQL, Structured Streaming, partitioning, caching, broadcast joins, and performance tuning.
- Advanced SQL experience with complex queries, stored procedures, and query optimization.
- Experience with data cleaning, wrangling, analysis, visualization, ETL, code maintenance, bug fixing, and production support.
- Knowledge of Linux or Unix environments, shell scripting, testing, documentation, build tools, DevOps tools, Agile, Scrum, and data engineering environments.
- Experience with object-oriented, API, microservices, design patterns, and development principles.
- Front-end technology knowledge, preferably Flask, is desirable.
Education and Behavioral Skills
- Bachelor's degree or equivalent.
- Ability to understand complex systems and provide a practical way forward.
- Ability to learn and work across diverse technologies, languages, frameworks, and tools.
- Self-motivated approach, interpersonal skills, communication, coordination, attention to detail, and results orientation.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Capco
Support Analyst (7-9 yrs) Bengaluru
Bengaluru, IND
Data Engineer
Republic, USA
Java Full Stack Developer
Chennai, IND
AI Architect
Republic, USA
Data Architect
Republic, USA
Senior Analyst - RMG
Bengaluru, IND
IT Control Testing
, IND
BA TPRM- GCOO
Bengaluru, IND