greenhouse
Lead Python Developer
iCapital
Jaipur, IND
Lead · 10+ years experience
Discovered 1 weeks ago
airflowawsaws-lambdacloudformationcloudwatchdocker
Free
Job Fit Check
Base Career helps you apply smarter for this job.
?%
Ready to ScanKey skills for this role
airflowawsaws-lambda
Key Skills for This Role
airflowawsaws-lambdacloudformationcloudwatchdocker
Full Job Posting
Responsibilities
- Assist with the design and implementation of advanced scraping solutions using Python, Playwright, and AWS services.
- Automate interactions with JavaScript-heavy and authentication-secured websites, including handling MFA, CAPTCHAs, and session or token-based login flows.
- Architect scraping pipelines using serverless AWS components such as Lambda, Step Functions, S3, CloudWatch, and Secrets Manager.
- Build systems that scale to support high volumes of data extraction with fault tolerance, retries, and intelligent logging.
- Integrate and manage complex workflows across multiple portals, APIs, and data sources.
- Contribute to architectural decisions, tooling, and best practices.
- Responsible for Code reviews and deployments.
Qualifications
- Bachelor’s or master’s degree in computer science, software engineering, or related field, or equivalent professional experience
- 10+ years of experience in Python development, with a strong focus on automation and data extraction
- Proven expertise in web scraping using tools like Playwright, Selenium, Scrapy, BeautifulSoup, and requests
- In-depth experience handling multi-step authentication flows, including multi-factor authentication (MFA), CAPTCHA solving, and session or cookie management
- Proficient in deploying and managing scraping workloads in AWS, particularly Lambda, S3, IAM, CloudWatch, and Secrets Manager
- Experience with asynchronous programming, headless browsers, and JavaScript-rendered content
- Solid understanding of web protocols (i.e. HTTP, HTTPS, cookies, headers), and the ability to reverse-engineer network calls and authentication mechanisms
- Comfortable working with APIs, JSON/XML, and data transformation
- Experience with CI or CD pipelines, Docker, and infrastructure as code (i.e. CloudFormation, Terraform)
- Familiar with data pipeline orchestration tools such as Apache Airflow or AWS Step Functions
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at iCapital
HR Total Rewards Operations - Assistant Vice President
Greenwich, USA
Executive
About the Role The Assistant Vice President, HR Operations for Total Rewards will serve as the People team’s key operational and technology partner for global Total Rewards programs, including benefits, payroll, retireme
Discovered TodayView →
HR Total Rewards Operations - Assistant Vice President
Salt Lake City, USA
Executive
About the Role The Assistant Vice President, HR Operations for Total Rewards will serve as the People team’s key operational and technology partner for global Total Rewards programs, including benefits, payroll, retireme
Discovered TodayView →
Regulatory and Compliance, Subscription Document Review - Associate
New York City, USA
Junior
Discovered TodayView →
Fund Reporting - Analyst
Boca Raton, USA
Discovered Today
Corporate Development & Strategy - Vice President
New York City, USA
Discovered Today
HR Total Rewards Operations - Assistant Vice President
Greenwich, USA
Discovered Today
HR Total Rewards Operations - Assistant Vice President
Salt Lake City, USA
Discovered Today
Event Operations Coordinator - Associate
Lisbon, USA
Discovered Today
Regulatory and Compliance, Subscription Document Review - Associate
New York City, USA
Discovered Today
Data Engineer - Vice President
New York City, USA
Discovered Today
Quant Engineer- Assistant Vice President
New York City, USA
Discovered Today
