Software Engineer III

Responsibilities

• Spider Development: Design and deploy robust, distributed spiders and crawlers to extract data from a variety of web architectures (SPAs, SSR, etc.).
• Bot Evasion Engineering: Research and implement strategies to bypass anti-scraping technologies, including proxy rotation, browser fingerprinting, and CAPTCHA solving.
• Database Management: Create and optimize SQL schemas for large-scale data storage and perform complex data transformations and validation.
• System Maintenance: Proactively monitor the health of extraction agents and refactor code quickly in response to target website updates or layout changes.
• Performance Optimization: Utilize asynchronous Python programming to maximize the throughput and efficiency of data collection pipelines.

Requirements

• Advanced Python: Mastery of Python 3.x with deep experience in extraction frameworks (Scrapy, Playwright, Selenium, or Puppeteer).
• Technical Resilience: Proven ability to bypass high-level bot detection (e.g., Cloudflare, Akamai, or PerimeterX).
• Database Mastery: Expert-level SQL skills and experience managing relational databases like PostgreSQL or MySQL.
• Network Proficiency: Expert understanding of HTTP/S, TCP/IP, TLS fingerprinting, and browser-header manipulation.
• Problem Solving: A specialized ability to reverse-engineer JavaScript-heavy websites and hidden API endpoints.
• Able to write, debug, and deploy complex Python code in a distributed environment.
• Must be able to analyze and interpret complex web structures and network traffic using browser developer tools.
• Ability to design and maintain relational database tables containing millions of rows.
• Able to pivot and respond quickly to technical "break-fixes" to ensure data continuity for the business.
• Collaboration with data analysts to define and validate data requirements and output formats.

Education & Experience

Benefits at Babel Street (just to name a few...)

• Health Benefits: Babel Street covers 85-100% monthly premium costs for Medical, Dental, Vision, Life & Disability insurances – for you and your family!
• Retirement Plans: Babel Street offers both a Traditional and Roth 401(K) with a very competitive match.
• Unlimited Flexible Leave: We trust our employees to manage their own time and balance their personal and work lives.
• Holidays: Babel Street provides employees with 12 paid Federal Holidays
• Tuition Reimbursement: We are committed to investing in our employees. One way we do that is with our Tuition Reimbursement Program for continuing education.
• Babel Street is an equal opportunity/affirmative action employer. All qualified applicants will receive consideration for employment without regard to sex, gender identity, sexual orientation, race, color, religion, national origin, disability, protected Veteran status, age, or any other characteristic protected by law. Further, Babel Street will not discriminate against applicants for inquiring about, discussing or disclosing their pay or, in certain circumstances, the pay of their co‐worker, Pay Transparency Nondiscrimination . In addition, Babel Street's policy is to provide reasonable accommodation to qualified employees who have protected disabilities to the extent required by applicable laws, regulations and ordinances where a particular employee works. Upon request, we will provide you with more information about such accommodations.