Parallel Software Engineer

Parallel Software Engineer Mountain View, CA / Hyderabad, IN / Remote About DataPelago:

DataPelago is at the forefront of revolutionizing data processing for traditional analytics and cutting-edge GenAI preprocessing.

We are building an innovative data processing engine that is transforming how Apache Spark, Apache Flink, Ray and others operate on diverse, large-scale data.

Our team of engineers drive and adopt advances in hardware-accelerated computing, parallel processing of large-scale data, query optimization, distributed systems, compilers, machine learning, and cloud-native computing.

We are looking for specialists to join our engineering team and shape the future of accelerated data processing.

The Opportunity: As a Parallel Software Engineer, you will be a key individual contributor in developing advanced parallel software that unlocks the full potential of diverse hardware accelerators including GPUs and SIMD-capable CPUs.

You will enhance functional breadth, performance, scale, and reliability of data processing operators that are integral to our data processing engine.

This is a unique opportunity to make a significant impact on a category-defining product and work with a talented team of engineers.

• Architect: Influence the architecture of how our data processing engine harnesses the par- allelism in GPUs and SIMD-capable CPUs efficiently in processing diverse large-scale data.
• Design: Lead design of functional and performance enhancements to operators and func- tions that are accelerated by our engine.
• Core Development: Individually design, implement, test, optimize, and maintain parallel implementations of operators and functions on diverse acceleration hardware.
• Innovation and Differentiation: Analyze advances in accelerated computing hardware, pro- gramming models, and related tools and ensure our engine extends technology and product

leadership.

• Collaboration: Partner effectively with the execution engine engineering team in integrating parallel software components with the overall engine.
• Continuous Improvement: Foster best practices in design and code reviews, testing, CI/CD, and issue resolution to maintain highest product quality, security, efficiency, & productivity. What You'll Bring: