greenhouse

Senior Data Platform Engineer

LawZero

Montréal, CAN

Full-time

Senior · 5+ years experience

Onsite

Discovered 3 days ago

RayApache SparkApache ArrowParquetKubernetesSlurm

Apply

Free

Job Fit Check

Base Career helps you apply smarter for this job.

Ready to Scan

Key skills for this role

RayApache SparkApache Arrow

Smart Apply

Full Job Posting

Key Responsibilities

Scale and automate the data processing stack to handle petabytes of data and ensure its smooth operation.
Design the execution layer for pipeline stages of differing computational shape, matching each stage to an appropriate engine and keeping the pipeline saturated end to end.
Ensure efficient use of compute resources, including GPU access for compute-intensive data processing tasks.
Make pipelines reliable at scale, with graceful failure recovery, restartability, and observability that attributes bottlenecks and cost to the responsible stage.
Ensure all datasets, including the intermediate outputs of each transformation stage, are versioned, reproducible, and fully traceable to meet specific and dynamic experiment needs, and are accompanied by datasheets, in accordance with internal Data Governance policies.
Partner with the Research team to ensure the datasets you produce integrate seamlessly with the training pipelines.

Skills and Qualifications

A bachelor's degree in a relevant field (e.g., computer science, computer engineering, software engineering) is required.
5+ years of experience designing, implementing, and managing large-scale distributed data processing systems, or working within large-scale distributed ML data frameworks, with recent experience using e.g. Ray, Apache Spark, workflow orchestrators, Apache Arrow, and/or Parquet.
Demonstrated ownership of a data processing system under real throughput, reliability, and cost pressure. The specific frameworks matter less to us than evidence that you have had to reason about where a large pipeline breaks and why.
Experience profiling and optimizing throughput and cost across heterogeneous workloads, including GPU-accelerated stages.
Experience with dataset versioning, lineage, and reproducibility tooling.
Ability to collaborate effectively with cross-functional teams, document best practices, and stay updated with the latest advancements in large-scale data processing and software development.

Apply for this job in 1 click

Skip the repetitive application forms

Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.

Trusted by over 500,000 job seekers on Base Career

Start Free Today

Senior Data Platform Engineer

Job Fit Check

Key Skills for This Role

Full Job Posting

Key Responsibilities

Skills and Qualifications

Apply for this job in 1 click

More jobs at LawZero

Senior ML Research Developer

Mathematical Scientist for AI Safety Research

Senior ML Research Scientist

Senior ML Data Processing Developer

Responsible AI & Data Governance Lead

Director, Evaluations

Senior Manager, Talent Acquisition

Operations Admin Lead – Executive Office