Sr. Staff Machine Learning Systems Engineer

Set the technical direction for our evaluation systems; metric, judge and scorer design, the statistical methodology behind regression decisions, and the infrastructure that tracks and categorizes failures over time.

Design and scale the data pipelines — ingestion, transformation, dataset versioning, labeling and calibration workflows — that both evaluation and downstream data science work depend on.

Proactively address challenges in scaling and complexity AI evaluation.

Define how we evaluate any new AI service from scratch. Drive multi-team initiatives like replacing manual, inconsistent review processes with statistically sound, automated gates.

Own our approach to adversarial and red-team evaluation as a risk-reduction program, designing the test suites and failure taxonomies that catch safety and edge-case issues before they reach patients.

Work through complex, cross-team technical disagreements and drive alignment across engineering, product and AI leaders.

Originate new approaches and methodology that becomes a reusable standard rather than a one-off fix.

Take vague, cross-team pain points ("we do this manually and it's inconsistent") all the way from a rough idea to a fully-specified, shipped system, without needing to hand off any part of the journey.

Lead major platform improvements; re-architecting core systems, removing brittle logic, modernizing how things run with impact that's felt org-wide, not just on your own team.

Build real working relationships across ML engineering, data science, platform engineering, clinical, legal, and product, the kind of trust that gets you looped in early, before decisions are locked in.

Become a go-to voice on evaluation methodology and data pipeline design: share what you've learned in internal talks, write things up so other teams can use them, and expect your ideas to shape how others approach similar problems.

Mentor other engineers, including experienced ones, and help raise the technical and statistical bar of the teams you work with.

10+ years of experience in ML infrastructure, data engineering, or evaluation/testing systems, with a track record of impact that reaches beyond a single team or project.

Hands-on depth in evaluation systems: designing and calibrating LLM judges/scorers, building statistically sound regression-testing methodology (e.g., paired significance testing with proper correction for multiple comparisons), measuring agreement against human labels, and designing adversarial/red-team evaluation approaches.

Hands-on depth in data pipeline engineering: dataset versioning, feature and benchmark pipelines, labeling and calibration workflows, and high-throughput ingestion and transformation systems.

A history of building things that became the standard approach for others — not just solving your own problem, but changing how a broader group of people tackle a category of problem.

Experience leading multi-team projects to completion, including navigating and resolving genuine technical disagreement along the way.

A track record of mentoring other engineers, including senior ones, and visibly raising the bar for the teams around you.

Excellent communication — comfortable adapting the same idea for different audiences, and confident building support for it well before launch.

Strong Python, and enough statistical fluency to design and defend a testing framework that real production decisions ride on.

Experience with Databricks, MLflow, Unity Catalog, or similar data/eval platforms.

Experience building reporting tools for people without direct engineering access (e.g., automated Slack digests, spreadsheet reports for non-technical teams).

Prior experience in a regulated industry (healthcare, fintech, life sciences).

A track record of company-wide talks or write-ups that changed how other teams approached a problem.