ashby
AI Engineer, Model Quality and Performance
Cerebras Systems
Sunnyvale, USA
Full-time
Mid
Onsite
Discovered 1 weeks ago
ClaudeDockerGitCIEvalScopelm-eval-harness
Free
Job Fit Check
Base Career helps you apply smarter for this job.
?%
Ready to ScanKey skills for this role
ClaudeDockerGit
Role Overview
You'll own model quality and performance for Cerebras' inference offerings. You will define what "good" looks like across the models we serve, building AI-driven systems to measure it at scale, and translating those signals into artifacts our customers and product team actually use.
You'll use AI agents to spin up custom eval suites per customer use case, mine trajectories for representative test data, automate the repetitive parts of release qual, and help build performance datasets and benchmarking workflows for customer use cases. We want someone whose first instinct is "how do I get an AI agent to do this on a loop."
You'll sit between engineering, product, and customer-facing teams.
Key Skills for This Role
ClaudeDockerGitCIEvalScopelm-eval-harness
Full Job Posting
About The Role
You'll own model quality and performance for Cerebras' inference offerings. You will define what "good" looks like across the models we serve, building AI-driven systems to measure it at scale, and translating those signals into artifacts our customers and product team actually use.
You'll use AI agents to spin up custom eval suites per customer use case, mine trajectories for representative test data, automate the repetitive parts of release qual, and help build performance datasets and benchmarking workflows for customer use cases. We want someone whose first instinct is "how do I get an AI agent to do this on a loop."
You'll sit between engineering, product, and customer-facing teams.
What You'll Do
Design eval suites with AI agents in the loop. For every model release, curate a thoughtful mix of advanced, basic, long-context, and customer-use-case-specific evals. Use Claude to generate, validate, and prune candidate test cases at speed.
Build custom evals for target customers by orchestrating AI agents to mine trajectories from their workloads and synthesize representative eval sets.
Automate eval execution end-to-end with AI-driven pipelines on top of standard tooling (Docker, Git, CI). The goal is a system that runs itself between releases, not a script you re-run by hand.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Cerebras Systems
Senior FE Engineer - Inference Console
Sunnyvale, USA
Discovered TodayFull-time
Software Engineer, Kernel Reliability
, USA
Discovered TodayFull-time
Senior Datacenter Construction Manager
, USA
Discovered TodayFull-time
DevOps Engineer Intern - PEY
Toronto, CAN
Discovered YesterdayInternship
Software Engineer - Host and Network IO
Sunnyvale, USA
Discovered 3 days agoFull-time
Staff GPU Inference SDET
Sunnyvale, USA
Discovered 3 days agoFull-time
Senior Operations Accountant
Sunnyvale, USA
Discovered 3 days agoFull-time
Staff Software Engineer, Inference Cloud
Sunnyvale, USA
Discovered 1 weeks agoFull-time