ashby
AI Evaluations Engineer
Ellipsis Health
Remote, USA
Full-time
Mid · 5+ years experience
Remote
USD 160000-210000 yearly / year
Discovered 1 weeks ago
PythonTypeScriptJavaCI/CDAPIsData Pipelines
Free
Job Fit Check
Base Career helps you apply smarter for this job.
?%
Ready to ScanKey skills for this role
PythonTypeScriptJava
Role Overview
We’re seeking a strong Software Engineer to join our AI Evaluation team, focused on building the infrastructure and internal tooling that enable reliable, repeatable evaluation of AI systems in production. In this role, you’ll develop the scaffolding around our evaluations platform—API/data integrations, test configuration management, and reporting tools that make evaluations easy to run, extend, and operate at scale. You’ll also build and maintain evaluation frameworks for individual system components such as ASR, LLMs, TTS, knowledge bases, and guardrails, productionalizing these frameworks to support regular analysis, regression detection, and continuous monitoring.
In addition, you’ll create debugging tools that make it easy to inspect end-to-end calls, trace failures, and surface all relevant signals in one place—empowering internal teams to diagnose issues quickly and confidently. The ideal candidate is a pragmatic, infrastructure-minded engineer who enjoys turning ad hoc analysis into durable systems, cares deeply about developer experience, holds a high bar for software-quality and takes pride in building tooling that makes complex AI systems observable, testable, and easier to operate.
Key Skills for This Role
PythonTypeScriptJavaCI/CDAPIsData Pipelines
Full Job Posting
About the Role
We’re seeking a strong Software Engineer to join our AI Evaluation team, focused on building the infrastructure and internal tooling that enable reliable, repeatable evaluation of AI systems in production. In this role, you’ll develop the scaffolding around our evaluations platform—API/data integrations, test configuration management, and reporting tools that make evaluations easy to run, extend, and operate at scale. You’ll also build and maintain evaluation frameworks for individual system components such as ASR, LLMs, TTS, knowledge bases, and guardrails, productionalizing these frameworks to support regular analysis, regression detection, and continuous monitoring.
In addition, you’ll create debugging tools that make it easy to inspect end-to-end calls, trace failures, and surface all relevant signals in one place—empowering internal teams to diagnose issues quickly and confidently. The ideal candidate is a pragmatic, infrastructure-minded engineer who enjoys turning ad hoc analysis into durable systems, cares deeply about developer experience, holds a high bar for software-quality and takes pride in building tooling that makes complex AI systems observable, testable, and easier to operate.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Ellipsis Health
Senior Software Development Engineer in Test (SDET), Salesforce
San Francisco, USA
SeniorFull-time
Discovered 1 weeks agoView →
Senior Backend Engineer, Core Services
San Francisco, USA
Discovered YesterdayFull-time
Senior Full Stack Engineer
, USA
Discovered 1 weeks agoFull-time
Forward Deployed Product Manager, AI Assistant
, USA
Discovered 1 weeks agoFull-time
Senior Backend Developer
, USA
Discovered 1 weeks agoFull-time
Senior Backend Engineer (Python)
, USA
Discovered 1 weeks agoFull-time
Senior Software Development Engineer in Test (SDET), Salesforce
San Francisco, USA
Discovered 1 weeks agoFull-time
Senior QA Engineer, Forward Deployed
San Francisco, USA
Discovered 1 weeks agoFull-time
Senior Data Platform Engineer
San Francisco, USA
Discovered 1 weeks agoFull-time