smartrecruiters
Staff Software Engineer, Agent Eval Platform
ServiceNow
Santa Clara, USA
Full-time
Onsite
Discovered 3 weeks ago
Distributed systemsWorkflow orchestrationOpenTelemetryPythonGoAsync programming
Free
Job Fit Check
Base Career helps you apply smarter for this job.
?%
Ready to ScanKey skills for this role
Distributed systemsWorkflow orchestrationOpenTelemetry
Role Overview
Build the judgement layer of an agent evaluation platform for AI agents that plan, call tools, and change state in enterprise systems.
Develop calibrated evaluation signals that explain agent failures and can ultimately be used as reward signals for agent optimization.
This is applied ML and systems work focused on LLM evaluation rather than pretraining or conventional testing.
Key Skills for This Role
Distributed systemsWorkflow orchestrationOpenTelemetryPythonGoAsync programming
Full Job Posting
The Role
Build the judgement layer of an agent evaluation platform for AI agents that plan, call tools, and change state in enterprise systems.
Develop calibrated evaluation signals that explain agent failures and can ultimately be used as reward signals for agent optimization.
This is applied ML and systems work focused on LLM evaluation rather than pretraining or conventional testing.
Eval Orchestration at Scale
- Build the runtime that executes multi-turn agent scenarios, including environment setup, user simulation, execution, collection, scoring, and teardown.
- Implement scheduling, retries, high-concurrency execution, run isolation, versioned specifications, datasets, reports, and run comparison.
- Consolidate one-off workflows into a single orchestration service with reliability targets and self-service evaluation execution.
Agent Observability and Tracing
- Lead the transition to OpenTelemetry-native observability for the agent platform.
- Define queryable span data for prompts, tool calls, plan updates, outcomes, and long-lived asynchronous sessions.
- Preserve prompts and completions, prevent evaluation traffic contamination, and support fault attribution and cross-run diffing.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at ServiceNow
Customer Success Executive, Strategic Accounts
Mumbai, IND
SeniorFull-time
We’re looking for someone who can walk into a boardroom with a CIO, CFO, or CHRO and be treated as a peer — not a vendor. This person owns post-sales outcomes across the full customer lifecycle — implementation, adoption
Discovered TodayView →
Senior Staff Software Engineer - Agentic AI
Santa Clara, USA
Discovered TodayFull-time
Customer Success Executive, Strategic Accounts
Mumbai, IND
Discovered TodayFull-time
Software Development Engineer - III
Hyderabad, IND
Discovered TodayContract
Staff Applications Dev Engineer
Hyderabad, IND
Discovered TodayFull-time
Senior Staff Software Engineer, Moveworks
Bengaluru, IND
Discovered TodayFull-time
Senior Advisory AI Foundry Architect
Sydney, AUS
Discovered TodayFull-time
Senior Advisory AI Foundry Architect
Melbourne, AUS
Discovered TodayFull-time
Customer Success Executive, Strategic Accounts
Mumbai, IND
Discovered TodayFull-time