Base Career helps you apply smarter for this job.
Key skills for this role
We are looking for a Staff Software Engineer to lead the design and development of our Agentic Harness and agent evaluation platform. The Agentic Harness provides the runtime, tools, context, state, policies, and observability required to build and operate production AI agents. The evaluation platform measures how well those agents complete real customer tasks and detects regressions before they reach production.
This is a hands-on technical leadership role. You will build production systems, establish architecture across team boundaries, and define the metrics and engineering practices used to improve agent quality. You will work with product, infrastructure, applied AI, security, and modeling teams to take new AI capabilities from prototype to dependable customer value.
We are looking for a Staff Software Engineer to lead the design and development of our Agentic Harness and agent evaluation platform. The Agentic Harness provides the runtime, tools, context, state, policies, and observability required to build and operate production AI agents. The evaluation platform measures how well those agents complete real customer tasks and detects regressions before they reach production.
This is a hands-on technical leadership role. You will build production systems, establish architecture across team boundaries, and define the metrics and engineering practices used to improve agent quality. You will work with product, infrastructure, applied AI, security, and modeling teams to take new AI capabilities from prototype to dependable customer value.
• Architect and build the Agentic Harness that executes complex, multi-step AI workflows across models, tools, data, and services.
• Design stable interfaces for tool execution, context construction, state management, memory, permissions, retries, fallbacks, and human review.
• Own agent quality end to end by building evaluation harnesses, representative datasets, automated graders, experiment pipelines, and release gates.
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
Dublin, USA
McLean, USA
, USA
New York City, USA
New York City, USA
London, GBR
London, GBR
London, GBR
• Convert ambiguous reports such as “the agent feels worse” into measurable failure modes, reproducible tests, and durable fixes.
• Analyze production agent trajectories to identify failures in reasoning, retrieval, tool use, context, orchestration, and application code.
• Close the loop between production incidents, root-cause analysis, evaluation coverage, and regression prevention.
• Develop offline and online measurements for task completion, correctness, groundedness, safety, latency, reliability, and cost.
• Build simulation and replay infrastructure for golden-set tests, adversarial scenarios, model comparisons, and large-scale experiments.
• Improve agent efficiency through model routing, prompt and semantic caching, context compaction, tool-result management, and token optimization.
• Productionize new model capabilities as secure, observable, multi-tenant services with clear operational controls.
• Establish standards for evaluation design, including sampling, ground-truth quality, grader calibration, leakage prevention, and statistical significance.
• Define technical direction across multiple teams and lead projects whose scope extends beyond a single service.
• Mentor engineers, raise the quality of architecture reviews, and remain directly involved in implementation and debugging.
• Building evaluation or observability infrastructure for agentic coding, data engineering, or analytics systems.
• Designing human-evaluation programs, scoring rubrics, annotation workflows, or grader-calibration methods.
• Working with multi-agent orchestration, long-running agents, asynchronous workflows, or durable execution.
• Developing synthetic tasks, simulations, adversarial tests, red-team exercises, or safety guardrails.
• Building retrieval systems that use vector search, hybrid search, semantic indexing, ranking, or caching.
• Operating multi-tenant systems that process sensitive enterprise data.
• Working with model training, fine-tuning, reinforcement learning, or feedback-driven optimization.
• Evaluating and onboarding frontier models based on measured product outcomes.
• Experience with databases, SQL engines, data platforms, Kubernetes, or cloud-native infrastructure.
• Treat evaluation as part of product engineering rather than a final validation step.
• Can move between agent behavior, distributed infrastructure, data analysis, and production debugging.
• Question metrics that do not reconcile and design tests that can expose misleading results.
• Take ownership from early architecture through deployment, operations, and measurable customer outcomes.
• Prefer evidence from representative tasks and production behavior over isolated benchmark results.
• Work effectively in fast-moving environments where requirements develop through experimentation.
Every Snowflake employee is expected to follow the company’s confidentiality and security standards when handling sensitive data. Protecting customer information is an essential part of every employee’s duties.
Snowflake is growing fast, and we are scaling our team to support that growth. We are looking for people who share our values, challenge ordinary thinking, and push the pace of innovation while building a future for themselves and Snowflake.
How do you want to make your impact?
Snowflake is growing fast, and we’re scaling our team to help enable and accelerate our growth. We are looking for people who share our values, challenge ordinary thinking, and push the pace of innovation while building a future for themselves and Snowflake.
For jobs located in the United States, please visit the job posting on the Snowflake Careers Site for salary and benefits information: careers.snowflake.com
Snowflake is a cloud-based data platform that enables organizations to store, manage, and analyze data at scale. The company provides solutions for data warehousing, data lakes, data engineering, and AI/ML workloads across major cloud providers.
Visit company websiteJobs and hiring trendsFull-time
Senior · 9+ years experience
Apply faster on company sites with our extension.