AI Quality and Evaluation Lead

Production readiness and risk

• Establish quality and evaluation gates covering UAT, parallel-run evidence, performance thresholds, guardrails, privacy, access controls, rollback criteria, kill switches, runbooks and approvals.
• Conduct failure-mode testing for prompt injection, data leakage, unsupported claims, policy deviations, tool misuse, unsafe write-back, inappropriate automation and model degradation.
• Monitor production quality using model and prompt versions, drift, latency, user feedback, exception rates, overrides, incidents and platform costs.
• Create defect taxonomies that distinguish model, retrieval, prompt, data, rules, integration, user experience and process issues.

Scaling the quality capability

• Develop reusable scorecards, automated continuous evaluation and squad-level quality practices as the AI Factory grows.
• Oversee AI Quality Analysts as volume grows and conduct independent model and vendor benchmarking.
• Maintain audit-ready quality evidence and work with product and engineering teams to prioritize durable fixes.

Experience required

• A bachelor’s degree in computer science, engineering, data science, statistics, mathematics, information systems or a related discipline is required.
• More than 7 years of progressive experience in quality engineering, software testing, model validation, machine learning, data science, risk analytics, AI governance or related technology disciplines is required.
• Experience designing evaluation frameworks and release criteria for production AI or generative AI systems is required.
• Experience with RAG, document intelligence, agents, model and prompt lifecycle management, observability and human-in-the-loop controls is required.
• Strong technical and analytical skills with Python, SQL, test automation, data analysis, statistical methods and evaluation tooling are required.
• Experience building or scaling a quality capability through reusable standards, automated assets, monitoring, governance evidence and coaching is required.
• Experience in financial services, mortgage lending, lending operations or regulated technology environments is preferred.

Working environment

• The work is performed in an office environment with periods of high volume and tight timelines.
• The role involves prolonged sitting, repetitive keyboard and mouse use, and extended computer-screen viewing.
• Multitasking may include speaking with customers by telephone while using a computer program.

The team