LLM Ops Lead

Own AI quality standards, evaluation frameworks, and release gates across Ask Sovi products

Lead prompt and model versioning, regression detection, and AI governance initiatives

Coordinate Safety, Guardrails, and Red Team testing programs

Build observability practices across latency, cost, quality, and safety metrics using tools such as Langfuse

Manage Jira epics, ticket triage, and cross-functional coordination across Product and Engineering teams

Develop LLMOps playbooks, best practices, and operational standards

Lead AI incident response, troubleshooting, and root-cause analysis

2-3 years of experience in Applied AI, LLMOps, AI Quality Engineering, AI Operations, Test Automation, or related fields

Hands-on experience with Langfuse, LangSmith, or similar AI observability and evaluation tools

Strong understanding of AI quality, evaluation pipelines, prompt versioning, and AI governance

Experience with AI safety, guardrails, content moderation, audit logging, and release management

Strong stakeholder management and communication skills

Ability to support teams operating across UK business hours

Applied AI Engineers from AI-native or AI-first organizations with hands-on Langfuse/LangSmith experience

Product-minded Engineers who combine strong technical skills with customer and business impact awareness

QA/Test Automation Engineers who have transitioned into AI evaluation, quality, or LLM testing

AI Quality Analysts with expertise in hallucination detection, model drift monitoring, LLM-as-a-Judge methodologies, and AI quality governance

Work on cutting-edge Generative AI products and platforms

Collaborate with global AI, Product, and Engineering leaders

Bi-Weekly Meeting Free Days

Mentoring and career development programs

Globally recognized training opportunities

Comprehensive health and wellness benefits

Flexible and collaborative work environment

CCAT & EPP Assessment → Automated Video Interview → Hiring Manager Interview → Final Interview