Principal Evals Engineer - AI & Agentic Systems

Organisation Context

Role Overview

Evaluation Strategy and Platform

• Define evaluation strategy, architecture, methodologies, and release decision criteria.
• Build evaluation harnesses, golden-set management, dataset versioning, automated grading, regression detection, and reporting infrastructure.
• Own judge-model selection, rubric design, and calibration against human labels.

AI and Multilingual Evaluation

• Measure grounding, citation correctness, tool use, multi-step reasoning, and failure recovery.
• Build Arabic evaluation datasets with dialect coverage, right-to-left validation, and calibrated judges.
• Evaluate non-deterministic systems where hallucination, poor grounding, behavioural drift, and prompt injection require more than conventional assertions.

Production and Release Quality

• Build online evaluation, sampling, human review, drift detection, alerting, and adversarial testing.
• Integrate AI evaluation with test automation and CI/CD quality gates.
• Use defensible evidence to support model swaps, prompt changes, framework migrations, and infrastructure decisions.

Evaluation Culture

• Enable engineering teams to run rigorous evaluations independently and design evaluation into systems from the beginning.
• Turn production quality escapes into evaluations that can detect the same failures before release.

Core Qualifications

• Staff- or Principal-level track record building evaluation, testing, or quality infrastructure at meaningful scale.
• Deep AI evaluation experience across output quality, retrieval, grounding, and regression detection.
• Experience with LLM-as-judge methods, rubric design, judge selection, human-label calibration, and automated-grading failure modes.
• Strong Python engineering skills for production evaluation harnesses, frameworks, and shared libraries.
• Statistical literacy covering sampling, confidence intervals, inter-rater agreement, and significance.
• Strong CI/CD engineering knowledge across API, web, and data surfaces.

Preferred Qualifications

• Experience with Ragas, DeepEval, Promptfoo, Braintrust, LangSmith, or Langfuse is strongly preferred.
• Arabic AI evaluation, adversarial AI security testing, voice evaluation, human evaluation programs, or regulated-environment delivery is preferred.
• Working knowledge of TypeScript or Java is valuable.

Technology Environment

Role Boundaries