LLM Reliability & Evaluation Engineer

ABOUT XENONSTACK

THE OPPORTUNITY

KEY RESPONSIBILITIES

• Evaluation Frameworks Design and implement LLM evaluation pipelines covering accuracy, robustness, safety, and bias. Develop automated systems for benchmarking models on enterprise-relevant tasks.
• Design and implement LLM evaluation pipelines covering accuracy, robustness, safety, and bias.
• Develop automated systems for benchmarking models on enterprise-relevant tasks.
• Reliability Engineering Conduct stress tests, adversarial testing, and edge-case evaluations . Build tools to measure latency, consistency, and error recovery in multi-turn interactions.
• Conduct stress tests, adversarial testing, and edge-case evaluations .
• Build tools to measure latency, consistency, and error recovery in multi-turn interactions.
• Metrics & Monitoring Define KPIs such as factual accuracy, hallucination rate, toxicity, and compliance alignment . Establish real-time monitoring for drift, anomalies, and performance regressions .
• Define KPIs such as factual accuracy, hallucination rate, toxicity, and compliance alignment .
• Establish real-time monitoring for drift, anomalies, and performance regressions .
• Collaboration & Alignment Partner with ML engineers, product managers, and domain experts to align evaluation with business objectives. Work with Responsible AI teams to implement ethical, explainable, and compliant evaluation practices .
• Partner with ML engineers, product managers, and domain experts to align evaluation with business objectives.
• Work with Responsible AI teams to implement ethical, explainable, and compliant evaluation practices .
• Continuous Improvement Feed insights from evaluation into fine-tuning, RLHF/RLAIF pipelines, and model selection . Maintain a central repository of test cases, benchmarks, and evaluation results .
• Feed insights from evaluation into fine-tuning, RLHF/RLAIF pipelines, and model selection .
• Maintain a central repository of test cases, benchmarks, and evaluation results .
• Research & Innovation Stay current with state-of-the-art LLM evaluation techniques , from academic benchmarks to applied enterprise metrics. Explore automated evaluation using agentic test harnesses and synthetic data generation .
• Stay current with state-of-the-art LLM evaluation techniques , from academic benchmarks to applied enterprise metrics.
• Explore automated evaluation using agentic test harnesses and synthetic data generation .

Must-Have

• 3–6 years in AI/ML, NLP, or applied model evaluation .
• Strong understanding of LLM architectures, prompt engineering, and failure modes .
• Hands-on with evaluation frameworks (Eval harnesses, Ragas, OpenAI Evals, DeepEval).
• Proficiency in Python and libraries like LangChain, LangGraph, LlamaIndex, Hugging Face .
• Experience with vector databases, RAG pipelines, and knowledge graph integration .
• Familiarity with bias/fairness testing and Responsible AI frameworks .

Good-to-Have

WHY SHOULD YOU JOIN US?

XENONSTACK CULTURE – JOIN US & MAKE AN IMPACT!

Our Cultural Values

Our Product Philosophy