linkedin
Senior Quality Assurance Engineer
Lattice AI
Dubai, UAE
Contract
Senior
3 weeks ago
PythonPytestPlaywrightSeleniumLLM EvaluationDeepEval
Free
Job Fit Check
Base Career helps you apply smarter for this job.
?%
Ready to ScanKey skills for this role
PythonPytestPlaywright
About the Role
Lattice AI is hiring a Senior QA Engineer to build automated testing frameworks for LLM outputs, including hallucination detection, consistency checks, and RAG pipeline validation.
Key Skills for This Role
PythonPytestPlaywrightSeleniumLLM EvaluationDeepEval
Responsibilities
- Build automated frameworks that test non deterministic LLM outputs for hallucination, consistency, and factual accuracy against gold standard datasets
- Develop evaluation metrics (RAGAS, faithfulness, answer relevance) to validate RAG pipelines and citations
- Create prompt regression suites to catch drift when underlying models or system instructions change
- Implement integration tests that keep AI agents honest against SaaS enterprise systems
- Run performance benchmarks (Locust, JMeter, K6) and enforce CI/CD quality gates in GitLab
Requirements
- 5+ years in QA automation
- At least 2 years on ML models, data heavy applications, or AI agents
- Strong Python (Pytest, Playwright/Selenium, Requests)
- Hands on with LLM evaluation frameworks (DeepEval, TruLens, or custom evaluators)
- SQL and data validation (Great Expectations)
- Experience with vector databases and modern QE practice
- Comfort defining pass/fail criteria for probabilistic systems
Full Job Posting
Company Description
- Lattice AI is an independent evaluation, red teaming, and assurance firm for AI systems, headquartered in the UAE and operating globally.
- The company provides evaluation, red teaming, and governance services mapped to standards such as SOC 2, ISO 42001, EU AI Act, and UAE AI Charter.
Role Overview
- We are hiring a QA Engineer (LLM Evals + Quality Engineering) for a client location in Abu Dhabi.
- This role sits at the intersection of large language models and rigorous quality engineering.
What you will own
- Automated frameworks that test non deterministic LLM outputs for hallucination, consistency, and factual accuracy against gold standard datasets.
- Evaluation metrics (RAGAS, faithfulness, answer relevance) that validate RAG pipelines and the citations behind them.
- Prompt regression suites that catch drift when underlying models or system instructions change.
- Integration tests that keep AI agents honest against SaaS enterprise systems.
- Performance benchmarks (Locust, JMeter, K6) and CI/CD quality gates in GitLab.
What we are looking for
- 5+ years in QA automation, with at least 2 years on ML models, data heavy applications, or AI agents.
- Strong Python (Pytest, Playwright/Selenium, Requests).
- Hands on with LLM evaluation frameworks (DeepEval, TruLens, or custom evaluators) and ground truth dataset creation.
- SQL and data validation (Great Expectations), vector databases, and modern QE practice (shift left, test pyramid, mono repo).
- Comfort defining pass/fail criteria for probabilistic systems and communicating confidence levels to engineering leadership.
Additional Notes
- If you can tell the difference between a model that looks right and one that is right, we want to talk.
- Send us a message if your profile matches the above expectation; only traditional quality engineering roles should not apply.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career