Senior Quality Assurance Automation Engineer
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
Role Overview
Senior QA Automation Engineer role for a UAE-based Sovereign AI company developing agentic AI products for construction, government, healthcare, and defense.
The role owns quality for AI systems that reason, call tools, and produce non-deterministic outputs.
This is a fully remote, contract-based role with onboarding as soon as possible.
Key Skills for This Role
Full Job Posting
About the Role
Senior QA Automation Engineer role for a UAE-based Sovereign AI company developing agentic AI products for construction, government, healthcare, and defense.
The role owns quality for AI systems that reason, call tools, and produce non-deterministic outputs.
This is a fully remote, contract-based role with onboarding as soon as possible.
What You Will Do
- Test agentic systems end to end, including tool use, trajectory evaluation, grounding, citation verification, and refusal calibration.
- Perform guardrail and adversarial testing for prompt injection and jailbreak vulnerabilities.
- Evaluate non-deterministic AI using hallucination, consistency, and accuracy scores against gold datasets.
- Use LLM-as-judge methods calibrated against human judgment.
- Own prompt and model regression suites that detect quality drift before release.
- Run continuous evaluation on production traffic with monitoring, alerting, and incident response.
- Perform integration, API, secure gateway, performance, and load testing.
- Validate RBAC and PII controls in secure gateways.
What We Are Looking For
- At least 5 years of QA or quality engineering experience, including at least 2 years testing ML, LLM, or AI agent systems.
- Hands-on experience with DeepEval, TruLens, RAGAS, custom Python evaluators, or comparable evaluation frameworks.
- Experience with LLM-as-judge evaluation and agent observability or tracing.
- Expert Python skills with Pytest and custom harnesses.
- Experience with Selenium or Playwright and API testing.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at GCS
Staff Machine Learning Engineer
Abu Dhabi, UAE
The employer is seeking a Staff Machine Learning Engineer to build, deploy, and optimize production ML and LLM systems for live AI products. The role requires 5+ years of production ML or AI engineering, strong Python, L
Senior DevOps Engineer
, UAE
GCS is hiring a Senior DevOps Engineer to own infrastructure and deployment pipelines for production AI systems across cloud and on-premise environments. The role requires five or more years of DevOps or infrastructure e
Senior MLOps Engineer
, UAE
The employer is seeking a Senior MLOps Engineer to own infrastructure for training, serving, and scaling large language models in production. The role requires 5+ years of MLOps or ML infrastructure experience, large-mod