Senior QA Automation Engineer
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
We are seeking a Senior QA Automation Engineer to lead validation and verification strategies for AI transformation initiatives. You will define rigorous standards for non-deterministic AI systems, creating automated testing frameworks for LLMs and predictive engines.
Key Skills for This Role
Responsibilities
- Lead validation and verification of AI systems by architecting automated testing frameworks for non deterministic outputs
- Design and implement automated evaluation metrics (RAGAS, faithfulness, answer relevance) for RAG pipelines
- Develop prompt regression suites to monitor prompt drift and maintain quality of AI generated documents
- Build integration tests to validate data consistency between AI agents and enterprise systems (SAP S/4HANA, Ariba)
- Create performance benchmarking tests for latency and throughput of forecasting models and risk scoring engines
- Automate API validation for secure gateways, verifying RBAC and PII redaction logic
- Ensure alignment of automated test cases with System Requirements and User Needs for V&V reports
- Prepare Test Readiness packages for Stage Gate reviews
- Manage defect lifecycle between Requirements Quality Assistants and development teams
- Bridge Agile development and formal Systems Engineering processes for defense sector deployment
Requirements
- 5+ years of experience in QA Automation
- At least 2 years focused on testing complex data driven applications, machine learning models, or AI agents
- Expert proficiency in Python for building custom test harnesses (Pytest) and automation libraries (Selenium/Playwright, Requests)
- Expert proficiency in performance testing tools (Locust, JMeter, K6)
- Experience with LLM evaluation frameworks (DeepEval, TruLens, or custom Python evaluators)
- Proficiency with SQL and data validation tools (e.g., Great Expectations)
- Strong experience integrating automated tests into GitLab CI/CD pipelines
- Familiarity with requirements management tools (Jira, Linear, Jama, Polarion)
- Hands on experience managing test reports and artifacts (TestRail, Allure)
- Knowledge of modern Quality Engineering best practices (Shift Left, Test Pyramid, Mono repo)
- Experience in Defense, Aerospace, or highly regulated industries is a strong plus
Full Job Posting
Company Overview
- Headquartered in Abu Dhabi, specializes in developing defense grade artificial intelligence solutions.
- Focuses on industries requiring mission critical AI, integrating automated validation frameworks.
Job Summary
- Seeking a Senior QA Automation Engineer to lead validation and verification strategies for AI transformation initiative.
- Define and enforce rigorous standards for non deterministic AI systems, ensuring LLMs and predictive engines meet defense sector reliability requirements.
- Bridge Agile development with formal Systems Engineering practices, creating automated testing frameworks.
Key Responsibilities
- Lead validation and verification of AI systems by architecting automated testing frameworks for non deterministic outputs.
- Design and implement automated evaluation metrics such as RAGAS, faithfulness, and answer relevance for RAG pipelines.
- Develop prompt regression suites to monitor prompt drift and maintain quality of AI generated engineering documents.
- Construct robust integration tests to validate data consistency between AI agents and enterprise systems (e.g., SAP S/4HANA, Ariba).
- Create performance benchmarking tests to validate latency and throughput of forecasting models and risk scoring engines.
- Automate API validation for secure gateways, verifying RBAC and PII redaction logic.
- Ensure alignment of automated test cases with System Requirements and User Needs for V&V reports.
- Prepare Test Readiness packages for Stage Gate reviews.
- Manage the defect lifecycle between Requirements Quality Assistants and development teams.
- Bridge Agile development and formal Systems Engineering processes for defense sector deployment.
Qualifications And Experience
- 5+ years of experience in QA Automation, with at least 2 years focused on testing complex data driven applications, machine learning models, or AI agents.
- Experience in Defense, Aerospace, or highly regulated industries is a strong plus.
- Expert proficiency in Python for building custom test harnesses (Pytest) and standard automation libraries (Selenium/Playwright, Requests).
- Expert proficiency in crafting Performance Test Plans and Implementations (e.g., Locust, JMeter, K6).
- Experience utilizing frameworks for evaluating Large Language Models (e.g., DeepEval, TruLens, or custom Python evaluators).
- Proficiency with SQL and data validation tools (e.g., Great Expectations).
- Strong experience integrating automated tests into GitLab CI/CD pipelines.
- Familiarity with requirements management tools (e.g., Jira, Linear, Jama, Polarion).
- Strong hands on experience managing Test Reports and Artifacts (e.g., TestRail, Allure).
- Strong knowledge of maintaining code based frameworks (e.g., Git, GitLab).
- Strong hands on knowledge of modern Quality Engineering best practices (Shift Left, Test Pyramid, Mono repo).
- Ability to define pass/fail criteria for probabilistic systems and communicate Confidence Levels.
Required Technical Skills
- Expert proficiency in Python for building custom test harnesses using Pytest and automation libraries.
- Expert proficiency in performance testing tools such as Locust, JMeter, and K6.
- Experience with LLM evaluation frameworks including DeepEval, TruLens, or custom Python evaluators.
- Proficiency with SQL and data validation tools (e.g., Great Expectations).
- Strong experience integrating automated tests into GitLab CI/CD pipelines.
- Familiarity with requirements management tools (e.g., Jira, Linear, Jama, Polarion).
- Hands on experience managing test reports and artifacts using TestRail and Allure.
- Knowledge of maintaining code based frameworks using Git and GitLab.
- Knowledge of modern Quality Engineering best practices (Shift Left, Test Pyramid, Mono repo).
- Architecting automated frameworks to evaluate non deterministic AI outputs.
- Implementing automated metrics for RAG pipelines.
- Designing regression suites to monitor prompt drift.
Company And Project Focus
- Join a dynamic organization where innovation and quality assurance intersect.
- Contribute to a high impact project focused on automating quality assurance processes.
- Project emphasizes continuous improvement, collaboration, and adoption of cutting edge automation tools.
Location and Project Focus
- Location: Abu Dhabi, United Arab Emirates
- Company: Sister Company of the Client
- Project Focus: Integration, Verification, and Validation (IV&V) for the AI platform and Intelligent Supply Chain systems
Why This Role Matters
- This role represents the critical final line of defense in ensuring AI integrity and reliability.
- Pioneer groundbreaking methodologies for testing Generative AI within safety critical environments.
- Shape the future of AI validation, ensuring intelligent systems meet highest safety and performance criteria.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at cander
Senior Backend Engineer
Abu Dhabi, UAE
Cander is seeking a Senior Backend Engineer to architect and develop production-grade application infrastructure for AI-driven transformation. The role involves designing secure, high-performance backend systems for AI a
Senior Data Scientist
Abu Dhabi, UAE
A defense-focused AI company in Abu Dhabi seeks a Senior Data Scientist to lead development and deployment of generative AI and NLP solutions for engineering and supply chain applications. The role requires 5+ years of e
Senior Data Engineer
Abu Dhabi, UAE
Cander seeks a Senior Data Engineer to architect defense-grade Data Lakehouse architectures for AI-driven supply chain intelligence in Abu Dhabi. Requires 5+ years data engineering experience, expertise in Python, SQL, S
Senior Backend Engineer
Abu Dhabi, UAE
Cander seeks a Senior Backend Engineer in Abu Dhabi to architect and develop production-grade backend systems for AI-driven transformation. The role involves designing secure, high-performance microservices for AI agent
Senior Data Scientist
Abu Dhabi, UAE
Cander is seeking a Senior Data Scientist to lead AI initiatives for defense applications, developing production-grade generative AI, NLP, and predictive models. The role involves deploying LLMs, RAG pipelines, and time-
Senior Data Engineer
Abu Dhabi, UAE
Cander seeks a Senior Data Engineer to architect and develop core data infrastructure for AI-driven supply chain solutions. The role involves designing defense-grade Data Lakehouse architectures, integrating SAP S/4HANA
Senior Data Scientist
Abu Dhabi, UAE
We are seeking a Senior Data Scientist to lead the development and deployment of AI initiatives for defense capabilities. You will design and deploy generative AI, NLP, and predictive models to transform unstructured dat
Senior QA Automation Engineer
Abu Dhabi, UAE
We are seeking a Senior QA Automation Engineer to lead validation and verification strategies for AI systems in the defense sector. You will architect automated testing frameworks for LLMs and predictive engines, ensurin
Senior Backend Engineer
Abu Dhabi, UAE
Senior Data Scientist
Abu Dhabi, UAE
Senior Data Engineer
Abu Dhabi, UAE
Senior Backend Engineer
Abu Dhabi, UAE
Senior Data Scientist
Abu Dhabi, UAE
Senior Data Engineer
Abu Dhabi, UAE
Senior Data Scientist
Abu Dhabi, UAE
Senior QA Automation Engineer
Abu Dhabi, UAE