Quality Assurance Engineer
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
AI71 is seeking a Quality Assurance Engineer to validate AI and LLM outputs, build automated test frameworks, and ensure system integrity in a defense/aerospace context. The role requires 5+ years in QA automation with Python, AI evaluation frameworks, and CI/CD integration.
Key Skills for This Role
Responsibilities
- Architect automated frameworks to evaluate Generative AI outputs for hallucination, consistency, and factual accuracy against Gold Standard datasets
- Implement automated metrics (e.g., RAGAS, faithfulness, answer relevance) to verify RAG pipelines
- Design regression suites to monitor prompt drift and ensure model updates do not degrade quality
- Build robust tests to validate data consistency between AI agents and critical systems (e.g., SAP S/4HANA, Ariba)
- Design tests to validate latency and throughput for forecasting models and risk scoring engines using Locust, JMeter, or K6
- Automate testing of secure API gateways, verifying RBAC and PII redaction logic
- Map automated test cases to System Requirements to create digital evidence for V&V reports
- Prepare Test Readiness packages for formal reviews
- Manage the feedback loop between Requirements Quality Assistants and development teams
Requirements
- 5+ years in QA Automation
- 2+ years focused on complex data driven applications, ML models, or AI agents
- Expert proficiency in Python (Pytest) and standard libraries (Selenium/Playwright, Requests)
- Hands on experience with LLM evaluation frameworks (e.g., DeepEval, TruLens)
- Proficiency in crafting Performance Test Plans and implementations (Locust, K6, etc.)
- Expertise in SQL and data quality tools (e.g., Great Expectations)
- Strong experience integrating quality gates into GitLab CI/CD pipelines
- Deep understanding of modern QE practices, including Shift Left, Test Pyramid, and Mono repo architectures
- Background in Defense, Aerospace, or highly regulated industries is a strong plus
- Familiarity with IV&V processes is highly desirable
Full Job Posting
Key Responsibilities
- Architect automated frameworks to evaluate Generative AI outputs for hallucination, consistency, and factual accuracy against Gold Standard datasets
- Implement automated metrics (e.g., RAGAS, faithfulness, answer relevance) to verify RAG pipelines
- Design regression suites to monitor prompt drift and ensure model updates do not degrade quality
- Build robust tests to validate data consistency between AI agents and critical systems (e.g., SAP S/4HANA, Ariba)
- Design tests to validate latency and throughput for forecasting models and risk scoring engines using Locust, JMeter, or K6
- Automate testing of secure API gateways, verifying RBAC and PII redaction logic
- Map automated test cases to System Requirements to create digital evidence for V&V reports
- Prepare Test Readiness packages for formal reviews
- Manage the feedback loop between Requirements Quality Assistants and development teams
What You'll Bring
- 5+ years in QA Automation, with 2+ years focused on complex data driven applications, ML models, or AI agents
- Expert proficiency in Python (Pytest) and standard libraries (Selenium/Playwright, Requests)
- Hands on experience with LLM evaluation frameworks (e.g., DeepEval, TruLens) and Ground Truth dataset management
- Proficiency in crafting Performance Test Plans and implementations (Locust, K6, etc.)
- Expertise in SQL and data quality tools (e.g., Great Expectations) for Data Lakehouses and Vector Databases
- Strong experience integrating quality gates into GitLab CI/CD pipelines
- Deep understanding of modern QE practices, including Shift Left, Test Pyramid, and Mono repo architectures
- Background in Defense, Aerospace, or highly regulated industries is a strong plus
- Familiarity with IV&V processes is highly desirable
Why AI71?
- Final Line of Defense: You define whether an AI agent is trusted to negotiate a contract or design a critical system component
- Tax Free Compensation: A market leading, 100% tax free salary and benefits package
- Full Relocation Support: Comprehensive assistance including flights, housing support, and visa sponsorship for you and your family to Abu Dhabi
- Pioneering Work: Set the global standard for how defense organizations validate intelligent systems
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at AI71
Deployment Strategist
Abu Dhabi, UAE
The role involves managing end-to-end deployment for enterprise clients, ensuring adoption and value realization while collaborating with technical stakeholders and improving de...
Software Engineer - Frontend
Abu Dhabi, UAE
AI71 is seeking a Senior Frontend Engineer to design and develop user interfaces for AI-powered applications in Abu Dhabi. The role involves building responsive frontends using modern JavaScript frameworks and collaborat
Technical Program Manager
Abu Dhabi, UAE
About ai71 AI71 is an industry leader in artificial intelligence, delivering innovative solutions that empower developers, businesses and governments to solve complex challenges...
Product Manager
Abu Dhabi, UAE
AI71 seeks a Product Manager to own a product domain, define vision, write PRDs, and work daily with engineering to ship features. Requires 7+ years in product management for technical/enterprise products and deep domain
DevOps Engineer
Abu Dhabi, UAE
AI71 is seeking a skilled DevOps Engineer to build, deploy, and maintain scalable systems and infrastructure. The role involves designing CI/CD pipelines, managing cloud resources, automating infrastructure, and ensuring
Senior Solution Architect
Abu Dhabi, UAE
We are seeking an experienced Solution Architect to design and own the solution architecture for scalable SaaS and custom products. The role requires strong expertise in microservices architecture, cloud-agnostic design,
Product Manager
Abu Dhabi, UAE
Lead product strategy and development for AI solutions, collaborating with customers and engineering teams, while utilizing strong technical and communication skills.
Senior Machine Learning Engineer
Abu Dhabi, UAE
You will be building the intelligence that drives critical national infrastructure. Your models will not just generate text or predictions; they will directly influence the desi...
Deployment Strategist
Abu Dhabi, UAE
Software Engineer - Frontend
Abu Dhabi, UAE
Technical Program Manager
Abu Dhabi, UAE
Product Manager
Abu Dhabi, UAE
DevOps Engineer
Abu Dhabi, UAE
Senior Solution Architect
Abu Dhabi, UAE
Product Manager
Abu Dhabi, UAE
Senior Machine Learning Engineer
Abu Dhabi, UAE