Quality Assurance Engineer
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
Arcus Search is seeking a Senior QA Automation Engineer to lead validation and verification strategies for AI transformation in defense and enterprise sectors.
Key Skills for This Role
Responsibilities
- Architect automated frameworks to evaluate Generative AI outputs for hallucination, consistency, and factual accuracy against Gold Standard datasets
- Implement automated metrics (e.g., RAGAS, faithfulness, answer relevance) to verify Retrieval Augmented Generation pipelines
- Design regression suites to monitor prompt drift for AI generated engineering documents
- Build robust tests to validate data consistency between AI agents and critical systems (e.g., SAP S/4HANA, Ariba)
- Design tests to validate latency and throughput for forecasting models and risk scoring engines using Locust, JMeter, or K6
- Automate testing of secure API gateways, verifying Role Based Access Control (RBAC) and PII redaction logic
- Map automated test cases to System Requirements to create digital evidence for formal V&V reports
- Prepare Test Readiness packages for formal reviews, providing quantitative evidence for MVP to Production transitions
- Manage the feedback loop between Requirements Quality Assistants and development teams, tracing AI logic defects to specific model versions
Requirements
- 5+ years in QA Automation, with 2+ years focused on complex data driven applications, ML models, or AI agents
- Expert proficiency in Python (Pytest) and standard libraries (Selenium/Playwright, Requests)
- Hands on experience with LLM evaluation frameworks (e.g., DeepEval, TruLens) and Ground Truth dataset management
- Proficiency in crafting Performance Test Plans and implementations (Locust, K6, etc.)
- Expertise in SQL and data quality tools (e.g., Great Expectations) for Data Lakehouses and Vector Databases
- Strong experience integrating quality gates into GitLab CI/CD pipelines
- Deep understanding of modern QE practices including Shift Left, Test Pyramid, and Mono repo architectures
- Background in Defense, Aerospace, or highly regulated industries is a strong plus
- Familiarity with IV&V processes is highly desirable
Full Job Posting
Overview
- Seeking a Senior QA Automation Engineer to lead validation and verification strategies for AI transformation in defense and enterprise sectors.
- Define 'what good looks like' for non deterministic AI systems, ensuring LLMs and predictive engines meet strict reliability standards.
- Act as bridge between Agile development and formal Systems Engineering.
Key Responsibilities
- AI & LLM Validation: Architect automated frameworks to evaluate Generative AI outputs for hallucination, consistency, and factual accuracy.
- RAG Evaluation: Implement automated metrics (e.g., RAGAS, faithfulness, answer relevance) to verify Retrieval Augmented Generation pipelines.
- Prompt Regression: Design regression suites to monitor prompt drift, ensuring model updates do not degrade AI generated document quality.
- Enterprise Integration: Build robust tests to validate data consistency between AI agents and critical systems (e.g., SAP S/4HANA, Ariba).
- Performance Benchmarking: Design tests to validate latency and throughput for forecasting models and risk scoring engines using Locust, JMeter, or K6.
- API & Security Validation: Automate testing of secure API gateways, verifying RBAC and PII redaction logic.
- V Model Alignment: Map automated test cases to System Requirements to create digital evidence for formal V&V reports.
- Stage Gate Compliance: Prepare Test Readiness packages for formal reviews, providing quantitative evidence for MVP to Production transitions.
- Defect Lifecycle Management: Manage feedback loop between Requirements Quality Assistants and development teams.
What You’ll Bring Technical Requirements
- Expert proficiency in Python (Pytest) and standard libraries (Selenium/Playwright, Requests).
- Hands on experience with LLM evaluation frameworks (e.g., DeepEval, TruLens) and Ground Truth dataset management.
- Proficiency in crafting Performance Test Plans and implementations (Locust, K6, etc.).
- Expertise in SQL and data quality tools (e.g., Great Expectations) for Data Lakehouses and Vector Databases.
- Strong experience integrating quality gates into GitLab CI/CD pipelines.
- Deep understanding of modern QE practices including Shift Left, Test Pyramid, and Mono repo architectures.
Professional Qualification
- 5+ years in QA Automation, with 2+ years focused on complex data driven applications, ML models, or AI agents.
- Background in Defense, Aerospace, or highly regulated industries is a strong plus.
- Familiarity with IV&V processes is highly desirable.
- Ability to define pass/fail criteria for probabilistic systems and communicate Confidence Levels to engineering leadership.
Contract Type
- This is a Fixed Term Contract role.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Arcus Search
SAP Procurement Architect - Commercial Shipping
Abu Dhabi Emirate, UAE
Arcus Search seeks an SAP Procurement Architect for a 24-month contract in Abu Dhabi, leading S/4HANA migration for commercial shipping. Requires strong SAP S/4HANA procurement experience and knowledge of maritime operat
SAP Finance Architect - Commercial Shipping
Abu Dhabi Emirate, UAE
Arcus Search is seeking an SAP Finance Architect with expertise in Commercial Shipping for a 24-month contract in Abu Dhabi. The role involves leading S/4HANA migration and solution design, engaging with CxOs, and requir
Quantitative Researcher
Dubai, UAE
A proprietary trading firm seeks a Quantitative Researcher to develop systematic trading strategies for crypto markets. You will research, backtest, and deploy quantitative models, working with experienced researchers an
BIM Software Developer
Abu Dhabi Emirate, UAE
Arcus Search seeks a BIM Software Developer for a remote 3-month contract to extract and transform Revit and IFC model data using Autodesk Platform Services. The role involves building data pipelines, geometry checks, an
Technical Program Manager
Abu Dhabi Emirate, UAE
Arcus Search seeks a Senior Technical Program Manager for a 3-month contract to lead the SuperHive AI platform delivery. You will own end-to-end execution across multiple squads, manage dependencies, and drive cross-func
Front End Engineer - Contract
Abu Dhabi, UAE
We are looking for a Front End Engineer for a 3-month remote contract. The ideal candidate will design, develop, and test responsive web and mobile applications using JavaScript, HTML, and CSS. Requires 2+ years of front
Senior AI Engineer
Abu Dhabi Emirate, UAE
Arcus Search is seeking a Senior AI Engineer to design and build the core AI orchestration layer for an enterprise platform. You will develop multi-agent workflows, RAG pipelines, and production APIs using FastAPI. Requi
Software Engineer - Frontend
Abu Dhabi Emirate, UAE
Arcus Search is seeking a highly skilled Frontend Engineer to design, develop, and enhance user interfaces for AI-powered applications. The role requires 8+ years of experience with React, Vue.js, or Angular, and strong
SAP Procurement Architect - Commercial Shipping
Abu Dhabi Emirate, UAE
SAP Finance Architect - Commercial Shipping
Abu Dhabi Emirate, UAE
Quantitative Researcher
Dubai, UAE
BIM Software Developer
Abu Dhabi Emirate, UAE
Technical Program Manager
Abu Dhabi Emirate, UAE
Front End Engineer - Contract
Abu Dhabi, UAE
Senior AI Engineer
Abu Dhabi Emirate, UAE
Software Engineer - Frontend
Abu Dhabi Emirate, UAE