Fullstack AI Quality Engineer – Applied & Agentic AI Systems

Description of the area

Job Responsibilities

• Generative & Agentic AI Testing Strategy: Design and implement comprehensive testing strategies specifically tailored for LLM-based applications and agentic systems, including evaluation of model responses, RAG pipelines accuracy, and overall system reliability. You will expand this to evaluate autonomous multi-agent behaviors and tool-use accuracy.
• Full-Stack Test Automation: Design and implement automated testing solutions for continuous evaluation of LLM applications, including integration tests, performance tests, and specialized AI behavior tests. This spans both our Python backend services and modern TypeScript/React/Angular frontends.
• Quality Assurance Framework & Compliance: Utilize Roche's testing frameworks that address both traditional software quality aspects and AI-specific concerns such as output consistency, contextual accuracy, and ethical compliance; co-create and maintain such frameworks when required. Create and maintain comprehensive test documentation testing strategy, test cases, and testing guidelines specific to AI applications and compliant with Roche practices.
• Edge Case, Bias, and Fairness Testing: Identify and develop test scenarios for edge cases in LLM behavior, including handling of ambiguous inputs, potential biases, and unexpected response patterns. Design and execute tests to identify potential biases in model outputs and ensure fair treatment across different user groups and use cases.
• Security & Performance Testing: Collaborate within development teams to test for potential vulnerabilities specific to LLM applications, including prompt injection, data leakage, and other AI-specific security concerns. Collaborate with other engineers to conduct thorough performance testing of GenAI applications, including response time analysis, load testing, and resource utilization monitoring.

Qualifications

• Hold B.Sc., B.Eng., M.Sc., M.Eng. in Computer Science, Software Engineering, or equivalent degree with strong background in testing methodologies.
• Be passionate about quality assurance and testing, particularly in the context of AI and GenAI applications and agents.
• Have strong analytical and problem-solving skills with attention to detail.
• Utilize critical thinking and problem-solving when using AI prompts (in e.g. NLP, data distributions and edge cases).
• Be proactive in identifying potential issues and suggesting improvements.

Education / Experience

Must-have

• Comprehensive Full-Stack Testing Mastery
• Core Languages & Testing Frameworks: Advanced command of Python and standard testing frameworks (e.g., pytest, allure), alongside practical familiarity with specialized AI evaluation tools. High competence in TypeScript-driven test frameworks (such as Playwright, Jest, and Cypress) for thoroughly verifying dynamic generative user interfaces.
• End-to-End Automation & CI/CD: Proven expertise in architecting and maintaining end-to-end automated testing suites that bridge traditional UI/API validation with complex, asynchronous AI/LLM evaluation pipelines.
• Pipeline Integration: Hands-on experience integrating these comprehensive suites within GitHub Actions or modern CI/CD pipelines to ensure continuous, reliable quality checks throughout the entire development lifecycle.
• Advanced AI Quality & Evaluation Mastery
• AI Evaluation Frameworks: Expertise in implementing evaluation harnesses (e.g., RAGAS, DeepEval, Promptfoo) to quantify LLM performance, hallucination rates, and answer relevancy.
• Agentic Workflow Validation: Ability to test autonomous agentic loops, including tool-use accuracy, step-by-step reasoning verification, and goal-directed performance metrics.
• Evaluation Datasets: Experience in curating, managing, and versioning 'Golden Datasets' for benchmarking model responses against ground truth in both development and production environments.
• Observability & Feedback Loops: Leveraging tracing (e.g., LangSmith, Arize Phoenix) to monitor production inputs/outputs, identify drift, and capture user feedback for continuous improvement.
• Basic broad AI knowledge: Understanding of LLM architectures, RAG systems, agents, and common failure modes in AI applications.
• Python Expertise: Advanced proficiency in writing robust, maintainable Python code for test automation (e.g., pytest). Experience using Python to orchestrate complex AI evaluation workflows, process large datasets for benchmarking, and automate interactions with LLM/AI services. Ability to integrate Python scripts into CI/CD pipelines for automated quality checks and performance observability in agentic systems.

Should-have

Could-have

Additional Qualifications

• Stay current with the latest developments in AI and GenAI testing methodologies and best practices.
• Have excellent communication skills and the ability to clearly document findings and recommendations.
• Be able to communicate in English at the level of C1+.
• Located in Hyderabad, India, with working hours structured to capture the 'golden hours' of overlap with Central European Time (typically running through the IST evening).
• #Hyd2026

Who we are