Python QA Automation Engineer

Responsibilities

• Develop automated testing frameworks to validate MCP Servers and related AI systems
• Design and implement evaluation strategies for LLM accuracy, safety, and reliability
• Create automated tests using Python, Pytest, and BDD frameworks
• Build quality gates into CI/CD pipelines to maintain continuous assurance
• Identify and address agentic AI failure modes such as hallucination, latency, and incorrect tool usage
• Collaborate with engineering, QA, and product teams to define quality metrics and acceptance criteria
• Contribute to Agile ceremonies, ensuring testing practices align with sprint goals
• Prepare detailed reporting on quality outcomes and improvement opportunities
• Maintain documentation for test cases, evaluation pipelines, and validation strategies

Requirements

• Strong programming experience in Python applied to test automation and evaluation
• Expertise in Pytest and familiarity with BDD frameworks such as Behave or Cucumber
• Knowledge of LLM evaluation approaches including RAGAS, DeepEval, or custom pipelines
• Understanding of common agentic AI issues such as hallucination, tool misuse, and performance bottlenecks
• Familiarity with automated testing of AI workflows, distributed systems, or microservices environments
• Strong grasp of Agile delivery methodologies and CI/CD integration for quality checks
• Excellent communication and problem-solving skills with a focus on accuracy and reliability

Nice to have