Automated AI Tester
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
Kayana seeks an experienced Automated AI QA Tester to ensure reliability and performance of AI-driven ordering and payment systems. The role involves building automated test frameworks, designing conversation-based test scenarios, and validating payment flows.
Key Skills for This Role
Responsibilities
- Build, maintain, and scale automated test frameworks for AI powered ordering and payment systems
- Design conversation based test scenarios for AI ordering assistants
- Develop AI driven and automated E2E test suites simulating real world customer journeys
- Validate payment flows (UPI, cards, wallets, failures, refunds, edge cases)
- Create synthetic user simulations and automated load scenarios to stress test AI systems
- Evaluate AI responses for accuracy, consistency, and task completion
- Perform regression testing for AI model updates, prompt changes, and product releases
- Identify issues in AI reasoning, API integrations, and transaction flows
- Integrate automated tests into CI/CD pipelines
- Collaborate closely with engineering, product, and AI teams to improve product quality
- Leverage AI assisted testing tools to increase test coverage and efficiency
Requirements
- 4+ years of experience in QA automation, software testing, or test engineering
- Strong experience designing and implementing automated E2E testing frameworks
- Demonstrated experience using AI tools and AI assisted testing methodologies
- Strong understanding of AI/LLM testing concepts (Prompt testing, Hallucination detection, etc.)
- Experience with automation tools such as Playwright, Selenium, Cypress, Postman
- Strong programming skills in Python and/or JavaScript
- Experience with API testing and service integration validation
- Familiarity with payment systems, fintech workflows, or transactional platforms
- Ability to design structured test cases for conversational AI systems
- Experience validating multi step user journeys across web, mobile, and backend systems
- Strong debugging, root cause analysis, and problem solving skills
Full Job Posting
Role Overview
- We are looking for an experienced Automated AI QA Tester to ensure the reliability, accuracy, safety, and performance of AI driven ordering and payment workflows at Kayana.
Key Responsibilities
- Build, maintain, and scale automated test frameworks for AI powered ordering and payment systems.
- Design conversation based test scenarios for AI ordering assistants (menu browsing, cart building, checkout, cancellations).
- Develop AI driven and automated E2E test suites that simulate real world customer journeys.
- Validate payment flows (UPI, cards, wallets, failures, refunds, edge cases).
- Create synthetic user simulations and automated load scenarios to stress test AI systems at scale.
- Evaluate AI responses for accuracy, consistency, and task completion.
- Perform regression testing for AI model updates, prompt changes, and product releases.
- Identify issues in AI reasoning, API integrations, and transaction flows.
- Integrate automated tests into CI/CD pipelines.
- Collaborate closely with engineering, product, and AI teams to improve product quality and reliability.
- Leverage AI assisted testing tools and frameworks to increase test coverage, automate test generation, and improve testing efficiency.
Required Skills
- 4+ years of experience in QA automation, software testing, or test engineering.
- Strong experience designing and implementing automated E2E testing frameworks.
- Demonstrated experience using AI tools and AI assisted testing methodologies for test creation, execution, analysis, and validation.
- Strong understanding of AI/LLM testing concepts, including Prompt testing, Hallucination detection, Response evaluation, Groundedness validation, AI safety testing & Evaluation metrics.
- Experience with automation tools such as: Playwright, Selenium, Cypress & Postman.
- Strong programming skills in Python and/or JavaScript.
- Experience with API testing and service integration validation.
- Familiarity with payment systems, fintech workflows, or transactional platforms.
- Ability to design structured test cases for conversational AI systems, chatbots, and AI agents.
- Experience validating multi step user journeys across web, mobile, and backend systems.
- Strong debugging, root cause analysis, and problem solving skills.
Nice to Have
- Experience testing LLM powered applications, AI agents, or autonomous workflows.
- Experience with Retrieval Augmented Generation (RAG) systems and vector databases.
- Familiarity with AI evaluation frameworks such as: OpenAI Evals, DeepEval, LangSmith, Ragas, Similar evaluation platforms.
- Experience in food ordering, restaurant technology, POS, e commerce, or fintech products.
- Experience with CI/CD platforms such as: GitHub Actions, Jenkins, GitLab CI/CD.
- Experience with performance, load, and reliability testing of AI powered applications.
Soft Skills
- Strong analytical and investigative mindset.
- Exceptional attention to detail and edge case discovery.
- Ability to work cross functionally with engineering, product, and AI teams.
- Excellent written and verbal communication skills.
- Comfortable working in fast paced startup environments with rapidly evolving products and requirements.
- Proactive ownership mindset with a focus on quality, reliability, and customer experience.
Why Join Us?
- At Kayana, you’ll be part of a fast growing global company where your contributions will make a direct impact.
- A clear path for career growth.
- The opportunity to set up and shape the compliance framework for a growing international business.
- A supportive and collaborative environment where your success is celebrated.
Compensation
- Starting salary: 8K AED per month
Work Arrangement
- This role requires full time, on site work from Monday to Saturday (6 days a week).
- Candidates should be comfortable with a fully office based setup, as remote or hybrid work is not available for this position.
About Kayana
- Kayana is one of the fastest growing companies, empowering small and medium sized businesses with cutting edge technology in ordering and payments.
- Operations across the USA, UK, Europe, Australia, and the Middle East.
- Every employee is supported with a personalized development plan, offering clear and achievable career paths for long term growth.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Kayana | Ordering & Payment Solutions
Group Finance Specialist
Dubai, UAE
Kayana is seeking a Group Finance Specialist to support financial operations across multiple group companies. The role involves managing payment requests, preparing payment schedules, monitoring recurring payments, and c
Compliance Specialist
Dubai, UAE
Kayana seeks a detail-oriented Compliance Specialist to support global compliance functions, focusing on client onboarding, AML/KYC regulations, and risk assessment. The ideal candidate has 3-6 years of experience in fin
Company Driver
Dubai, UAE
Kayana is seeking a reliable and professional Driver to transport employees, visitors, and materials across Dubai. The role requires a valid UAE driving license, minimum 2 years of UAE driving experience, and strong know
Business Development Associate
Dubai, UAE
Kayana seeks a results-driven Business Development Associate to join its Dubai team. The role involves converting inbound enquiries and proactively generating leads through outbound prospecting. Candidates should have at
Full Stack Developer
Dubai, UAE
Kayana is hiring a Full Stack Developer for an on-site role in Dubai. You will design, develop, and maintain customer-facing applications and backend services for ordering and payment solutions. Requires 2-6 years of exp
Business Development Associate - Spanish Speaker
Dubai, UAE
Kayana seeks an enthusiastic Business Development Associate (Spanish Speaker) to join their Dubai team. The role is a mix of inbound and outbound sales, converting enquiries and generating leads via phone, email, and liv
Android Developer
Dubai, UAE
Kayana is seeking an Android Developer with 3+ years of experience to develop and maintain Android applications. The role requires expertise in Kotlin, Jetpack Compose, Flutter, and Android SDK. This is a full-time, on-s
iOS Developer
Dubai, UAE
Kayana is seeking an iOS Developer to join their mobile engineering team in Mumbai. The role involves developing secure, high-performance iOS apps integrating payment gateways and financial APIs. Requires strong proficie
Group Finance Specialist
Dubai, UAE
Compliance Specialist
Dubai, UAE
Company Driver
Dubai, UAE
Business Development Associate
Dubai, UAE
Full Stack Developer
Dubai, UAE
Business Development Associate - Spanish Speaker
Dubai, UAE
Android Developer
Dubai, UAE
iOS Developer
Dubai, UAE