Principal Software Engineer
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
Role Overview
Role: AI Test / Automation Engineer Location: Bangalore India
Department: AI Engineering / Quality Assurance Experience Level: Mid to Senior
We are looking for a highly motivated Senior AI Test / Automation Engineer to design and scale automated validation frameworks for AI/ML models, LLM-based applications, and agentic systems . This role is critical to ensure that AI solutions meet enterprise standards for quality, reliability, safety, and compliance before and after production deployment.
Key Skills for This Role
Full Job Posting
Overview
Role: AI Test / Automation Engineer Location: Bangalore India
Department: AI Engineering / Quality Assurance Experience Level: Mid to Senior
We are looking for a highly motivated Senior AI Test / Automation Engineer to design and scale automated validation frameworks for AI/ML models, LLM-based applications, and agentic systems . This role is critical to ensure that AI solutions meet enterprise standards for quality, reliability, safety, and compliance before and after production deployment.
Key Responsibilities
- Build and maintain AI test automation frameworks for pre-qualification and continuous validation of models and agent workflows
- Develop comprehensive test suites , including: Unit, integration, and end-to-end (E2E) Functional, regression, performance, and safety testing
- Unit, integration, and end-to-end (E2E)
- Functional, regression, performance, and safety testing
- Validate AI system behavior , including: Non-deterministic LLM outputs Hallucinations and edge cases Multi-step agent decision-making
- Non-deterministic LLM outputs
- Hallucinations and edge cases
- Multi-step agent decision-making
- Design and manage evaluation systems : Golden datasets Benchmarking pipelines (accuracy, latency, reliability)
- Golden datasets
- Benchmarking pipelines (accuracy, latency, reliability)
- Automate testing within CI/CD pipelines for model updates, prompt changes, and tool integrations
- Implement observability and telemetry to enable traceability, monitoring, and audit readiness
- Collaborate cross-functionally with ML, MLOps, Product, and Security teams to define quality gates and release criteria
- Track and report quality KPIs , including test coverage, defect leakage, and system reliability
- Drive root-cause analysis and continuous improvement across the AI testing lifecycle
Core Engineering
Strong programming skills in Python ; familiarity with Bash, TypeScript, or Go
Experience with test automation frameworks such as PyTest, Playwright, Selenium, or Cypress
Proficiency in CI/CD tools (GitHub Actions, Jenkins, GitLab CI)
Experience with cloud platforms (AWS, Azure, GCP) and containers (Docker, Kubernetes)
AI / ML & Agentic Systems
Hands-on experience with LLM ecosystems (OpenAI, Anthropic, Bedrock)
Familiarity with: RAG architectures and vector databases (Pinecone, Weaviate) Agent frameworks (LangChain, LlamaIndex, AutoGen)
RAG architectures and vector databases (Pinecone, Weaviate)
Agent frameworks (LangChain, LlamaIndex, AutoGen)
AI Testing Techniques
Experience with non-deterministic testing approaches (statistical assertions, tolerance thresholds)
Knowledge of evaluation methods : LLM-as-a-judge BLEU, ROUGE, semantic similarity scoring
LLM-as-a-judge
BLEU, ROUGE, semantic similarity scoring
Experience with prompt and agent regression testing
Understanding of AI safety testing , including adversarial testing, bias/fairness validation, and jailbreak detection
Tooling (Preferred)
AI testing & observability tools: LangSmith, TruLens, Arize, Weights & Biases
Evaluation tools: DeepEval, Ragas, PromptFoo, Giskard
Monitoring: Prometheus, Grafana, OpenTelemetry
Soft Skills
- Strong analytical and problem-solving skills
- Excellent communication and cross-functional collaboration
- Data-driven mindset with focus on quality KPIs
- Detail-oriented with a strong bias toward automation and scalability
Experience Requirements
- 7+ years in QA, SDET, or test automation engineering
- Proven experience building and scaling automation frameworks
- Hands-on experience with AI/ML systems or LLM-based applications
- Experience testing RAG pipelines or agentic workflows
- Owned end-to-end AI test strategy and architecture
- Defined quality metrics and release gates
- Delivered scalable validation pipelines for production AI systems
- Supported audit and compliance readiness
Preferred
Experience in enterprise or regulated environments (SOC2, ISO 27001, etc.)
Exposure to: Shift-left testing practices Production observability and monitoring Chaos or resilience testing
Shift-left testing practices
Production observability and monitoring
Chaos or resilience testing
Education
Bachelor’s or Master’s degree in Computer Science, Software Engineering, or related field
Nice-to-have
ISTQB certification
Cloud/ML certifications (AWS, Azure, GCP)
AI testing certifications
What Success Looks Like
AI systems that are accurate, reliable, and safe
Fully automated test pipelines integrated into CI/CD
Measurable improvements in defect leakage and model quality
Strong observability and auditability across AI systems
Scalable validation frameworks supporting rapid AI innovation
About Cadence
Verified company details for this employer are not available yet.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Cadence
Experienced Die Maker / EDM Technician (Deep Draw Stamping)
Watertown, USA
Lead Verification Engineer
Bengaluru, IND
Sr Sales Finance Analyst
Bracknell, GBR
Product Engineering Architect
Noida, IND
Experienced Die Maker / EDM Technician (Deep Draw Stamping)
Watertown, USA
Facilities Maintenance Technician
Watertown, USA
Outbound Team Supervisor
, USA
Quality Inspector
Watertown, USA
Validation Engineer II
Cranberry Township, USA