Associate Lead - Testing (QA + MLOps)

LLM-based applications using Amazon Bedrock

Prompt engineering validation

Multi-agent orchestration testing

Chatbot & Voice bot conversational testing

Intent classification validation

Conversation drift & fallback validation

API contract validation for LLM integrations

BLEU / ROUGE scoring

Embedding similarity scoring

Response consistency

Safety scoring frameworks

Design reusable AI testing accelerators

Create AWS-aligned AI test automation frameworks (Python-first)

Develop synthetic data generation strategies

Establish AI quality scorecards

Build an internal AI QA Center of Excellence

Lead AI/ML quality strategy workshops

Perform AI risk & readiness assessments

Present quality architecture to CXOs

Drive QA transformation programs

Mentor QA teams on AWS-based AI testing

Own delivery for AI testing engagements end-to-end

8–12+ years in Quality Engineering

Strong test strategy, automation & governance experience

Experience leading QA transformation initiatives

Experience building frameworks from scratch AI/ML & GenAI Expertise

Deep understanding of ML lifecycle

Experience testing ML models (NLP preferred)

Hands-on experience validating LLM applications

Strong understanding of:

Prompt engineering

RAG architecture

Embeddings

Bias & explainability AWS AI/ML Expertise

Hands-on experience with:

Amazon SageMaker (training, deployment, monitoring)

Amazon Bedrock (LLM integration & evaluation)

S3-based data pipelines

AWS IAM (security validation)

CloudWatch monitoring

Lambda & API Gateway integrations

AWS CI/CD (CodePipeline / CodeBuild preferred)

Infrastructure as Code (Terraform / CloudFormation)

Observability in AI systems

Cost monitoring for ML workloads