smartrecruiters
Lead AI Engineer – Quality Engineering
KMS Technology
USA
Full-time
Senior · 6+ years experience
Hybrid
Discovered 2 days ago
PythonTypeScriptREST APIsLangGraphOpenAI Agents SDKCrewAI
Free
Job Fit Check
Base Career helps you apply smarter for this job.
?%
Ready to ScanKey skills for this role
PythonTypeScriptREST APIs
Key Skills for This Role
PythonTypeScriptREST APIsLangGraphOpenAI Agents SDKCrewAI
Full Job Posting
More about KMS Technology
Website: https://kms-technology.com
Linkedin: https://www.linkedin.com/company/kms-technology
We are looking for a Lead AI Engineer to design, build, and operate intelligent agents that automate software testing end-to-end.
This is an AI engineering role applied to quality engineering — not a traditional test automation role with AI added on top. You will own agentic workflows across test-case generation, automation-script creation, execution, validation, failure analysis, self-healing, and continuous test maintenance. The role integrates AI agents with application knowledge, automation tools, APIs, databases, and CI/CD pipelines to enable reliable end-to-end testing and sets the technical direction for how agentic quality engineering is delivered across teams.
You will lead the technical direction of this capability and mentor a team of engineers (team size: TBC) as the practice scales.
Key Responsibilities
- Build agentic systems
- Design and develop autonomous and multi-agent workflows for test automation.
- Build AI agents using LLMs, prompt engineering, RAG, tool calling, and MCP-based integrations.
- Develop agents for test generation, code creation, validation, debugging, failure triage, and self-healing.
- Maintain application knowledge bases (embeddings, retrieval, context strategy) and continuously improve agent decision-making.
- Make them reliable
- Define agent instructions, guardrails, validation checkpoints, and human-in-the-loop review processes.
- Establish evaluation practice: golden datasets, regression evals, LLM-as-judge where appropriate, and clear accuracy/reliability metrics.
- Instrument agents with tracing and observability; monitor and improve accuracy, reliability, latency, token cost, and maintainability in production.
- Integrate and ship
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career