Lead / Manager - AI Engineering

Key Responsibilities

• Translate business needs into testable GenAI and Agentic Engineering solutions, clear outputs, and measurable success criteria; define scope boundaries (what the system should not attempt), including risks.
• Run feasibility assessments to choose the right approach: prompting vs RAG vs fine-tuning vs classical ML.
• Select and develop models based on task requirements (reasoning vs extraction vs classification) working with AI Engineering to understand latency/cost, and risk profile.
• Design prompting strategies: instruction design, few-shot sets, structured outputs, tool/agent prompts, and robustness patterns. This will be implemented as an MVP and iterate based on eval results.
• Establish prompt iteration methodology driven by evals (not anecdotal testing): prompt versioning, ablations, and change control.
• Define the evaluation plan for GenAI systems and agentic workflows- designing and implementing evaluation from LLM as a judge and ensure evaluation includes fairness and bias considerations where applicable. Define acceptance thresholds and release gates tied to these metrics.
• Own experimentation and model improvements: Run structured experiments ( across prompts, retrievers, chunking, models ).
• Develop out methods for identifying model failures such as hallucination types, retrieval misses, instruction-following errors, formatting failures etc
• Provide recommendations for improvements grounded in evidence: what to change, expected lift, and trade-offs.
• Deliver an engineering-ready handoff: prompt packages and versioning approach, RAG configuration, tool schemas (if agentic), evaluation harness, datasets/ground truth, metric definitions, and go/no-go gates .
• Design scalable and secure Agentic AI architectures adhering to best practices in data engineering, MLOps and LLMOps .
• 5-10 years of overall AI/ML experience out if which at least 2 to 3 years of Generative AI solutions.
• Strong background in applied ML, data science, LLM and Agentic AI Engineering Systems with demonstrated delivery and client facing experience.
• Deep expertise in evaluation design, metrics, and dataset curation for LLM systems.