Technical Program Manager, AI Benchmarking

• Serve as PxD's day-to-day point of contact for the benchmarking program, working through PxD's embedded PMU team to align the Ministry of Agriculture and Farmers' Welfare on priority use cases, target languages and geographies, candidate models, benchmarking baselines, and approval pathways.
• Establish the governance framework for the consortium — partner roles, decision rights, steering cadence, and escalation routes — and keep it functioning as the program moves through its stages.
• Negotiate system and data access, including access to government advisory environments for substitution testing, and maintain a permissions tracker across government, academic, and advisory data sources.
• Facilitate workshops and consultations with government counterparts, model providers, and domain experts; manage the funder relationship and reporting.

• Own the benchmarking framework end to end: advisory use cases, benchmark tasks, datasets, human-rating rubrics, safety and language criteria, tool-use and source-grounding assessments, cost comparison templates, and decision thresholds.
• Work with existing benchmark initiatives to adapt and localise established assets rather than building a parallel benchmark.
• Coordinate technical benchmarking across three layers — standalone model evaluation, substitution testing within the live government advisory stack, and a neutral benchmarking harness where direct system access is unavailable — alongside assessment of speech recognition, translation, text-to-speech, and multilingual performance.
• Manage latency, reliability, and total-cost-of-ownership analysis, and oversee the cloud compute, GPU, and API resources that represent the single largest cost line in the program.
• Safeguard the independence and neutrality of the benchmarking by maintaining clear separation between benchmark development, model evaluation, and model development roles of partner organisations, with transparent disclosure of potential conflicts of interest.

• Oversee pre-deployment user evidence generation with agronomists, extension personnel, and compensated farmer participant testers, assessing whether advisory outputs are comprehensible, actionable, trustworthy, and usable in local languages.
• Manage IRB submission and third-party field vendor selection and oversight for user testing.
• Consolidate findings across technical performance, user evidence, safety, operational readiness, and cost into a single decision framework, and translate them into recommendations a non-technical government audience can act on.
• Author the final evidence synthesis and decision report identifying models ready for pilot deployment, models requiring further optimization, and key implementation risks.

• Contract and manage academic and technical partners, sandbox and compute hosts, domain-expert consultants, and field vendors, monitoring performance against agreed deliverables.
• Ensure contractual and donor compliance throughout implementation, including data-processing agreements and dataset licenses.
• Coordinate with finance and operations teams on budgets, procurement, and financial reporting against grant milestones.

• Work alongside research and field staff supporting the program, providing technical direction and day-to-day coordination.
• Document benchmarking tools, protocols, and methodology so future evaluations across agricultural and other public-sector advisory systems can build on this work.
• Contribute to PxD's institutional knowledge management and cross-country learning on AI advisory.

• Master's degree (or equivalent experience) in Computer Science, Data Science, Engineering, Public Policy, Economics, Agriculture, Development Studies, Management, or another relevant field.
• 5–8 years of progressively responsible experience managing complex technical programs or projects.
• Working fluency with modern AI systems — LLMs, retrieval-augmented generation, prompting, model evaluation, and the practical trade-offs between accuracy, latency, safety, and cost.
• Demonstrated experience managing multi-stakeholder projects involving government agencies, academic institutions, and external technical partners.
• Strong project management skills with the ability to run parallel workstreams against fixed donor milestones.
• Ability to translate technical evidence into clear, decision-oriented recommendations for non-technical senior audiences.
• Strong stakeholder management and relationship-building abilities.
• Excellent organizational skills and exceptional attention to detail.

• Demonstrated experience with AI model evaluation (academic/ personal/ work projects).
• Direct experience designing or running model evaluations, benchmarks, or structured A/B testing of AI systems.
• Experience with Indian-language NLP, speech technology (ASR/TTS), or multilingual model performance.
• Experience working with state or national government programs, or with digital public infrastructure.
• Background in agriculture, climate, technology, or international development.
• Familiarity with causal inference and experimental design.
• Comfort operating with significant ambiguity and limited existing infrastructure, having built or launched a new program from scratch.
• Proficiency in one or more Indian languages.

To apply, please visit the PxD website and use the online application form to submit a CV, cover letter, and brief responses to the screening questions.

Applications will be reviewed on a rolling basis until the application deadline of Sep 18, 2026, 6 pm IST.

PxD is committed to building an inclusive and diverse organisation.

We don’t make decisions based on gender, gender identity, class, ethnicity, sexual orientation, or any other personal characteristics that are unrelated to the role.

We understand that everyone has different lifestyles and we accommodate flexible working to make sure you are as productive and happy as you can be.

We will use any personal data you provide for recruitment purposes only.