GDM Staff Software Engineer, Agent Data Quality, DeepMind

Responsibilities

• Construct rigorous quantitative benchmarks and automated evaluation frameworks (including LLM-as-a-judge) to measure agent capabilities in reasoning, planning, and tool use.
• Develop end-to-end data flywheel on agent evalset curation, and automate and productionize the flywheel with agentic data pipelines.
• Develop a dedicated platform for "Self-Service Judges," allowing researchers to rapidly prototype and test new LLM-based classifiers and annotations against historical trajectory data.
• Establish a formal ecosystem for metric lifecycle management, including a sandbox for validating new metrics against a library of high-confidence calibration experiments.
• Productionize automated insights tools to identify the top movers of metric deltas across complex slices such as task type and user cohort.
• - Construct rigorous quantitative benchmarks and automated evaluation frameworks (including LLM-as-a-judge) to measure agent capabilities in reasoning, planning, and tool use. - Develop end-to-end data flywheel on agent evalset curation, and automate and productionize the flywheel with agentic data pipelines. - Develop a dedicated platform for "Self-Service Judges," allowing researchers to rapidly prototype and test new LLM-based classifiers and annotations against historical trajectory data. - Establish a formal ecosystem for metric lifecycle management, including a sandbox for validating new metrics against a library of high-confidence calibration experiments. - Productionize automated insights tools to identify the top movers of metric deltas across complex slices such as task type and user cohort.

Minimum qualifications:

Preferred qualifications: