gem
AI Behavior Researcher - Agent Alignment
Transluce
San Francisco, USA
Full-time
Entry
Onsite
USD 250000-450000 yearly / year
Discovered 2 weeks ago
PythonLLM-as-a-judgeAI coding agents
Free
Job Fit Check
Base Career helps you apply smarter for this job.
?%
Ready to ScanKey skills for this role
PythonLLM-as-a-judgeAI coding agents
Key Skills for This Role
PythonLLM-as-a-judgeAI coding agents
Full Job Posting
- Develop novel, valid automated evaluations of AI agents' honesty and alignment.
- Write code to implement and run automated evaluations, such as environment simulators or LLM-as-a-judge pipelines.
- Design methods to improve the ecological of automated evaluations, and especially to measure which effects are increasing or decreasing as models become more capable.
- Collaborate with our governance team to deliver high-impact evaluations for public policy.
- Collaborate with scientists and research engineers to productionize best practices in AI behavior evaluation.
- Expertise on quantitative generative AI evaluation and measurement. Good intuition about how to systematize and operationalize complex concepts and to work backwards from possible failure modes of agents.
- Relevant experience designing and validating automated AI evaluation methods, such as LLM-as-a-judge systems or multi-turn benchmarks.
- Proficiency in Python to implement analysis and evaluation tooling.
- Meticulous, good experimental design, epistemic self-awareness and transparency.
- Ability to iterate quickly and balance between scrappiness and thoroughness based on the impact needs of a project.
- Strong communication skills, low ego, openness to giving and receiving feedback.
- Experience running automated evaluations at scale or in a production context.
- Experience conducting controlled human subjects experiments to validate automated evaluation methods.
- Experience in customer-facing, consulting, or forward-deployed roles translating ambiguous stakeholder needs into concrete deliverables.
- Experience and comfort using AI coding agents at work.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Transluce
AI Behavior Researcher - Child Safety and Mental Health
San Francisco, USA
MidFull-time
Discovered 1 weeks agoView →
ML Research Engineer
San Francisco, USA
Discovered 1 weeks agoFull-time
Full-Stack Product Engineer
San Francisco, USA
Discovered 1 weeks agoFull-time
AI Systems Engineer
San Francisco, USA
Discovered 1 weeks agoFull-time
Governance & Policy Fellow
San Francisco, USA
Discovered 1 weeks agoFull-time
AI Behavior Researcher - Child Safety and Mental Health
San Francisco, USA
Discovered 1 weeks agoFull-time
People Operations Manager
San Francisco, USA
Discovered 1 weeks agoFull-time
Business Operations Manager
San Francisco, USA
Discovered 1 weeks agoFull-time
VP of Engineering
San Francisco, USA
Discovered 2 weeks agoFull-time