Senior Python Engineer - AI Coding Agent Evaluation
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
Develop realistic developer environments, design tasks for AI evaluation, write tests, and iterate based on feedback using Python, JavaScript, and Docker.
Key Skills for This Role
Responsibilities
- Build realistic developer environments with codebase, infrastructure, and context
- Design tasks from intermediate states of these environments
- Write tests that verify AI agent solutions
- Iterate on tasks and tests based on QA feedback
Requirements
- 8+ years in software development
- Proficiency in Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
- Experience writing tests (functional, integration)
- English proficiency B2+
Full Job Posting
Overview
- We're building a dataset to evaluate AI coding agents how well a model handles real world developer tasks.
- You'll create challenging tasks and evaluation criteria within realistic simulated environments.
Responsibilities
- Build realistic developer environments a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history.
- Design tasks from intermediate states of these environments craft the prompt, define what 'solved' means, and ensure the task is solvable by an AI agent.
- Write tests that verify agent solutions accept all valid approaches and reject incorrect ones, neither too strict nor too lenient.
- Iterate on tasks and tests based on QA feedback review agent solutions, analyze failures, and refine until the evaluation is fair and robust.
What we look for
- 8+ years in software development.
- Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis.
- Experience writing tests (functional, integration).
- English proficiency B2+.
What this is NOT
- Not data labeling.
- Not prompt engineering.
- Not writing code from scratch the agent writes most of the code; you guide and evaluate.
Why this is hard
- Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non trivial.
- Tasks have many valid solutions writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Mindrift
Ex-MBB Strategy Consultant - AI Training (Remote)
Abu Dhabi, UAE
Mindrift is seeking ex-MBB strategy consultants to build realistic consulting project environments for AI training. You will design structured tasks, evaluation criteria, and grading frameworks to teach AI agents high-le
Ex-MBB Strategy Consultant - AI Training (Remote)
Doha, QAT
Mindrift is seeking ex-MBB strategy consultants to build realistic consulting project environments and design structured tasks for AI training. This remote, project-based role involves creating detailed project scenarios
Freelance Data Scraping Engineer (Python)
Saudi Arabia, KSA
Mindrift is looking for a Freelance Data Scraping Engineer to drive specialized data scraping workflows within a hybrid AI + human system. You will own end-to-end data extraction, use tools like Apify and OpenRouter, and
Senior Python Data Scraping Engineer (Freelance)
Saudi Arabia, KSA
Mindrift is seeking a Senior Python Data Scraping Engineer for a freelance role on the Tendem project. You will collaborate with AI agents, providing critical thinking and domain expertise for web scraping and data extra
Freelance Presentation Designer
Doha, QAT
Mindrift is looking for a Presentation Designer to create high-stakes pitch decks and professional one-pagers for the Tendem project. Requires 2+ years of experience in graphic or presentation design, expertise in PowerP
Freelance Presentation Designer
Abu Dhabi, UAE
Please submit your resume in English and indicate your level of English. Mindrift is looking for a highly skilled Presentation Designer to join the Tendem project (\https://tendem.ai/\) and drive specialized storytelling
Senior Python Engineer - AI Coding Agent Evaluation (Freelance)
Abu Dhabi, UAE
Mindrift is seeking a Senior Python Engineer to create challenging tasks and evaluation criteria for AI coding agents. You will build realistic developer environments, design tasks, and write tests to evaluate AI agent s
Senior Python Engineer - AI Coding Agent Evaluation (Freelance)
Doha, QAT
Mindrift is building a dataset to evaluate AI coding agents. You will create challenging tasks and evaluation criteria within simulated developer environments. Requires 8+ years in software development with Python, JavaS
Ex-MBB Strategy Consultant - AI Training (Remote)
Abu Dhabi, UAE
Ex-MBB Strategy Consultant - AI Training (Remote)
Doha, QAT
Freelance Data Scraping Engineer (Python)
Saudi Arabia, KSA
Senior Python Data Scraping Engineer (Freelance)
Saudi Arabia, KSA
Freelance Presentation Designer
Doha, QAT
Freelance Presentation Designer
Abu Dhabi, UAE
Senior Python Engineer - AI Coding Agent Evaluation (Freelance)
Abu Dhabi, UAE
Senior Python Engineer - AI Coding Agent Evaluation (Freelance)
Doha, QAT