naukri
Software Engineering Evaluation Specialist
Mindrift
KSA
Senior
Onsite
Discovered 4 weeks ago
PythonpytestDockerLinuxAI Coding AgentsEnglish
Free
Job Fit Check
Base Career helps you apply smarter for this job.
?%
Ready to ScanKey skills for this role
PythonpytestDocker
Role Overview
You will design coding tasks that challenge frontier AI coding agents. Each task is a self-contained Docker environment with a broken piece of software; an AI agent attempts the fix; automated tests verify the outcome. Your deliverable is the full task package: broken code, tests, instructions, and
Key Skills for This Role
PythonpytestDockerLinuxAI Coding AgentsEnglish
Full Job Posting
About the Role
You will design coding tasks that challenge frontier AI coding agents. Each task is a self-contained Docker environment with a broken piece of software; an AI agent attempts the fix; automated tests verify the outcome. Your deliverable is the full task package: broken code, tests, instructions, and
Responsibilities
- Invent a realistic developer scenario a real bug, a broken ETL, a missing feature not a toy problem.
- Build a reproducible Docker environment with pinned dependencies.
- Write a pytest that verifies outcomes, not specific commands deterministic, non-flaky, and does not leak the fix.
- Write an instruction.md that reads like a Jira ticket a developer would receive.
- Write a reference solve.sh proving the task is solvable.
- Calibrate difficulty so current state-of-the-art agents solve the task 20 60% of the time.
- Iterate based on feedback from expert QA reviewers.
- Later: review other authors tasks as a QA reviewer.
Requirements
- 3+ years of production software development in one backend stack Python, Go, Node.js, Java, or Rust.
- Depth in one stack beats breadth.
- Python + pytest fluency required regardless of primary stack. The task harness is pytest-based even when the broken app is in another language. Fixtures, parametrize, monkeypatch, timeouts, conftest.py.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Mindrift
Freelance Graphic Designer
, QAT
EntryFreelance
Mindrift is seeking a freelance Graphic Designer for the Tendem project to create polished layouts, infographics, social media templates, and marketing assets. The work is project-based and remote, with tasks available d
Discovered 1 weeks agoView →
Freelance Brand Designer
, QAT
EntryFreelance
Mindrift is seeking a Freelance Brand Designer for the Tendem project to build visual identities, brand assets, and scalable design systems for real-world brands and products. The role requires a strong branding portfoli
Discovered 1 weeks agoView →
Senior Python Data Scraping Engineer (Freelance)
, QAT
SeniorFreelance
Mindrift is seeking a Senior Python Data Scraping Engineer for the Tendem project to build and operate reliable web extraction workflows. The role requires 5+ years of relevant experience, strong Python scraping and data
Discovered 1 weeks agoView →
Senior Python Data Scraping Engineer (Freelance)
, UAE
SeniorFreelance
Mindrift is seeking a Senior Python Data Scraping Engineer for the Tendem project to build and operate reliable web extraction workflows. The role requires 5+ years of relevant experience, strong Python scraping and data
Discovered 1 weeks agoView →
Freelance Web Designer
, GBR
Mid-SeniorFreelance
Mindrift is seeking a freelance Web Designer for the Tendem project to create polished, conversion-focused marketing pages, landing experiences, and web layouts. Candidates need strong Figma and web/UI design skills, a r
Discovered 2 weeks agoView →
Freelance Graphic Designer
, GBR
Freelance
Mindrift is seeking a freelance Graphic Designer to create polished layouts, infographics, social media templates, one-pagers, and other marketing and communication assets. The role requires strong typography, compositio
Discovered 2 weeks agoView →
Freelance Brand Designer
, IND
Freelance
Mindrift is seeking a freelance Brand Designer to build visual identities, brand assets, and scalable design systems for real-world brands and products. The role requires a strong branding portfolio, proficiency with Ado
Discovered 2 weeks agoView →
Freelance Web Designer
, AUS
Freelance
Mindrift is seeking a freelance Web Designer for the Tendem project to create polished, implementation-ready marketing pages, landing experiences, and web layouts. The role requires strong Figma, responsive UI design, de
Discovered 2 weeks agoView →
Freelance Graphic Designer
, QAT
Discovered 1 weeks agoFreelance
Freelance Brand Designer
, QAT
Discovered 1 weeks agoFreelance
Senior Python Data Scraping Engineer (Freelance)
, QAT
Discovered 1 weeks agoFreelance
Senior Python Data Scraping Engineer (Freelance)
, UAE
Discovered 1 weeks agoFreelance
Freelance Web Designer
, GBR
Discovered 2 weeks agoFreelance
Freelance Graphic Designer
, GBR
Discovered 2 weeks agoFreelance
Freelance Brand Designer
, IND
Discovered 2 weeks agoFreelance
Freelance Web Designer
, AUS
Discovered 2 weeks agoFreelance