Software Engineering Evaluation Specialist

About the opportunity

Responsibilities

• Invent realistic developer scenarios involving bugs, broken ETL processes, or missing features.
• Build reproducible Docker environments with pinned dependencies.
• Write deterministic pytest tests that verify outcomes without leaking the fix.
• Write instruction.md files that read like Jira tickets.
• Write reference solve.sh scripts proving that tasks are solvable.
• Calibrate difficulty so agents solve tasks 20% to 60% of the time.
• Iterate based on feedback from expert QA reviewers.
• Review other authors' tasks as a QA reviewer later.

Required qualifications

• 3+ years of production software development in one backend stack: Python, Go, Node.js, Java, or Rust.
• Python and pytest fluency, including fixtures, parametrization, monkeypatching, timeouts, and conftest.py.
• Docker authoring experience with reproducible Dockerfiles, pinned dependencies, multi-stage builds when needed, and non-root users.
• Linux and Bash comfort debugging inside containers, including tools such as strace, lsof, and journalctl.
• Experience with Claude Code, Cursor, Roo Code, or a similar AI coding agent on non-trivial work.
• B2+ written English proficiency.

Preferred qualifications

• Domain depth in security, system administration, scientific computing, DevOps, or Git internals.
• Experience with modern Python tooling such as uv, Poetry, or pyproject.toml.
• Experience with coverage tooling such as pytest-cov, coverage.py, gcov, llvm-cov, or kcov.
• Fuzzing or property-based testing experience, including Hypothesis.
• Prior contribution to agent-evaluation benchmarks or related frameworks.

Scope and fit

Process

• The process is apply, pass a 90-minute sample-task screen and short behavioral interview, join a project, complete tasks, and get paid.
• Applicants should submit an English CV, indicate English proficiency, mention Software Engineering Evaluation Specialist—Terminal Bench, and include a GitHub link if available.

Time commitment

• Onboarding takes approximately 10 hours for the first task.
• Steady-state work is approximately 5 hours per task with 2 to 4 parallel tasks per author.
• The realistic weekly load is 8 to 20 hours, with higher volume available for top performers.
• Contributors choose when to work, subject to task deadlines and acceptance criteria.

Compensation

• Paid contributions have rates up to $35 per hour.
• Compensation is task-based and equivalent to an hourly rate depending on performance and volume.
• Some projects include incentive payments.
• Rates vary by expertise, skills assessment, location, project needs, and other factors; payment details are shared per project.