Human Data Quality Engineer (Founding Team)

Work with clients as a strategic thought partner, challenging annotation schemas and data requirements when they won't produce the signal the model needs.

Advise on project design and how the choice of schema can impact data quality.

Build quality upstream across the operational workflow, from recruitment, screening, and training through to writing guidelines and running calibration sessions.

Build scalable quality systems, measurement frameworks and automated checks using Python and SQL.

Partner with product and engineering to build the quality infrastructure that delivers high-quality human data at scale.

Investigate data quality and integrity issues, identifying root causes and turning insights into scalable improvements.

Architect and build dashboards, monitoring and reporting that provide clear visibility into quality and operational performance.

Raise the quality capability across the company, upskilling operations and acting as a thought mentor to junior analysts.

Help define how Prolific approaches quality across new AI domains, shaping best practice as the team grows.

What You’ll Bring to the Role

5+ years of experience in building quality, evaluation or annotation systems within AI, machine learning, LLMs or human data environments.

Strong Python and SQL skills, with a passion for using data to solve complex quality problems.

A solid understanding of machine learning pipelines and how human data impacts model performance.

Strong analytical and statistical thinking, with experience designing scalable quality frameworks.

The confidence and credibility to interact with stakeholders at frontier labs and act as a partner.

The ability to leverage your experience and expertise to influence and guide stakeholders at every level, both client side and internally. in

The ability to turn your own data analysis and quality methodology into requirements that product and engineering can build into systems.

The ability to explain yout data analysis and findings clearly to non-technical stakeholders, so they can act on them.

A proactive, builder's mindset -you enjoy creating new systems, navigating ambiguity and improving how things work. Even Better if you have:

Experience with LLM evaluation, RLHF, AI safety or red teaming.

Experience translating vague model or evaluation goals into clear annotation specifications.

Experience working with human annotation programmes or human data operations.

Familiarity with calibration, inter-rater agreement, drift detection or other evaluation methodologies.

Why Prolific is a great place to work

We've built a unique platform that connects researchers and companies with a global pool of participants, enabling the collection of high-quality, ethically sourced human behavioral data and feedback. This data is the cornerstone of developing more accurate, nuanced, and aligned AI systems.

We believe that the next leap in AI capabilities won't come solely from scaling existing models, but from integrating diverse human perspectives and behaviors into AI development. By providing this crucial human data infrastructure, Prolific is positioning itself at the forefront of the next wave of AI innovation – one that reflects the breath and the best of humanity.

Working for us will place you at the forefront of AI innovation, providing access to our unique human data platform and opportunities for groundbreaking research.

Join us to enjoy a competitive salary, benefits, and remote working within our impactful, mission-driven culture. At Prolific, our compensation packages for eligible roles include base salary, equity, and benefits. Many roles also include the opportunity to earn a cash variable element, such as a bonus or commission. Each job posting shows a salary range that reflects the minimum and maximum target for new hires, based on the role’s location as well as your skills, experience, and relevant education or training. Your recruiter will also be happy to share the specific salary range for your preferred location during the hiring process.