QA / Evaluation Lead

• Design and own the inter-annotator agreement (IAA) methodology for the Phase 1 demonstration corpus — metric selection (Cohen's kappa, Fleiss, Krippendorff's alpha), sampling design, adjudication workflow, and agreement thresholds
• Define evaluation framework architecture: test and evaluation plans, IAA targets, drift detection gates, and model performance metrics per SOW Section 2.9
• Configure and operate sampling-based quality control across the self-service and white-glove annotation paths during Phase D corpus production
• Design and implement confidence-threshold escalation routing from automated annotation to senior-annotator adjudication
• Validate quality scoring and IAA computation within the Innodata data layer
• Support AI Solutions Engineer on evaluation design for SAM 2 and Frontier model API validation — define what 'good enough' looks like quantitatively
• Produce evaluation framework documentation for the Phase 1 NPP closeout package, including per-DataCard documentation with the SA

• Bachelor's degree in Statistics, Data Science, Computer Science, or related quantitative field required; Master's degree preferred. Equivalent experience may substitute for degree on a 2-for-1 basis.
• 6+ years total professional experience, 4+ years in data quality, evaluation methodology, or QA on AI/ML programs
• IAA methodology expertise — Cohen's kappa, Fleiss' kappa, Krippendorff's alpha: hands-on, not theoretical
• Evaluation framework design for AI/ML training data programs
• QC process design: sampling methodology, escalation workflows, adjudication protocols
• Python for QC tooling, metric computation, and statistical analysis
• Active Secret clearance with TS/SCI eligibility

• Prior DoD or IC data quality program experience
• CVAT or equivalent annotation platform QC workflow configuration
• Drift detection and model monitoring methodology
• Experience with FMV / video annotation quality standards

The expected hourly salary range for this position is $45 to $50 p/hour, based on experience, skills, and qualifications.

Note to Candidates:

This role is not a project manager with QC responsibilities — it is a methodology expert who owns the intellectual framework behind data quality on a federal AI program.

The right candidate can walk into a meeting with Government evaluators and explain exactly why the evaluation design produces trustworthy labels.

That conversation is part of Phase 2 positioning.

Please be aware of recruitment scams involving individuals or organizations falsely claiming to represent employers.

Innodata will never ask for payment, banking details, or sensitive personal information during the application process.

To learn more on how to recognize job scams, please visit the Federal Trade Commission’s guide at https://consumer.ftc.gov/articles/job-scams.

If you believe you’ve been targeted by a recruitment scam, please report it to Innodata at verifyjoboffer@innodata.com and consider reporting it to the FTC at ReportFraud.ftc.gov .