AI/ML Evaluator

Project Details

Responsibilities

• Review and evaluate AI system behaviour, outputs, and relevant technical data according to defined project guidelines.
• Analyse system telemetry, event data, signals, and other technical information to identify meaningful patterns.
• Evaluate patterns and signals that may help distinguish legitimate user activity from automated or bot activity.
• Apply AI/ML knowledge and expert human judgment when reviewing complex or ambiguous cases.
• Identify unusual patterns, inconsistencies, or behaviours that may require further review.
• Evaluate complex cases using available evidence, context, and defined project requirements.
• Classify, label, or annotate assigned data accurately and consistently.
• Provide high-quality human evaluations to support AI system training, evaluation, and improvement.
• Document decisions and supporting reasoning clearly when required.
• Identify unclear or unusual cases and flag them according to defined project processes.
• Apply detailed project guidelines consistently across assigned tasks.
• Maintain high levels of quality, accuracy, consistency, and attention to detail.

Required Qualifications

• Educational background or equivalent experience in Artificial Intelligence, Machine Learning, Computer Science, Data Science, Statistics, Engineering, or a related field.
• Experience working with AI/ML systems, machine learning models, AI evaluation, data analysis, or technical system analysis.
• Strong understanding of core AI and machine learning concepts.
• Ability to analyse complex technical data, system behaviour, and patterns.
• Experience evaluating AI systems, models, outputs, or data is preferred.
• Familiarity with system telemetry, event data, logs, or other technical signals is preferred.
• Strong analytical and problem-solving skills.
• Ability to identify meaningful patterns and make informed decisions based on complex or incomplete information.
• Strong attention to detail and the ability to apply detailed guidelines consistently.
• Experience with data annotation, AI evaluation, human feedback, model evaluation, or structured data review is preferred.
• Familiarity with bot detection, automated abuse detection, anomaly detection, fraud detection, or similar areas is preferred.
• Strong written English comprehension and documentation skills.
• Part Time
• Associate
• Remote