Service Desk Artificial Intelligence Quality Analyst

The Service Desk Artificial Intelligence Quality Analyst is responsible for ensuring that AI-assisted Service Desk capabilities deliver effective, reliable, and measurable support outcomes. The role combines traditional IT Service Desk operations with AI quality monitoring, data analysis, user experience assessment, and continual service improvement.

Role AI Quality & Performance Management

Monitor and report on key Artificial Intelligence (AI) performance indicators, including deflection, resolution, escalation, fallback, and containment rates.

Review AI interactions and identify quality issues, failed intents, and inappropriate or incomplete responses.

Categorize and analyze AI failure scenarios to identify root causes and recurring patterns.

Identify opportunities to improve AI intents, conversational flows, knowledge sources, and automation.

Validate the effectiveness of AI improvements and monitor performance following changes.

Maintain accurate records of AI quality issues, trends, and improvement opportunities. Data Analysis & Reporting

Analyze Service Desk ticket data, AI interaction data, and user feedback to identify trends and service gaps.

Develop regular AI quality and Service Desk performance reports.

Create and maintain dashboards and analytical views using Excel, Power BI, or equivalent tools.

Translate data into clear insights, recommendations, and actionable improvement initiatives.

Track key performance indicators and identify areas requiring further investigation.

Knowledge & Automation Improvement

Identify knowledge gaps that contribute to AI failures or poor user outcomes.

Recommend improvements to knowledge articles, categorization, intents, and automation coverage.

Provide structured feedback to Knowledge Management, Automation, AI, and Service Desk teams.

Support testing and validation of new or enhanced AI and automation capabilities.

Help ensure knowledge and automation solutions reflect actual user needs and Service Desk demand patterns. Service Desk Operations

Handle incidents, service requests, and user support inquiries as required.

Participate in Service Desk queue management and operational activities.

Apply standard ITSM processes when handling incidents and requests.

Escalate issues appropriately and maintain accurate ticket information.

Use operational experience to identify recurring issues and opportunities for AI or process improvement. Continual Service Improvement

Identify recurring Service Desk and AI-related issues and recommend improvement actions.

Participate in service improvement initiatives and working groups.

Monitor the outcomes of improvement initiatives and provide evidence of their effectiveness.

Promote a culture of data-driven, user-focused continual improvement.

Share insights and trends with relevant stakeholders to support informed decision-making.

Requirements Education & Experience

At least 4-6 years’ experience working in an IT Service Desk, IT operations, or IT Service Management (ITSM) environment.

Practical understanding of incident and service request management.

Experience with ServiceNow or a comparable ITSM platform is preferred.

Experience analyzing operational data, ticket trends, or service performance is preferred.

Experience working with AI-assisted support, chatbots, automation, or conversational AI is advantageous. Technical & Analytical Skills

Strong analytical and problem-solving capabilities.

Good working knowledge of Microsoft Excel, including data analysis and reporting.

Experience with Power BI or similar business intelligence and visualization tools.

Ability to interpret data, identify patterns, and translate findings into actionable recommendations.

Basic understanding of conversational AI concepts, including intents, fallback scenarios, escalation, automation, and knowledge-driven responses.

Understanding of ITSM principles and Service Desk workflows. Service & Business Skills

Strong customer and user service orientation.

Excellent attention to detail and ability to identify recurring patterns and anomalies.

Ability to understand both technical and business perspectives.

Strong written and verbal communication skills.

Ability to present analytical findings clearly to both technical and non-technical stakeholders.

Strong collaboration skills and ability to work across Service Desk, AI, Automation, Knowledge Management, and other technology teams.

Ability to manage multiple priorities and work effectively in an operational environment.