AI Quality Automation Engineer – Assistant Vice President

Key Responsibilities & Competencies

• Quality Framework Architecture: Design, build, and maintain scalable automated testing frameworks specifically tailored for AI/ML systems, including LLMs, RAG pipelines, and traditional ML models.
• Model & Output Validation: Design and execute automated evaluation pipelines to measure critical metrics such as hallucination rates, context retention, retrieval precision, semantic similarity, and answer relevance (using tools like RAGAs, DsPy).
• Data Pipeline & ETL Auditing: Validate the quality, integrity, and schema compliance of data pipelines and ETL processes (PySpark, Hive, Kafka, Parquet, Iceberg) to ensure clean data flows into AI models.
• CI/CD Integration: Integrate automated AI quality gates into enterprise CI/CD pipelines (using Tekton, Harness, UDeploy, Jenkins) to enable continuous testing and prevent regressions.
• Risk Stewardship & Compliance: Lead adversarial testing (Red Teaming) to identify vulnerabilities like prompt injection, jailbreaking, and data leakage. Validate real-time safety guardrails to ensure compliance with ethical AI guidelines, corporate policies, and regulatory standards.
• Operational Excellence & Observability: Implement monitoring and observability solutions (using MLFlow, Dagster, Vertex AI, SageMaker) to detect model drift, performance degradation, and latency bottlenecks in production.
• Stakeholder Engagement: Demonstrate exceptional communication and diplomacy skills, effectively engaging with cross-functional stakeholders to articulate quality risks, negotiate solutions, and align on quality benchmarks.

Required Skills & Technical Qualifications

• Experience: 5-8 years of professional experience implementing data-intensive solutions or test automation frameworks using agile methodologies, with at least 2-3 years of direct experience validating AI/ML, Generative AI, or data-intensive systems.
• Programming Languages: Primary: Python, Shell Scripting, Jupyter Notebook Scripting Secondary: Java
• Primary: Python, Shell Scripting, Jupyter Notebook Scripting
• Secondary: Java
• Machine Learning & AI Evaluation Frameworks: Core ML: Scikit-learn, Keras, TensorFlow, PyTorch, spaCy, NLTK Advanced NLP & GenAI Evaluation: RAGAs, DsPy, Hugging Face Transformers, RASA, Large Language Models (LLM)
• Core ML: Scikit-learn, Keras, TensorFlow, PyTorch, spaCy, NLTK
• Advanced NLP & GenAI Evaluation: RAGAs, DsPy, Hugging Face Transformers, RASA, Large Language Models (LLM)
• MLOps & ML Lifecycle Management: Platforms: MLFlow, Dagster, Google Cloud Vertex AI, Amazon SageMaker
• Platforms: MLFlow, Dagster, Google Cloud Vertex AI, Amazon SageMaker
• Database Management Systems: Relational: Oracle, MySQL, PostgreSQL NoSQL: MongoDB
• Relational: Oracle, MySQL, PostgreSQL
• NoSQL: MongoDB
• Infrastructure & DevOps: CI/CD & DevOps Tooling: UDeploy, Harness, Tekton, Jenkins, Maven Version Control: Git, Subversion (SVN) Containerization & Orchestration: Docker, OpenShift, Amazon ECS
• CI/CD & DevOps Tooling: UDeploy, Harness, Tekton, Jenkins, Maven
• Version Control: Git, Subversion (SVN)
• Containerization & Orchestration: Docker, OpenShift, Amazon ECS
• Data Engineering & Processing: Messaging & Asynchronous Communication: Kafka, RabbitMQ ETL & Data Warehousing: Sqoop, PySpark, Hive, Hadoop, HDFS File Formats: Avro, Parquet, Iceberg Job Scheduling & Orchestration: Autosys, Cron
• Messaging & Asynchronous Communication: Kafka, RabbitMQ
• ETL & Data Warehousing: Sqoop, PySpark, Hive, Hadoop, HDFS
• File Formats: Avro, Parquet, Iceberg
• Job Scheduling & Orchestration: Autosys, Cron
• Web Technologies: Web Frameworks: FastAPI, Flask Web Servers & Reverse Proxies: NGINX
• Web Frameworks: FastAPI, Flask
• Web Servers & Reverse Proxies: NGINX

Education

Most Relevant Skills

• ------------------------------------------------------

Other Relevant Skills

• Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.
• If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi . View Citi’s EEO Policy Statement and the Know Your Rights poster.