AI Engineer

Role Overview

Key Responsibilities

• Build and maintain LLM harnesses covering agent loops, tool or function calling, context construction, memory, retries, and failure handling.
• Design, iterate, version, and evaluate prompts using structured experiments and evaluation datasets.
• Build and operate RAG pipelines for document extraction, chunking, indexing, retrieval, reranking, prompt assembly, and response handling.
• Generate embeddings and manage vector indexes such as OpenSearch, Pinecone, or pgvector.
• Build parsing and extraction pipelines for documents, structured records, and customer datasets.
• Write accuracy validation scripts, maintain evaluation datasets, and report quality metrics.
• Apply classification, clustering, and lightweight fine-tuning where appropriate.
• Perform sample inspection, error analysis, and data analysis to guide AI design decisions.

Required Skills and Experience

• Bachelor's or Master's degree in Computer Science, Data Science, Statistics, or a related technical field.
• 3+ years of AI or data science experience with hands-on LLM-based application development.
• Strong Python skills, including pandas, NumPy, and scikit-learn.
• Hands-on LLM harness experience with agent loops, tool or function calling, and structured outputs.
• Strong prompt engineering practice with prompt versioning and dataset-based evaluation.
• Experience with at least one orchestration framework and at least one vector database.
• Working knowledge of embeddings, similarity search, classical machine learning, and retrieval evaluation.
• Experience writing clean, tested production code with Git, code review, and basic CI/CD.

Preferred Skills and Experience

• Experience with agentic frameworks such as LangGraph, Claude Agent SDK, or OpenAI Agents, or with custom production agent harnesses.
• Experience building structured evaluation harnesses with golden sets, regression suites, or LLM-as-judge patterns.
• Hands-on AWS experience with Bedrock, SageMaker, or OpenSearch.
• Experience with model fine-tuning or distillation.
• Guidewire or insurance domain knowledge is a plus.
• Experience with document-heavy, regulated, insurance, or finance datasets.

Success Metrics

About Guidewire