LLM Evaluation Engineer

Design and build real-time evaluation logic that determines whether LLM prompts or outputs violate enterprise policies.

Implement evaluation strategies using a mix of semantic similarity, foundation model scoring, rule-based systems, and statistical checks.

Integrate model outputs with downstream enforcement actions (e.g. redaction, escalation, blocking).

Prototype, tune, and productize small language models and prompt templates for classification, labeling, or scoring.

Collaborate with data infrastructure engineers to connect evaluation logic with ingestion and storage layers.

Build tools to observe, debug, and improve evaluator performance across real-world data distributions.

Define abstractions for reusable evaluation components that can scale across use cases.

7+ years of experience in ML systems or AI engineering roles, with at least 1–2 years working directly with LLMs, NLP pipelines, or semantic search.

Deep understanding of foundation models (e.g. OpenAI, Claude, Mistral, Llama) and how to work with them via APIs or open source.

Hands-on experience with vector search (e.g. FAISS, Qdrant, Weaviate) and embeddings pipelines.

Proven ability to implement real-time or near-real-time evaluation logic using semantic similarity, classifier scoring, or structured rules.

Strong in Python, with familiarity using libraries like Hugging Face Transformers, LangChain, and PyTorch or TensorFlow.

Ability to reason about model behavior, test prompt configurations, and debug complex decision logic in production.

Experience with OpenTelemetry, Model Context Protocol (MCP), or structured tracing of multi-agent or multi-model pipelines.

Experience with red-teaming, AI risk taxonomies, or safety audits for LLM-based systems.

Based in or willing to spend time in the San Francisco Bay Area for in-person collaboration.

Our team is small and focused, valuing autonomy and real impact over titles and management. We need strong technical skills, a proactive mindset, and clear written communication, as much of our work is asynchronous. If you're organized, take initiative, and want to work closely with customers to shape our products, you'll fit in well here.

Finally, we pay market cash compensation and generally above-market equity. The compensation package for this role is benchmarked using Carta Total Compensation and reflects real-time market data for our company’s size, this role’s level, and your geographic location. We have well-designed and generous benefits.

https://www.thirdlaw.io/