AI Evals Engineer — Evaluation Datasets & Ground Truth

Production sampling: pull stratified, de-identified samples from real traffic so eval sets reflect what the system actually sees, including the long tail.

Human labeling: scope and run labeling programs, write annotation guidelines, build calibration sets, measure inter-annotator agreement, and manage vendors (including offshore labeling teams) or internal subject-matter experts. You own label quality, not just label throughput.

Synthetic / oracle-generated ground truth: where a task is tractable for a frontier model given enough compute (long context, multi-pass, tool use, self-consistency) but too expensive to run that way in production, design the oracle harness that produces labels for the cheap production path to be measured against. Then verify the oracle : calibrate its output against a human-labeled sample before anyone trusts it.

Programmatic and adversarial construction: heuristic labels, templated edge cases, backtests built from past incidents (“what test would have caught this?”).

Version every dataset; track lineage, splits, and which model/prompt versions have seen which examples.

Guard against contamination and leakage (eval examples drifting into few-shot prompts, the oracle model also being the production model, etc.).

Slice by customer segment, input type, and difficulty so a headline number can’t hide a regression.

Refresh sets as the product and traffic change; retire stale examples.

Calibrate automated graders (LLM-as-judge, similarity metrics, exact-match) against your human gold sets, and be the person who says when an automated judge is good enough to gate on.

Report metrics correctly: precision/recall/F1, confusion matrices, calibration, confidence intervals, sample sizes needed to detect a given effect.

Partner with engineers who own the eval harness and CI so your datasets are actually run, and with ML engineers on classifier features when the data tells you the features are the problem.

Have run a human-labeling program end to end, including vendor selection, guideline authoring, QA sampling, and cost/quality trade-offs.

Experience with eval tooling.

Experience with data labeling platforms.

Have used a frontier model as a distillation/oracle source and can articulate where that assumption breaks.

Verify the verifier. A label is a claim, not a fact, until something independent agrees with it.

Start simple: binary before graded, one judge before five, a hundred well-understood examples before ten thousand noisy ones.

You measure business outcomes, not model vibes, and you’re comfortable telling a senior engineer their favorite change didn’t move the number.

You’d rather own an unglamorous problem completely than a glamorous one partially.

You’ll report to the Chief AI Officer and work across all product/pipeline teams. Over time it is expected that you’ll lead a small team of eval engineers. You’ll have a generous budget for labeling vendors and oracle compute.

The team. Wide expertise, sharp, low ego, and genuinely fun to work with. We have each other's backs and we celebrate wins together.

Real ownership and impact from day one. Our customers make million-dollar decisions on our platform. Your work matters immediately.

AI-native product and AI-native workflows. Every team — engineering, sales, operations — runs on AI tooling daily.

Founder-led transparency. Leadership shares the financials, the strategy, and the hard calls with the whole team.

High growth, high demand. The product works, customers love it, and we are hiring to keep up. No shortage of opportunity here.

Flexibility. Remote, hybrid, or in-office with a great Portland, OR headquarters.

Self-starters who thrive in ambiguity. We're a startup. If you need someone to tell you what to do every morning, this isn't the place; if you come alive solving hard problems with smart people, it is.

These four themes are how we think, build, and lead:

Truth — Say the hard thing. We speak plainly, flag problems early, and share what's actually going on.

Precision — Sign your name to it. Our customers make million-dollar decisions on our data. We get the details right.

Velocity — Speed compounds. Waiting doesn't. We ship daily and launch features every two to three weeks.

Ownership — Own it, then solve it. We own outcomes, not tasks. If something's broken, we fix it.

EEO:

Prophetic Software is an equal opportunity employer. We are committed to creating an inclusive environment for all employees and applicants, and we make employment decisions without regard to race, color, religion, sex (including pregnancy, childbirth, or related medical conditions), gender identity or expression, sexual orientation, national origin, age, disability, genetic information, veteran or military status, marital status, or any other characteristic protected under applicable federal, state, or local law. We actively seek to build a diverse team and encourage candidates from underrepresented backgrounds to apply. Reasonable accommodations are available upon request for qualified individuals with disabilities throughout the application and employment process.