Data Scientist

Red-team the system: prompt injection, jailbreaks, tool-misuse, and edge-case discovery.

Design chaos and stress tests that probe model and agent reliability under degraded or hostile conditions.

Characterize failure modes and feed them back into guardrails and regression coverage.

Evaluate retrieval quality over the document corpus — recall@k, MRR/nDCG, context precision and recall — and run experiments on chunking, indexing, and hybrid retrieval strategies.

Analyze multi-step agent trajectories: tool-call correctness, trajectory efficiency, replayable-state inspection, and guardrail-breach behavior.

Assess intent classification and routing quality as measurable components, not black boxes.

Build standing evaluation that catches quality and behavioral regressions when a model in the suite is swapped, upgraded, or re-quantized, or when prompts and pipelines change.

Monitor output-distribution and quality drift in production; distinguish genuine regressions from noise on stochastic outputs.

Recommend and validate fixes through the levers available with local models — prompt changes, retrieval and grounding adjustments, routing changes, or model selection.

Build, ship, and own production models that forecast and surface trends from operational telemetry — capacity and resource forecasting, anomaly prediction, and early-warning signals on metrics and logs.

Take these from prototype to production and keep them healthy: deployment, monitoring, recalibration, and retraining as data and behavior shift.

Define accuracy and lead-time metrics that matter operationally — precision/recall on predicted incidents, forecast error, how far ahead a signal fires — not just offline scores.

Wire predictive signals into the LLM and agentic layer so forecasts and trends feed reasoning, advisories, and operator-facing recommendations.

Apply AIOps/NOC analysis where it's the product: log anomaly detection, event correlation, and root-cause and problem analysis.

Quantify the economics of the system — cost and token consumption per interaction, interaction-type taxonomies — and connect them to customer-facing value metrics like MTTR and operator-hours.

Communicate findings to engineering and product stakeholders through clear, in-context analysis.

Use LLM-assisted workflows to scale the work itself — drafting analyses, generating synthetic evaluation cases, and bootstrapping labeled data for human refinement.

Track and adopt state-of-the-art evaluation, retrieval, and agentic-analysis techniques; bring the useful ones into the team's workflow.

Bachelor's or Master's in Data Science, Computer Science, Statistics, Mathematics, or a related field or equivalent experience.

3+ years in data science, ML, or applied quantitative analysis.

Strong applied statistics, with the judgment to design sound experiments and significance tests on noisy, non-deterministic outputs (not just clean A/B conversion).

Experience building, deploying, and monitoring predictive or time-series models in production: forecasting, anomaly detection, or trend analysis, including recalibration as data shifts.

Demonstrated work evaluating, analyzing, or improving LLM or NLP systems: eval design, quality measurement, retrieval evaluation, or agent analysis.

Proficiency in Python.

Strong SQL and comfort querying large analytical datasets.

Fluency with foundation models and hands-on experience with the modern LLM evaluation and tooling layer — eval/harness frameworks, judge pipelines, and the libraries used to serve, prompt, and test models.

Ability to build analysis and visualization in code.

ScienceLogic is a leader in IT Operations Management, providing modern IT operations with actionable insights to resolve and predict problems faster in a digital, ephemeral world. Its solution sees everything across cloud and distributed architectures, contextualizes data through relationship mapping, and acts on this insight through integration and automation.

www.sciencelogic.com