Infrastructure Engineer

LGTM Stack Architecture & Operations: Deploy and manage production-scale Loki, Grafana, Tempo, and Mimir alongside Prometheus and Thanos/VictoriaMetrics; design retention strategies, aggregation rules, and query patterns that scale to thousands of GPUs

Custom Exporter Development: Write Prometheus exporters in Go or Python for GPU metrics, platform services, and infrastructure components; implement proper metric naming, labeling strategies, and follow OpenMetrics standards

Kubernetes Controller Development: Build custom controllers and operators for GPU workload management, scheduling, and resource allocation; instrument controllers with comprehensive metrics and tracing for observability

Training & Inference Monitoring: Design and implement specialized observability for AI training workloads (GPU efficiency, distributed training performance, resource utilization) and inference services (latency percentiles, throughput, cost analytics)

SLURM Integration & Monitoring: Deploy and manage SLURM clusters for HPC workloads, build observability for batch jobs, create bridges between SLURM and Kubernetes, and design unified monitoring across orchestrators

Systemd Service Development: Write and deploy systemd services for bare-metal GPU nodes including monitoring agents, metric collectors, and platform daemons; implement proper logging and error handling

GitOps Platform Engineering: Manage infrastructure using ArgoCD and GitOps workflows, create observable platform abstractions that teams consume declaratively, build self-service capabilities with integrated monitoring

Alerting & SRE: Design intelligent, actionable alerting systems for GPU failures, thermal throttling, performance degradation, and workload anomalies; define platform SLOs and implement comprehensive monitoring to track reliability

Multi-Tenant Monitoring Isolation: Build secure observability isolation ensuring customers access only their metrics and logs while maintaining platform-wide visibility for operations; implement query-time filtering and RBAC

Metrics Pipeline Engineering: Implement automated collection of GPU and platform telemetry; integrate with OpenTelemetry for unified observability; manage cardinality and storage costs through intelligent aggregation

Cost & Utilization Analytics: Build monitoring pipelines that track GPU utilization, idle time, efficiency metrics, and cost allocation per tenant; create dashboards for platform economics and provider payout calculations

Dynamic Workload Management: Design systems that use observability data to detect hardware failures, trigger workload migration, and handle graceful degradation; ensure monitoring survives infrastructure changes

Self-Service Observability Platforms: Build intuitive dashboards, APIs, and alerting interfaces enabling providers to monitor hardware contributions and customers to track workload performance in real-time

Incident Response & Debugging: Use observability systems to rapidly identify root causes during production issues - GPU hardware failures, network bottlenecks, or workload problems; build tooling that reduces MTTD and MTTR

Performance Optimization: Leverage observability data to identify bottlenecks in distributed training, inference latency issues, networking inefficiencies, and opportunities for GPU utilization improvement