Software Engineer - ML Infrastructure

Own Training & Inference Infrastructure: Design and maintain multi-tenant scheduling systems that automatically place training and inference jobs based on hardware topology, cost, and priority, while enforcing fair resource sharing and preemption policies.

Scale Distributed Training: Partner with researchers to scale JAX and PyTorch-based training loops across heterogeneous GPU/TPU clusters with minimal friction, ensuring rock-solid checkpointing and metrics collection.

Optimize Performance & Hardware Bounds: Profile and improve memory usage, device utilization, throughput, and distributed synchronization, specifically navigating edge hardware bottlenecks like on-chip video decoders and memory bandwidth.

Enable Rapid Iteration: Build clean abstractions for launching, monitoring, debugging, and reproducing experiments so researchers can submit massive jobs without needing to manage underlying cluster state.

Contribute to Core Training Code: Evolve our core JAX model code and training pipelines to natively support new architectures, multimodal video/telemetry data streams, and robust evaluation metrics.

Manage Compute Resources: Ensure highly efficient allocation and utilization of massive cloud-based compute clusters while aggressively monitoring and controlling resource costs.