ML Cloud Infrastructure Engineer

Build pipelines that transform raw multi-modal data—including telemetry, imagery, video, sensor, and simulation data—into curated, versioned training datasets.

Develop reproducible training and evaluation workflows that scale across cloud compute and GPU resources.

Build and maintain model deployment infrastructure for packaging, serving, inference, versioning, and rollback.

Implement experiment tracking, dataset lineage, model versioning, and other capabilities required for reproducible ML development.

Own data schema versioning and migration across pipelines, data lakes, and services as datasets and models evolve.

Design, build, and operate scalable AWS infrastructure using Infrastructure as Code.

Build and maintain Kubernetes/EKS workloads and containerized environments for training, batch processing, evaluation, and model serving.

Develop self-service tooling and paved paths for compute scheduling, storage, data access, training, and deployment.

Improve utilization, scalability, and cost efficiency across cloud and accelerator infrastructure.

Build infrastructure that enables engineering teams to launch workloads safely without unnecessary operational overhead.

Build evaluation frameworks and regression testing for model quality, dataset integrity, and pipeline correctness.

Establish quality and reliability signals that help determine whether models are ready for production use.

Develop monitoring, logging, tracing, and observability across training jobs, data pipelines, and deployed models.

Diagnose and resolve performance, scaling, reliability, and infrastructure bottlenecks.

Maintain high standards for automation, testing, documentation, and operational readiness.

Partner with Autonomy, Software, Data, Simulation, and Security teams to build ML infrastructure spanning edge data capture through cloud training and model deployment.

Contribute to CI/CD and release processes for models, datasets, and ML pipelines.

Translate engineering requirements into scalable platform capabilities that can support multiple teams and use cases.

Incorporate feedback from engineers and users to continuously improve ML development workflows.

Implement secure infrastructure practices, including IAM least privilege, secrets management, access controls, and secure handling of sensitive and defense-related data.

Build data and ML workflows with reproducibility, traceability, and appropriate controls from the start.

Partner with security and infrastructure teams to ensure ML systems meet applicable operational and compliance requirements.

3+ years of experience in software engineering, infrastructure engineering, data engineering, ML infrastructure, or a related field.

Strong programming experience in Python, with experience in Go, C++, or another systems-oriented language preferred.

Experience building and operating production services, APIs, data pipelines, developer platforms, or infrastructure.

Hands-on experience with ML workflows such as dataset preparation, model training, evaluation, or deployment.

Experience with cloud infrastructure, preferably AWS, and Infrastructure as Code.

Hands-on experience with Kubernetes and containerized environments.

Strong understanding of production engineering fundamentals, including reliability, observability, testing, automation, and maintainability.

Ability to work effectively across engineering disciplines and solve ambiguous technical problems with a high degree of ownership.

U.S. Citizenship and ability to obtain and maintain a U.S. Government security clearance.

Experience with MLOps and workflow platforms such as MLflow, Weights & Biases, Kubeflow, Ray, Airflow, or Dagster.

Experience with GPU/accelerator scheduling, distributed training, or large-scale ML workloads.

Experience working with multi-modal datasets including imagery, video, telemetry, sensor, or simulation data.

Experience supporting autonomy, robotics, simulation, or real-time systems.

Experience deploying ML models to edge or embedded environments.

Experience with AWS GovCloud, GCP Assured Workloads, or compliance-driven environments such as FedRAMP or IL4/IL5.

Within your first 12 months, you will have:

Built reliable and reproducible pipelines that move data from raw capture through curated, versioned training datasets.

Enabled training and evaluation workloads to run at scale with clear quality and reliability signals before deployment.

Delivered self-service ML infrastructure that reduces friction and accelerates Autonomy, Data, and Software teams.

Improved the observability, reliability, security, and cost efficiency of HavocAI’s ML infrastructure.

Established scalable foundations that allow ML development and deployment to grow alongside HavocAI’s autonomy capabilities.

Innovation: We are driven to break new ground. Every day presents an opportunity to challenge the status quo, think boldly, and deliver advanced solutions that transform the future of defense technology.

Integrity: We hold ourselves to the highest ethical standards, ensuring transparency, accountability, and trust in all our actions and partnerships.

Mission-Driven: We are focused on achieving impactful outcomes that align with our core mission—protecting lives through innovation.

Forward-Leaning: We continuously seek out new opportunities and remain at the forefront of technological advancements. We embrace change and anticipate the challenges of tomorrow with confidence and creativity.

Ownership of All Tasks: At HavocAI, no problem is too complex or too trivial. We believe that greatness comes from tackling the hardest challenges, but also in handling the smallest, sometimes thankless, tasks with the same level of commitment and care.

Servant Leadership: We lead by serving others, whether it’s supporting our employees, partners, or the broader community. Empowering those around us is key to achieving long-term success and making a lasting impact.

HavocAI is an Equal Opportunity Employer and is committed to creating an inclusive and diverse workplace. We welcome applicants from all backgrounds and do not discriminate based on race, color, religion, gender, sexual orientation, age, national origin, disability, veteran status, or any other legally protected status.