LLM Ops Engineer
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
Heidi is looking for an LLM Ops Engineer to design and maintain AWS/EKS infrastructure for GPU-backed model workloads, build evaluation tooling, and optimize cost and performance. You will work closely with product engineers and clinicians.
Key Skills for This Role
Responsibilities
- Design, deploy and maintain AWS/EKS infrastructure running GPU backed model workloads
- Manage GPU node pools, tune autoscaling for inference traffic patterns, and own the full model serving lifecycle
- Write and maintain infrastructure as code in Terraform
- Build tooling that measures model performance: clinical accuracy, latency, cost efficiency
- Design offline evaluation harnesses, automated regression tests, and dashboards
- Own GPU utilization, quantization, request batching, model routing, and spot/on demand node strategy
- Collaborate with product engineers and clinicians on prompt engineering, context window management, and model data pipelines
- Instrument token usage, latency P99s, GPU memory pressure, hallucination rates, and error classes
Requirements
- Strong AWS and Kubernetes experience, with hands on depth in EKS, GPU workload scheduling, and IAM patterns
- Practical LLMOps experience: model serving frameworks (vLLM, TGI or similar), prompt versioning, model registry management, A/B deployment, shadow traffic, rollback strategies
- Comfort with Python and enough ML context to discuss fine tuning, RLHF, RAG architectures, and evaluation methodology
- Infrastructure as code fluency in Terraform
- Experience building evaluation frameworks for generative models
- A bias toward observable, auditable systems
- Strong engineering habits: small PRs, meaningful code review, test coverage
Full Job Posting
Who We Are
- Heidi is building an AI Care Partner that works alongside clinicians.
- Backed by nearly $100 million in funding, we're growing in the US, UK, Canada, and Europe.
What You’ll Do
- Design, deploy and maintain AWS/EKS infrastructure running GPU backed model workloads
- Manage GPU node pools, tune autoscaling for inference traffic patterns, and own the full model serving lifecycle
- Write and maintain infrastructure as code in Terraform
- Build tooling that measures whether models are performing — clinically accurate, latency appropriate, cost efficient
- Design offline evaluation harnesses, automated regression tests, and dashboards
- Own GPU utilization, quantization, request batching, model routing, and spot/on demand node strategy
- Work closely with the Models Team on fine tuning workflows and model selection tradeoffs
- Collaborate with product engineers and clinicians on prompt engineering, context window management, and model data pipelines
- Instrument token usage, latency P99s, GPU memory pressure, hallucination rates, and error classes
- Define alerting thresholds and build self serve model health tooling
What We’re Looking For
- Strong AWS and Kubernetes experience, with hands on depth in EKS, GPU workload scheduling, and IAM patterns
- Practical LLMOps experience: model serving frameworks (vLLM, TGI or similar), prompt versioning, model registry management, A/B deployment, shadow traffic, rollback strategies
- Comfort with Python and enough ML context to hold a real conversation about fine tuning, RLHF, RAG architectures, and evaluation methodology
- Infrastructure as code fluency in Terraform
- Experience building evaluation frameworks for generative models — not just accuracy metrics, but latency, cost, and output safety
- A bias toward observable, auditable systems — especially in a regulated industry context
- Strong engineering habits: small PRs, meaningful code review, test coverage, and a low tolerance for tech debt
- Willingness to get close to the clinical domain
Bonus
- Experience working with HIPAA, the Australian Privacy Act, or similar healthcare data requirements
- Prior work on latency sensitive inference pipelines for consumer or clinical products
- Familiarity with agent orchestration frameworks (LangChain, LangGraph, or similar)
- Experience with Karpenter or cluster autoscaler tuning for bursty GPU workloads
Why you will flourish with us
- Flexible hybrid working environment, with 3 days in the office
- A generous personal development budget of $500 per annum
- Learn from some of the best engineers and creatives, joining a diverse team
- Become an owner, with shares (equity) in the company
- The rare chance to create a global impact as you immerse yourself in one of Australia’s leading healthtech startups
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Heidi
LLM Ops Engineer
Sydney, KSA
Heidi, a healthtech startup building an AI Care Partner, is seeking an LLM Ops Engineer to design and maintain AWS/EKS infrastructure for GPU-backed model workloads. You will own model serving, evaluation, cost optimizat
Customer Success Manager
United Arab Emirates, UAE
The Role As a Customer Success Manager at Heidi Health, you ll be the bridge between technology and care - helping hospitals, clinics, and clinicians across the Middle East adop...
Customer Success Manager | GCC
Dubai, UAE
Who Are Heidi Heidi is building an AI Care Partner that supports clinicians every step of the way, from documentation to delivery of care. We exist to double healthcare’s capacity while keeping care deeply human. In 18 m
Business Development - Kitchen & Bakery Equipment
Dubai, UAE
• Identify and develop new business opportunities within cafés, bakeries, restaurants, hotels, and food facilities. • Build and maintain strong relationships with clients, consultants, and contractors • Conduct market re
Customer Success Manager | GCC
Dubai, UAE
Who Are Heidi Heidi is building an AI Care Partner that supports clinicians every step of the way, from documentation to delivery of care. We exist to double healthcare’s capacity while keeping care deeply human. In 18 m
Customer Success Manager | GCC
Dubai, UAE
Who are Heidi Heidi is building an AI Care Partner that supports clinicians every step of the way, from documentation to delivery of care. We exist to double healthcare’s capacity while keeping care deeply human. In 18 m
SMB Growth Manager (GCC)
Dubai, UAE
Who is Heidi? Heidi is building an AI Care Partner that supports clinicians every step of the way, from documentation to delivery of care. We exist to double healthcare’s capacity while keeping care deeply human. In 18 m
SMB Growth Manager (GCC)
Dubai, UAE
Who is Heidi? Heidi is building an AI Care Partner that supports clinicians every step of the way, from documentation to delivery of care. We exist to double healthcare’s capacity while keeping care deeply human. In 18 m
LLM Ops Engineer
Sydney, KSA
Customer Success Manager
United Arab Emirates, UAE
Customer Success Manager | GCC
Dubai, UAE
Business Development - Kitchen & Bakery Equipment
Dubai, UAE
Customer Success Manager | GCC
Dubai, UAE
Customer Success Manager | GCC
Dubai, UAE
SMB Growth Manager (GCC)
Dubai, UAE
SMB Growth Manager (GCC)
Dubai, UAE