LLM Ops Engineer
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
Heidi, a healthtech startup building an AI Care Partner, is seeking an LLM Ops Engineer to design and maintain AWS/EKS infrastructure for GPU-backed model workloads. You will own model serving, evaluation, cost optimization, and observability.
Key Skills for This Role
Responsibilities
- Design, deploy and maintain AWS/EKS infrastructure running GPU backed model workloads
- Manage GPU node pools, tune autoscaling for inference traffic patterns, and own the full model serving lifecycle from container builds to production rollouts
- Write and maintain infrastructure as code in Terraform
- Build tooling that measures whether models are performing — clinically accurate, latency appropriate, cost efficient
- Design offline evaluation harnesses, automated regression tests, and dashboards that surface regressions before they reach clinicians
- Own GPU utilization, quantization, request batching, model routing, and spot/on demand node strategy
- Work closely with the Models Team on fine tuning workflows and model selection tradeoffs
- Collaborate with product engineers and clinicians on prompt engineering, context window management, and model data pipelines
- Instrument token usage, latency P99s, GPU memory pressure, hallucination rates, and error classes
- Define alerting thresholds and build self serve model health tooling so the team is not relying on Slack threads to know something is wrong
Requirements
- Strong AWS and Kubernetes experience, with hands on depth in EKS, GPU workload scheduling, and IAM patterns
- Practical LLMOps experience: model serving frameworks (vLLM, TGI or similar), prompt versioning, model registry management, A/B deployment, shadow traffic, rollback strategies
- Comfort with Python and enough ML context to hold a real conversation about fine tuning, RLHF, RAG architectures, and evaluation methodology
- Infrastructure as code fluency in Terraform
- Experience building evaluation frameworks for generative models — not just accuracy metrics, but latency, cost, and output safety
- A bias toward observable, auditable systems — especially in a regulated industry context
- Strong engineering habits: small PRs, meaningful code review, test coverage, and a low tolerance for tech debt
- Willingness to get close to the clinical domain: you do not need to be a clinician, but you need to care about the consequences of model failures in healthcare
Full Job Posting
Who We Are
- Healthcare needs a better rhythm: one that keeps care continuous and deeply human.
- Heidi is building an AI Care Partner that works alongside clinicians to make that possible.
- We’re a team of doctors, engineers, designers, researchers, and creatives building tools that help clinicians stay focused on what matters most: their patients.
- In just 18 months, Heidi has given back more than 18 million hours to healthcare professionals — supporting 73 million patient visits in 116 countries.
- Backed by nearly $100 million in funding, we’re growing in the US, UK, Canada, and Europe.
What you’ll do
- Design, deploy and maintain AWS/EKS infrastructure running GPU backed model workloads.
- Manage GPU node pools, tune autoscaling for inference traffic patterns, and own the full model serving lifecycle from container builds to production rollouts.
- Write and maintain infrastructure as code in Terraform.
- Build tooling that measures whether models are performing — clinically accurate, latency appropriate, cost efficient.
- Design offline evaluation harnesses, automated regression tests, and dashboards that surface regressions before they reach clinicians.
- Own GPU utilization, quantization, request batching, model routing, and spot/on demand node strategy.
- Work closely with the Models Team on fine tuning workflows and model selection tradeoffs.
- Collaborate with product engineers and clinicians on prompt engineering, context window management, and model data pipelines.
- Instrument token usage, latency P99s, GPU memory pressure, hallucination rates, and error classes.
- Define alerting thresholds and build self serve model health tooling so the team is not relying on Slack threads to know something is wrong.
What we’re looking for
- Strong AWS and Kubernetes experience, with hands on depth in EKS, GPU workload scheduling, and IAM patterns that do not cut corners on healthcare data requirements.
- Practical LLMOps experience: model serving frameworks (vLLM, TGI or similar), prompt versioning, model registry management, A/B deployment, shadow traffic, rollback strategies.
- Comfort with Python and enough ML context to hold a real conversation about fine tuning, RLHF, RAG architectures, and evaluation methodology.
- Infrastructure as code fluency in Terraform.
- Experience building evaluation frameworks for generative models — not just accuracy metrics, but latency, cost, and output safety.
- A bias toward observable, auditable systems — especially in a regulated industry context.
- Strong engineering habits: small PRs, meaningful code review, test coverage, and a low tolerance for tech debt.
- Willingness to get close to the clinical domain: you do not need to be a clinician, but you need to care about the consequences of model failures in healthcare.
Why you will flourish with us
- Flexible hybrid working environment, with 3 days in the office.
- A generous personal development budget of $500 per annum.
- Learn from some of the best engineers and creatives, joining a diverse team.
- Become an owner, with shares (equity) in the company, if Heidi wins, we all win.
- The rare chance to create a global impact as you immerse yourself in one of Australia’s leading healthtech startups.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Heidi
LLM Ops Engineer
Melbourne, UAE
Heidi is looking for an LLM Ops Engineer to design and maintain AWS/EKS infrastructure for GPU-backed model workloads, build evaluation tooling, and optimize cost and performance. You will work closely with product engin
Customer Success Manager
United Arab Emirates, UAE
The Role As a Customer Success Manager at Heidi Health, you ll be the bridge between technology and care - helping hospitals, clinics, and clinicians across the Middle East adop...
Customer Success Manager | GCC
Dubai, UAE
Who Are Heidi Heidi is building an AI Care Partner that supports clinicians every step of the way, from documentation to delivery of care. We exist to double healthcare’s capacity while keeping care deeply human. In 18 m
Business Development - Kitchen & Bakery Equipment
Dubai, UAE
• Identify and develop new business opportunities within cafés, bakeries, restaurants, hotels, and food facilities. • Build and maintain strong relationships with clients, consultants, and contractors • Conduct market re
Customer Success Manager | GCC
Dubai, UAE
Who Are Heidi Heidi is building an AI Care Partner that supports clinicians every step of the way, from documentation to delivery of care. We exist to double healthcare’s capacity while keeping care deeply human. In 18 m
Customer Success Manager | GCC
Dubai, UAE
Who are Heidi Heidi is building an AI Care Partner that supports clinicians every step of the way, from documentation to delivery of care. We exist to double healthcare’s capacity while keeping care deeply human. In 18 m
SMB Growth Manager (GCC)
Dubai, UAE
Who is Heidi? Heidi is building an AI Care Partner that supports clinicians every step of the way, from documentation to delivery of care. We exist to double healthcare’s capacity while keeping care deeply human. In 18 m
SMB Growth Manager (GCC)
Dubai, UAE
Who is Heidi? Heidi is building an AI Care Partner that supports clinicians every step of the way, from documentation to delivery of care. We exist to double healthcare’s capacity while keeping care deeply human. In 18 m
LLM Ops Engineer
Melbourne, UAE
Customer Success Manager
United Arab Emirates, UAE
Customer Success Manager | GCC
Dubai, UAE
Business Development - Kitchen & Bakery Equipment
Dubai, UAE
Customer Success Manager | GCC
Dubai, UAE
Customer Success Manager | GCC
Dubai, UAE
SMB Growth Manager (GCC)
Dubai, UAE
SMB Growth Manager (GCC)
Dubai, UAE