Senior Platform Reliability Engineer

AI FactoryOS Operations

Role Summary

Key Responsibilities

• Operate the multi-tenant control and management plane that Firmus' AI and infrastructure services depend on, including the tenancy, quota and access controls that support separation between tenant workloads and data.
• Operate Firmus' exabyte-scale distributed and high-performance filesystems (for example VAST, WEKA, Ceph) and S3-compatible object storage to their declared service levels, own the operational automation around them, and drive continuous improvement in how they are operated, feeding platform improvement requirements to AI Infrastructure with evidence.
• Operate the GPU compute fleet in production: node health and readiness, GPU and node fault detection and handling, firmware and driver currency to the supported baselines, remediation and return-to-service, and the hardware fault and replacement workflow with vendors and site operations.
• Build and maintain the guarded automation and remediation tooling that turns manual response into a self-healing capability, delivered as controlled code and reviewed by AI Infrastructure where it affects service behaviour.
• Diagnose and tune performance across the full data path, applying a deep understanding of operating systems, computer networks and software-defined storage, and working with technologies including RDMA, GPU Direct Storage, RoCE and InfiniBand.
• Deliver change as code, and drive continuous improvement in cluster validation, CI/CD automation, and provisioning and testing frameworks.
• Operate the observability infrastructure as a shared service across metrics, logs, traces, alerting and retention, and work with service owners so that telemetry becomes alerting that is actionable and tied to a runbook.
• Operate each service in the portfolio to its published service level, carry new services through production readiness review, and execute the monthly patching cycle and urgent vulnerability remediation.
• Provide the deepest technical expertise for these platforms, taking on and diagnosing the faults that require internals-level knowledge to root cause, and driving the permanent fix to closure through AI Infrastructure , and manage vendor escalations at engineering level.
• Lead technical recovery during major incidents, drive the changes that remove repeat causes, share the follow-the-sun on-call roster, and mentor the engineers who carry frontline diagnosis, documenting operational procedures, runbooks and performance results.

Skills & Experience

• Required Skills
• Strong skills in infrastructure, systems and platform engineering, with 8+ years of experience including substantial ownership of production storage and shared infrastructure services in a 24/7 environment.
• Extensive experience with scale-out, parallel or enterprise storage supporting demanding workloads (for example VAST, WEKA, Ceph, Lustre, GPFS or NetApp), including diagnosis of performance and capacity problems across the full data path using evidence rather than assumption.
• Substantial experience operating a virtualisation platform (for example Proxmox, VMware or KVM) and building shared services such as databases and object storage to a defined service level.
• Expert-level knowledge of Linux systems, including storage and file system internals, kernel and driver behaviour, networking, memory and I/O subsystems, and systematic performance analysis.
• Extensive experience operating observability infrastructure as a service across metrics, logs, traces and alerting (for example Prometheus, Grafana, OpenTelemetry, Loki or Elasticsearch), and designing alerting that is actionable.
• Strong skills in infrastructure automation, infrastructure-as-code and GitOps practices (for example OpenTofu or Terraform, Ansible, Argo CD, CircleCI), with change delivered through peer review, automated testing and progressive rollout.
• Practical experience with scripting or programming for operational automation and tooling, such as Python, Go or Bash.
• Working competence with Kubernetes, sufficient to diagnose faults where shared services meet the container estate.
• Proven ability to act as a senior escalation point in production, including major incident response, on-call participation, post-incident review, vendor escalation, and the production of runbooks that others can execute successfully.
• Practical application of least privilege, secrets management, certificate-based access and audited privileged operations, and an understanding of how operational practice produces audit evidence.
• Clear technical judgement and communication skills, with the ability to produce incident updates, design notes and escalations that non-specialists can act on, and a working style suited to a small team that shares accountability for a broad portfolio.
• Preferred Experience
• Experience in storage and infrastructure operations for GPU or HPC environments, including the demands of large-scale distributed training and inference.
• Experience in multi-tenant service provider, cloud or colocation environments with contractual response commitments and customer-facing reporting.
• Familiarity with bare-metal lifecycle and provisioning automation (for example NVIDIA Infra Controller, MAAS, Ironic or Metal3), and standard operating environments with known-good-state enforcement.
• Experience with multi-tenant Kubernetes patterns and GPU-enabled Kubernetes.
• Knowledge of data centre and hardware fundamentals, including firmware management, DPU or SmartNIC-based networking, and high-performance fabrics such as InfiniBand or RoCE.
• Familiarity with policy-as-code and software-supply-chain controls, such as image signing, software bills of materials and vulnerability gating.
• Experience preparing for or operating under ISO 27001, or SOC 2, and responding to enterprise customer due diligence.
• A Bachelor's degree in computer science, engineering or a related discipline, or an equivalent combination of relevant experience and training.

Location & Reporting

• Location : Based in Australia or Singapore, with travel to Australian AI Factory sites as required.
• On-call: The function runs 24/7. First line monitoring and first response sit with the operations centre. This role shares the after-hours escalation roster for its domain with the other senior engineers in the function.
• Reporting to: Reports to the Head of AI FactoryOS Operations while the function is being established, working under broad direction with a high degree of autonomy and direct access to the decision makers. As the function reaches its planned structure, the role will report to the Service Reliability Manager, with the Head of AI FactoryOS Operations remaining accountable for the function. The scope, level and remit of the role do not change under either arrangement.

Employment Basis

Diversity