Senior System Architect, Infrastructure Reliability
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
NVIDIA is seeking a Senior System Architect to solve failure attribution at scale in accelerated computing. You will architect a framework to ingest telemetry from CPU and GPU clusters, identify root causes of job failures in real-time, and distinguish hardware faults, infrastructure instability, and software defects.
Key Skills for This Role
Responsibilities
- Architect Failure Attribution Frameworks: Build a scalable 'flight recorder' for EDA jobs capturing high fidelity state across CPU, GPU, and Fabric at failure
- Build automated diagnostics correlating GPU XID errors, PCIe bus failures, and CUDA memory exceptions with system level events
- Implement low overhead tracing mechanisms for job execution across multi node Slurm or Kubernetes clusters
- Develop heuristics and ML models to classify failures as Hardware Fault, Software Bug, or Environment Issue
- Work with hardware and infrastructure teams to define signals of impending failure for proactive job migration or checkpointing
Requirements
- BS, MS, or PhD in Computer Science or Electrical Engineering (or equivalent experience) with 6+ years in systems programming
- Experience building automated RCA pipelines for HPC or cloud scale environments
- Expert knowledge of x86/ARM node level metrics: IPC, cache contention, NUMA imbalance, hardware interrupts
- Strong C++ and Python skills, with ability to build high performance daemons
- Familiarity with cluster resource managers (Slurm, LSF, or Kubernetes)
- Expert knowledge of Linux kernel error reporting interfaces (/dev/mcelog, dmesg, journald) (preferred)
- Deep experience with NVIDIA DCGM and NVML (preferred)
- Experience with checkpoint/restore technologies like CRIU (preferred)
Full Job Posting
Job Description
- NVIDIA is seeking a Senior System Architect: Heterogeneous EDA Systems to solve failure attribution at scale in accelerated computing.
- Develop and build an automated framework to ingest telemetry from CPU and GPU clusters to identify root cause of job failures in real time.
- Distinguish between hardware faults, infrastructure instability, and software defects.
What You'll Be Doing
- Architect Failure Attribution Frameworks: Build a scalable 'flight recorder' for EDA jobs capturing high fidelity state across CPU, GPU, and Fabric at failure.
- Build automated diagnostics correlating GPU XID errors, PCIe bus failures, and CUDA memory exceptions with system level events.
- Distributed Logging & Tracing: Implement low overhead tracing mechanisms for job execution across multi node Slurm or Kubernetes clusters.
- Root Cause Automation: Develop heuristics and ML models to classify failures as Hardware Fault, Software Bug, or Environment Issue.
- Resiliency Engineering: Work with hardware and infrastructure teams to define signals of impending failure for proactive job migration or checkpointing.
What We Need To See
- BS, MS, or PhD in Computer Science or Electrical Engineering (or equivalent) with 6+ years in systems programming.
- Experience building automated RCA pipelines for HPC or cloud scale environments.
- Expert knowledge of x86/ARM node level metrics: IPC, cache contention, NUMA imbalance, hardware interrupts.
- Strong C++ and Python skills, with ability to build high performance daemons.
- Familiarity with cluster resource managers (Slurm, LSF, or Kubernetes).
Ways To Stand Out From The Crowd
- Expert knowledge of Linux kernel error reporting interfaces (/dev/mcelog, dmesg, journald).
- Deep experience with NVIDIA DCGM and NVML for monitoring device health.
- Experience with non intrusive monitoring of application health and syscall level failure patterns.
- Experience with checkpoint/restore technologies like CRIU.
Compensation
- Base salary range: 184,000 USD 287,500 USD for Level 4, and 224,000 USD 356,500 USD for Level 5.
- Also eligible for equity and benefits.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at NVIDIA AI
Senior Solutions Architect, Customer Success
Dubai, UAE
Job Requisition ID JR2019277 Job Category Professional Services Time Type Full time NVIDIA is looking for a Senior Solutions Architect, Customer Success to join its NVIDIA Infrastructure Specialist Team. Academic and com
Solutions Architect - NVIDIA Cloud Partners and Datacentre Infrastructure
Dubai, UAE
Job Requisition ID JR2013942 Job Category Sales Time Type Full time NVIDIA has been transforming computer graphics, PC gaming, and accelerated computing for more than 25 years. It’s a unique legacy of innovation that’s f
Senior Solutions Architect, Customer Success
Dubai, UAE
Job Requisition ID JR2016438 Job Category Professional Services Time Type Full time NVIDIA is looking for a Senior Solutions Architect, Customer Success to join its NVIDIA Infrastructure Specialist Team. Academic and com
Senior Solutions Architect, Cloud Infrastructure and DevOps
Dubai, UAE
Job Requisition ID JR2016420 Job Category Professional Services Time Type Full time NVIDIA is looking for a Senior Cloud Infrastructure and DevOps Solutions Architect to join its NVIDIA Infrastructure Specialist Team. Ac
Senior Solutions Architect, Customer Success
Dubai, UAE
Solutions Architect - NVIDIA Cloud Partners and Datacentre Infrastructure
Dubai, UAE
Senior Solutions Architect, Customer Success
Dubai, UAE
Senior Solutions Architect, Cloud Infrastructure and DevOps
Dubai, UAE