Systems Administrator Lead (Night Shift)

• Install, configure, validate, and troubleshoot NVIDIA GPU drivers, CUDA components, firmware, and supporting system software using tools such as Nvidia Smi, DCGM, and NVIDIA Fabric Manager.
• Investigate GPU Xid errors, NVLink/NVSwitch faults, PCIe errors, ECC events, GPU resets, and thermal or power-related issues.
• Diagnose communication and performance issues involving NCCL, GPUDirect RDMA, PCIe topology, NUMA placement, and multi-GPU systems, validating GPU-to-GPU, GPU-to-network, and GPU-to-storage communication after repairs.
• Coordinate replacement and RMA activities for GPUs, GPU trays, baseboards, NVSwitch components, system boards, and NICs.

• Administer enterprise Linux distributions across the production fleet, troubleshooting boot failures, kernel issues, systemd services, filesystems, and OS performance.
• Analyze system and kernel logs (journalctl, dmesg, lspci, dmidecode, ipmitool, ethtool, ss, sar, perf) and manage kernel modules, device drivers, DKMS packages, and OS updates.
• Support Linux networking, storage mounts, authentication, and security controls, and investigate OS crashes, kernel panics, and out-of-memory conditions.

• Support enterprise server platforms from vendors such as Dell, HPE, ASUS, Supermicro, and NVIDIA, troubleshooting processors, memory, PCIe devices, NICs, storage controllers, and other hardware components.
• Manage BIOS, BMC, CPLD, NIC, GPU, switch, and drive firmware, and perform remote console troubleshooting, power cycling, and log bundle generation.
• Partner with on-site Data Center technicians on component replacement, cabling validation, break-fix activity, and post-repair testing.

• Troubleshoot Ethernet, InfiniBand, and RoCE connectivity within HPC and AI environments, including link state, VLAN/bonding issues, MTU mismatches, packet loss, and RDMA/GPUDirect RDMA connectivity.
• Collaborate with Deployment Engineering to troubleshoot fabric instability, link degradation, congestion, and node-level connectivity issues.

• Troubleshoot server connectivity to parallel, distributed, and enterprise storage platforms such as NFS, Lustre, BeeGFS, GPFS/Spectrum Scale, and Ceph, diagnosing mount failures, throughput degradation, and capacity issues.
• Validate storage network paths and coordinate with storage and networking teams during service-impacting events

• Lead or support incident response for critical infrastructure events, participating in technical bridges and providing clear status updates during outages.
• Create detailed timelines, root-cause analyses, and corrective-action plans, maintaining accurate records in Jira Service Management or similar ITSM platforms.

Leadership and Collaboration

• Mentor junior and mid-level Systems Administrators and HPC Engineers, providing technical guidance during complex troubleshooting and infrastructure changes.
• Promote consistent troubleshooting methods, documentation standards, and operational discipline across shift handoffs.

• Bachelor's degree in Computer Science, Engineering, Information Technology, or related field.
• 7+ years of experience in Linux systems administration, HPC/AI infrastructure or data center operations, including hands-on troubleshooting in production environments.
• Hands-on experience supporting enterprise server hardware, bare-metal compute systems, and large-scale NVIDIA GPU infrastructure or accelerated computing platforms.
• Strong understanding of server hardware, firmware, BIOS, BMC, PCIe, NUMA, memory, storage, and network interfaces, with experience diagnosing failures using OS logs, BMC logs, and vendor diagnostics.
• Experience with NVIDIA drivers, CUDA compatibility, and GPU-management tools such as DCGM, Fabric Manager, or nvidia-smi.
• Working knowledge of high-performance networking (InfiniBand, RDMA, RoCE, NVIDIA/Mellanox adapters) and out-of-band management tools (Redfish, IPMI, iDRAC, or iLO).
• Proficiency in one or more automation/scripting languages (Python, Bash, or Ansible) and familiarity with observability platforms such as Prometheus, Grafana, or Datadog.
• Proven ability to troubleshoot across hardware, firmware, OS, network, storage, and cluster-management layers, backed by strong documentation and communication skills.
• Willingness to work a fixed, recurring shift and participate in scheduled on-call escalation.

5C Data Centers?

At 5C, we believe great people build great companies.

You'll build a rewarding career while helping shape the future of digital infrastructure - one of the fastest-growing industries in the world.

Build a rewarding career in one of the world's fastest-growing industries.

Help power the infrastructure behind AI and high-performance computing.

Your ideas matter - we empower employees to help shape our future.

Competitive pay plus meaningful, lasting impact on the work you do.

Life at 5C

We're more than a workplace - we're a team of builders, innovators, and problem-solvers united by a shared purpose: creating infrastructure that powers the technologies transforming our world.

Your voice matters here, and we encourage fresh ideas at every level.

Ready to Apply?

If this opportunity sounds like the right fit for you, we'd love to hear your story.

Apply today and discover what your future could look like at 5C Data Centers.

5C Data Centers is an equal opportunity employer.

We celebrate diversity and are committed to creating an inclusive environment where everyone can thrive. 5C evaluates qualified applicants without regard to race, color, religion, gender, national origin, age, sexual orientation, gender identity or expression, disability status, or any other legally protected characteristic.

#LI-LS1