Site Reliability Engineer

ESSENTIAL DUTIES & RESPONSIBILITIES

• Describe the duties/functions essential to performing the job.
• Works an onsite 5-day weekly schedule consisting of Owl (midnight–8 am) shifts to monitor the NERSC HPC Facility.
• Review and respond to alerts from computer systems, storage, network, and other data center/facility-related systems by triaging or calling the appropriate on-call staff.
• Create appropriate solutions to improve processes, prevent issue recurrence, and automate responses to all routine service conditions.
• Identify issues and propose solutions that will improve monitoring capabilities or provide better automation for triage.
• Possess expertise in ServiceNow and its usage to develop and implement customized service management solutions.
• Respond to alerts from multiple systems to ensure that data collection continues 24/7, providing real-time information for diagnoses.
• Develop and maintain tools within the monitoring pipeline in collaboration with the Operations Team.
• Create new software to provide alerts and notifications from HPC system APIs into the monitoring pipeline.
• Builds and maintains application/tool configurations to ensure software runs reliably as data and user demands grow.
• Collaborate with other groups at NERSC to ensure that communication and workflows are clearly understood.
• Work closely with other NERSC groups to coordinate center-wide maintenance activities and manage diagnostic and notification software during maintenance periods.
• Perform regular physical and logical walkthroughs of the data center floor to monitor environmental health, power distribution units, and cooling infrastructure to ensure peak operational efficiency.
• Provide accurate information in the trouble ticketing system for outages, maintenance updates, and other incidents so that workflows and protocols can be appropriately tracked by others.
• Work on and resolve problems of diverse scope where data analysis requires the evaluation of identifiable factors.
• Demonstrate good judgment in selecting methods and techniques for obtaining solutions.