Senior Splunk Engineer - Infrastructure Operations

GovCIO is currently hiring for Senior Splunk Engineer - Infrastructure Operations of Infrastructure Operations to support our Administrative Office of the US Courts NLS project.

The NLS currently ingest an average of 18-20TB of logging data daily across 60 indexers distributed in 2 data centers.

This position is located within the United States and is fully remote.

• Design, implement, and operate the Splunk Core, Enterprise Security, IT Service Intelligence (i.e., ITSI), Phantom (Security Orchestration, Automation, and Response (SOAR)), Splunk Cloud, Splunk On-Call, and Multi-Site Index Clustering environment.
• Monitor overall Splunk health through the Monitoring Console (DMC) including indexer, search head, and cluster master status.
• Track indexing rates, license usage, queue health, and search concurrency to identify performance or ingestion issues early.
• Monitor CPU, memory, and disk utilization across all Splunk components to ensure optimal resource usage.
• Respond promptly to health alerts, DMC warnings, or anomalies observed on monitoring dashboards.
• Investigate and resolve common user-reported issues such as access problems, failed searches, or non-triggering alerts.
• Troubleshoot data ingestion, parsing, and indexing issues across Universal Forwarders, Heavy Forwarders, and HEC endpoints.
• Investigate missing or duplicate logs, timestamp errors, or sourcetype misassignments and escalate complex parsing issues to Engineering.
• Validate new data source onboardings by confirming sourcetype assignment, timestamp accuracy, and field extraction integrity.
• Support data source owners with forwarder deployment, syslog setup, and connectivity troubleshooting during initial onboarding.
• Maintain data flow visibility from source → forwarder → indexer to confirm data completeness and performance.
• Rotate and update credentials, API keys, or tokens used in data inputs, integrations, alerts, and scheduled searches.
• Manage RBAC user and role mappings, handling access requests, entitlement reviews, and permission troubleshooting.
• Provide end-user assistance with SPL searches, reports, alerts, and dashboards, including query optimization tips.
• Maintain and update knowledge base articles, SOPs, and FAQs for repeatable issues and troubleshooting steps.
• Log and escalate platform or parsing issues to the Engineering team with evidence such as logs, screenshots, and correlation IDs.