Senior Principal Application Support Engineering Lead – AI Products & Agents

About the Technology Organization

Role Summary

What You'll Be Doing (Key Responsibilities)

• 1) Support Operations Leadership & Shift Ownership
• Lead end-to-end support operations for a technology team, ensuring consistent execution across shifts and time zones.
• Act as the primary operational leader during assigned shifts, accountable for incident response quality, prioritization, and decision-making.
• Ensure effective shift handovers, operational continuity, and shared accountability across global support teams.
• Establish and evolve shift-level operating models, escalation paths, and decision frameworks.
• 2) Major Incident Command & Executive Escalation
• Serve as the incident commander for the most complex, high-impact production incidents.
• Lead war-room execution, cross-team coordination, and recovery strategy across engineering, product, platform, security, and vendor teams.
• Provide clear, timely, and confident communication to senior technology and business stakeholders during outages.
• Ensure incidents are handled with rigor, consistency, and accountability.
• 3) Enterprise Problem Management & Defect Elimination
• Own Problem Management for recurring and systemic issues across the supported technology landscape.
• Drive high-quality Root Cause Analysis (RCA) and ensure corrective actions address root causes—not symptoms.
• Hold teams accountable for long-term fixes and track outcomes to measurable reliability improvements.
• Identify cross-product failure patterns and influence architectural or platform-level remediation.
• 4) Reliability Strategy & Operational Excellence
• Define and drive operational reliability strategy for the technology team, aligned with SRE and production engineering principles.
• Influence the adoption of SLIs, SLOs, error budgets, and reliability reporting across products.
• Champion improvements in availability, performance, scalability, resilience, and recovery capabilities.
• Establish and enforce operational readiness standards (runbooks, rollback plans, monitoring coverage, post-release validation).
• 5) Observability, Automation & Toil Reduction
• Set direction for observability strategy across logs, metrics, and traces, ensuring actionable insights and high signal quality.
• Drive automation initiatives that significantly reduce manual effort, human error, and MTTR.
• Promote standard tooling, reusable runbooks, and automated remediation patterns across teams.
• Ensure support operations scale through systems and automation—not heroics.
• 6) Deployment, Change & Release Governance
• Provide senior operational oversight for releases and deployments, including risk assessment and go/no-go decisions.
• Partner with Engineering and Platform teams to improve CI/CD operational safety and post-release validation.
• Ensure operational risks are identified early and mitigated before production impact.
• 7) Compliance, Security & Regulated Environment Readiness
• Ensure support operations comply with Lilly standards and applicable regulatory requirements.
• Promote secure operational practices, auditability, and proper handling of sensitive data during incidents.
• Act as a trusted operational leader in regulated and validated environments.
• 8) Organizational Leadership & Talent Development
• Set the operational bar for the organization through standards, expectations, and role modeling.
• Mentor and develop R2/R3 engineers, building deep operational expertise and leadership capability.
• Influence managers, architects, and engineering leaders through credibility and outcomes rather than authority.
• Contribute to the evolution of enterprise-wide support and reliability practices.

How You Will Succeed (R4 Success Profile)

What You Should Bring (Qualifications)

• Required
• 10–14+ years of experience in Application Support, Production Engineering, SRE, or Software Engineering with deep operational ownership.
• Extensive experience leading high-severity incident response and operational execution.
• Deep hands-on troubleshooting across distributed applications, integrations, databases, and cloud platforms.
• Strong experience with monitoring, logging, and alerting platforms (e.g., Datadog, Splunk, ELK, AppDynamics, CloudWatch).
• Advanced scripting and automation skills (e.g., Python, Bash) to drive operational efficiency.
• Proven ability to operate in regulated and enterprise environments.
• Proven ability with skills like Power Automate, LLM models, Gen AI, Agentic, Gen AI.
• Exceptional communication and leadership skills under pressure.
• Preferred / Nice to Have
• Experience operationalizing SLIs, SLOs, and error budgets.
• Familiarity with containers, Kubernetes, CI/CD pipelines, and Infrastructure as Code concepts.
• Experience leading globally distributed support teams or shift-based operations.
• Track record of influencing enterprise-wide operational standards.

Leadership Expectations

Additional Information