Principal Software Engineer - Observability & Telemetry Data

Required Experience

• 10+ years building and operating large-scale distributed systems, data platforms, or observability infrastructure, including Principal or Staff-level experience.
• Hands-on production ownership of metrics, logs, and distributed tracing at scale.
• Depth in large-scale data architecture, open table formats, distributed processing, streaming ingestion, schema evolution, and query tuning.
• Practical Databricks and MLflow experience for model and agent telemetry.
• Practical OpenTelemetry experience with collectors, semantic conventions, context propagation, and sampling.
• Experience with high-volume telemetry pipelines, FluentBit or Fluentd, Kinesis or Kafka, and search backend design.
• Advanced production experience with AWS technologies and Kubernetes, including EKS, ECS Fargate, EC2, Lambda, and CloudWatch.
• 10+ years of programming experience with Go, Java, Python, or Scala, plus strong SQL.
• Experience with Terraform and GitOps workflows such as Flux or ArgoCD.
• Track record of technical direction adopted across multiple teams, strong incident response, and improved mean-time-to-resolution.
• Computer Science, Engineering, or related degree, or equivalent practical experience; ongoing U.S. work eligibility.

Preferred Qualifications

• Experience instrumenting LLM or agentic systems, including OpenTelemetry GenAI conventions and agent tool-call tracing.
• Data reliability engineering practice involving freshness, quality, lineage, and production pipeline SLOs.
• Telemetry cost engineering or FinOps experience at scale.
• Experience with regulated environments such as FedRAMP or GovCloud.
• Experience with Prometheus, Grafana, Alertmanager, Snowflake, Backstage, or similar platforms.

Workplace

• This is a full-time role that can be performed remotely from anywhere in the United States where Smartsheet is a registered employer.
• The role may also be based in the Bellevue, Washington office.

Compensation

• The U.S. base salary range is $222,500–$257,500 USD per year.
• The role is eligible for a market-competitive incentive opportunity.

Benefits

• Full-time employees receive subsidized medical, vision, and dental coverage, a 401(k) match, flexible time away, and sick time off.
• Benefits include life insurance, disability plans, paid holidays, parental leave, a volunteer day, professional development, and company-funded perks.

About Smartsheet