Principal Databricks Software Developer

Three months in: you own the design and improvement plan for at least one critical Databricks capability or platform standard, and your recommendations are showing up in how teams build, review, schedule, and operate Databricks workloads.

Six months in: teams seek your input before building high-risk workloads, not after they fail.

Platform standards are clearer, recurring anti-patterns are declining, performance and reliability trends have improved, and Databricks cost decisions are being made with discipline instead of guesswork.

• 8+ years in data engineering, platform engineering, or related software engineering roles, with at least 3+ years of deep hands-on Databricks experience.
• Production experience with Spark internals and performance tuning, including shuffle behavior, query planning, adaptive query execution, memory pressure, skew, spill, partitioning, and broadcast strategy.
• Strong working knowledge of Databricks compute options, cluster policies, job clusters, all-purpose clusters, instance pools, autoscaling, serverless eligibility, Photon, and workload right-sizing.
• Real experience reviewing and improving notebooks, jobs, and pipelines that other people built, including the ability to quickly identify the few design choices causing most of the runtime, cost, or reliability risk.
• Production experience with Delta Lake, Delta maintenance strategies, Structured Streaming, Auto Loader, materialized views or streaming tables, and data quality expectations.
• Strong understanding of Unity Catalog, workspace governance, lineage, permission models, catalog/schema organization, table documentation, and secure handling of sensitive data.
• Experience designing reliable job orchestration: task decomposition, dependency management, retries, repairs, timeouts, alerting, scheduling, max concurrent runs, and recoverability.
• A track record of identifying systemic reliability, performance, governance, or cost issues and driving practical remediation across teams.
• Strong communication skills, including the judgment to explain technical findings in a way that helps engineering leaders, product owners, and stakeholders make better decisions.
• A STEM degree, or equivalent depth earned through substantial production experience.

Also valuable

• Experience with Azure-based data platforms and enterprise cloud operations.
• Experience partnering with FinOps or platform teams to reduce cloud spend without sacrificing reliability or service-level expectations.
• Experience building reusable Databricks templates, platform automation, internal standards, or reference architectures.
• Familiarity with data privacy, security, auditability, and regulated business environments.
• Opinions, backed by evidence, about when Spark or Databricks is the right tool — and when it is not.
• Experience helping teams adopt AI-assisted engineering practices responsibly, especially for code review, test generation, documentation, and operational analysis.

Why this role

You’ll shape how PowerPlan uses Databricks for critical data workloads, and your judgment will directly influence reliability, performance, governance, and cost.

This is a senior, hands-on role where the work is visible, the problems are meaningful, and the best answer is not always obvious.

The right person will look at something that “works” and see what is fragile, wasteful, insecure, or difficult to operate before it becomes an incident.

You’ll set the standard for Databricks technical excellence, and the improvements you drive will make the platform stronger for every team that depends on it.

PowerPlan is an EOE