Senior Data Platform Engineer

• Design, build, and maintain scalable data and ML pipelines using Python and SQL, processing large-scale datasets across batch and streaming workloads
• Own and evolve core platform infrastructure on Databricks — including Delta Lake table architecture, Unity Catalog governance, Databricks Workflows orchestration, and compute optimization
• Build and maintain end-to-end ML pipelines: feature engineering, model training pipelines, experiment tracking (MLflow), and model deployment/serving infrastructure
• Collaborate with data scientists to operationalize models — bridging the gap between experimentation and production-grade ML systems
• Define and enforce data platform standards: ingestion patterns, data modeling conventions, medallion architecture (Bronze/Silver/Gold), and pipeline reliability practices
• Implement data quality, observability, and monitoring frameworks to ensure platform health and data trustworthiness
• Optimize pipelines for performance, cost, and reliability at scale using Spark and PySpark
• Evaluate, integrate, and govern new platform tooling and data sources within the Databricks ecosystem
• Contribute to architectural decisions and help drive the long-term data platform roadmap
• Participate in code reviews, technical design discussions, and engineering standards
• Mentor junior engineers and elevate overall platform and data engineering practices
• Document platform architecture, pipeline design, and operational runbooks

What you will have

• 6+ years of experience in data engineering, data platform, or ML engineering roles
• Strong proficiency in Python and SQL, with a track record of building production-grade data pipelines using both
• Hands-on Databricks expertise: Delta Lake, Unity Catalog, Databricks Workflows, PySpark, and the Databricks ecosystem broadly
• Experience building and maintaining ML pipelines in production — feature engineering, training pipelines, experiment tracking, and model deployment
• Familiarity with MLflow or comparable experiment tracking and model registry tools
• Experience working on cloud data platforms (AWS, Azure, or GCP)
• Strong understanding of data modeling, dimensional design, and analytics-friendly data architecture
• Experience with batch and incremental/CDC pipeline patterns
• Proficiency with Git, version control, and CI/CD practices for data and ML workflows
• Strong engineering judgment — you think about reliability, maintainability, and cost, not just correctness
• Clear communication and comfort working with both technical and non-technical stakeholders

Nice to have

• Experience with streaming or near real-time pipelines (Kafka, Kinesis, Spark Structured Streaming)
• Familiarity with feature store platforms (Databricks Feature Store, Feast, or Tecton)
• Experience with LLM pipelines, RAG architectures, or AI/BI tooling (Genie, AI Functions)
• Knowledge of data quality and observability tooling (Great Expectations, Monte Carlo, etc.)
• Exposure to dbt or similar SQL-based transformation frameworks
• Infrastructure-as-code experience (Terraform, Databricks Asset Bundles)
• Experience working in Agile or Scrum environments
• Prior experience mentoring engineers or shaping platform standards