Manager Big Data Engineering - Databricks Lead - Hybrid

Set Yourself Apart With

Qualifications

• Your Skills and Experience - 10+ years of demonstrated experience leading the implementation of enterprise-scale data platforms and end-to-end data engineering solutions in production environments. - Hands-on experience with Databricks as a primary data engineering platform, including Delta Lake, Databricks Workflows, Databricks SQL, notebooks, jobs, and modern Lakehouse architecture patterns. - Strong experience designing and implementing scalable data platforms on one or more public cloud platforms including Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP). - Advanced proficiency in Python and practical experience using Python-based frameworks for data engineering, platform automation, and AI-enabled engineering workflows. - Strong expertise in Apache Spark, PySpark, Spark SQL, and distributed data processing technologies. - Experience implementing both batch and real-time data pipelines using technologies such as Spark Streaming, Glue ETL, Lambda, Dataflow, Azure Data Factory, Databricks, or similar frameworks. - Experience with data modeling, dimensional modeling, data warehousing, and modern architectural patterns including Lakehouse and data mesh approaches. - Experience with columnar data platforms such as Snowflake, BigQuery, Redshift, Vertica, or similar technologies. - Experience with NoSQL technologies such as DynamoDB, Bigtable, Cosmos DB, or equivalent distributed databases. - Experience implementing software engineering best practices including source control, CI/CD, automated testing, release management, infrastructure automation, and production support processes. - Familiarity with MLOps concepts and supporting data engineering responsibilities related to model deployment, validation, monitoring, rollback, governance, and operational reliability. - Experience leading engineering teams, managing delivery workstreams, and collaborating effectively across cross-functional and client-facing environments. - Strong communication, stakeholder management, and consulting skills. AI Engineering & Modern Data Platform Experience - Experience supporting AI-enabled solutions through the design and implementation of scalable, production-grade data platforms. - Exposure to AI engineering concepts including context engineering, retrieval-augmented generation (RAG), agentic architectures, semantic search, and production data services supporting AI-powered experiences. - Experience building and maintaining the data pipelines that support retrieval systems, including document ingestion, parsing, chunking, metadata extraction, embedding generation, and incremental indexing processes. - Familiarity with vector databases, semantic search platforms, graph-based knowledge stores, and modern retrieval architectures. - Exposure to cloud AI services such as Vertex AI, Azure AI Services, AWS AI Services, or similar AI platforms. - Experience supporting AI and machine learning lifecycle requirements, including evaluation datasets, monitoring, operational telemetry, validation workflows, release management, and platform observability. - Understanding of platform requirements for managing agent state, conversation history, session context, memory stores, and durable retrieval structures that support AI-enabled applications. - Ability to apply enterprise data engineering principles such as lineage, governance, provenance, observability, and data contracts to AI-enabled platforms and retrieval systems. - Experience supporting agentic frameworks, orchestration platforms, or emerging AI engineering technologies is a plus. - Experience with Snowflake and zero-copy architecture patterns is a plus, particularly within retail, financial services, energy, logistics, manufacturing, or CPG industries. Set Yourself Apart With<

Responsibilities

• UNAVAILABLE