AWS Databricks Architect -Sr Staff

General Summary:

The Databricks Architect will be responsible for technical decision-making, code quality, and the successful delivery of complex data projects using Delta Lake, Unity Catalog, and Mosaic AI. The Databricks Architect/Lead owns the end-to-end strategy for data ingestion, storage, and analytics and serve as a liaison between business stakeholders and engineering teams, ensuring that the platform is secure, cost-optimized, and capable of supporting advanced AI/ML workloads. Key Responsibilities • Lakehouse Design: Architect end-to-end data solutions using the Medallion Architecture (Bronze/Silver) and Delta Lake. • Platform Governance: Implement and manage Unity Catalog for centralized data governance, fine-grained access control, and lineage tracking. • Technical Leadership: Lead and mentor cross-functional teams of data engineers; conduct code reviews and define engineering best practices in Python, PySpark, and SQL. • Optimization & Performance: Drive cost-efficiency by tuning Spark jobs, managing cluster policies, and leveraging serverless compute and auto-scaling. • DevOps & DataOps Integration: Integrate Databricks workflows into CI/CD pipelines using tools like Databricks Asset Bundles, Terraform, and Azure DevOps or GitHub. • AI/ML Support: Provide architectural guidance for integrating machine learning models, including feature engineering and model deployment via MLflow. • Deep hands-on technical understanding in building complex data models for supporting various analytics needs • Comfortable in working with cross-functional teams such as Data Engineering, Analytics, SMEs and Business stakeholders • Passion to understand complex existing footprint of data solutions and build out the strategy for migrating to Databricks medallion architecture • Accountability and Ownership of the delivery in driving the projects to completion

• 14+ years of experience in data management, data engineering, or data science roles, with at least 7 years in building enterprise level data models as part of data modernization and DW migration efforts. • 8+ years of work experience with Data Structures and algorithms • 8+ years of work experience with programming (e.g., Java, Python) • 8+ years of work experience with SQL or NoSQL Databases • 3+ years of work experience in a role requiring interaction with senior leadership, leading modernization of Data platforms.

Bachelor's or Master's degree in Computer Science, Data Science, Information Technology, or a related field. • Strong understanding and hands-on experience with lakehouse medallion data architecture and govenance • Proven implementation experience with migrating legacy DW data models to 3-layer medallion architecture. • Strong passion in exploring GenAI based solutions for documenting the existing DW footprint and build out strategic data model for Databricks platform • Experience with machine learning, artificial intelligence, and predictive analytics is a plus. • Excellent leadership, communication, and collaboration skills in a global hybrid delivery model • Strong problem-solving skills with strategic thought process and self-initiative and proctive soft skills