Lead Data Architect

Senior/Lead technical data architect to design, build, and operate enterprise data platforms that power GenAI and AI/ML use cases.

This is a highly technical, handson role responsible for data platform architecture, endtoend data engineering, ML/LLM pipeline design, production model onboarding, and delivery of scalable Databricks- centric solutions across cloud environments.

Candidate must be AWS Certified Machine Learning – Specialty.

• Architect and implement enterprise data platforms (batch + streaming) optimized for ML, LLMs, and GenAI workloads.
• Lead design and hands on implementation of Databricks workspaces, Unity Catalog, Delta Lake design patterns, cluster policies, and performance tuning.
• Build and own end to end data pipelines (ingest, transform, feature engineering, serving) using PySpark, Databricks Jobs, Spark SQL, Delta Lake, and orchestration tools.
• Design and operationalize model training, fine tuning (LLM), evaluation, deployment, and monitoring pipelines (MLOps/RAG/CAG) integrating Databricks MLflow, CI/CD, and infra-as-code.
• Implement vectorless and vectorization/embedding pipelines, vector store integrations, and retrieval layers for RAG (FAISS, Pinecone, Weaviate, Milvus).
• Define data schemas, governance, lineage, access controls, and data product APIs; implement Unity Catalog or equivalent for centralized governance.
• Drive cost/performance optimization for storage, compute (spot/preemptible),and query patterns.
• Collaborate with engineers, data scientists, product owners, and security to translate business needs into production GenAI solutions.
• Mentor and lead engineering teams; conduct architecture reviews, code reviews, and run technical deep dives.
• Implement observability for data and ML pipelines (metrics, logging, data quality tests, alerting).
• Create reproducible experiment tracking, model registry, and rollout strategies (canary, shadow testing, rollback).
• Stay current on GenAI/LLM architectures and evaluate/introduce new tooling and frameworks.