workday
Senior Lead Data & AI Engineer
Vanguard
Toronto, CAN
Full-time
Senior · 9+ years experience
Hybrid
Discovered 3 weeks ago
DatabricksDelta Live TablesLakeflowDatabricks WorkflowsAgentic systemsAgentOps
Free
Job Fit Check
Base Career helps you apply smarter for this job.
?%
Ready to ScanKey skills for this role
DatabricksDelta Live TablesLakeflow
Key Skills for This Role
DatabricksDelta Live TablesLakeflowDatabricks WorkflowsAgentic systemsAgentOps
Full Job Posting
Key Responsibilities
- Architect and build reusable, metadata-driven data and AI engineering frameworks that standardize ingestion, transformation, feature engineering, and AI workflow deployment. Leverage Databricks, lakehouse architecture, declarative pipelines, and cloud-native services to enable scalable, governed, and reusable data products across the organization.
- Architect and deploy Delta Live Tables and Lakeflow jobs on Databricks to automate data processing, AI pipelines, and agent data refresh cycles.
- Leverage Databricks Workflows and Job Orchestration to schedule and monitor AI agent deployments across multiple business workflows.
- Integrate Lakeflow for real-time data stream processing, ensuring AI agents are updated and responsive to live data.
- Ensure seamless orchestration between AI models and data pipelines, using event-driven architectures for real-time inference and deployment.
- Implement and orchestrate AI agents using frameworks such as Agentic systems, AgentOps tooling, and solutions like Agents on Databricks (Agent-bricks).
- Hands-on experience deploying AI agents using RAG, Graph RAG, MCP-enabled integrations, and agent orchestration frameworks such as AgentOps, AgentBricks, LangGraph, or cloud-native orchestration services.
- Manage AI agent lifecycles, monitoring, and scaling using tools like SageMaker, Bedrock, or AI orchestration frameworks on AWS.
- Ensure robust data governance, metadata management, and AI observability through Unity Catalog, AWS Glue, or custom metadata layers.
- Design for scalability and modularity, ensuring AI agents are reusable across multiple business processes.
Qualifications
- 9–12+ years in data engineering, specializing in AI deployment within cloud ecosystems (Databricks, AWS).
- Hands-on experience deploying AI agents using frameworks like RAG, graph RAG, and orchestrating agents (AgentOps, Agent-bricks, etc.).
- Proficient in AWS AI/ML services (SageMaker, Bedrock) and orchestration tools (MWAA, Step Functions).
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Vanguard
Finance - Fund and Investment Operations
London, GBR
Discovered TodayFull-time
Financial Crime Operations Specialist – Quality Control
Manchester, GBR
Discovered TodayFull-time
Finance - Fund and Investment Operations
London, GBR
Discovered TodayFull-time
Receptionist
Manchester, GBR
Discovered TodayFull-time
Application Engineering Technical Lead - II
Malvern, USA
Discovered TodayFull-time
Product Manager, Adobe Journey Optimizer
Malvern, USA
Discovered TodayFull-time
Product Manager, Multilingual Experience & Adobe Experience Manager
Malvern, USA
Discovered TodayFull-time
AI Engineer
Malvern, USA
Discovered TodayFull-time