Data Engineer 2

· Agent Frameworks: Hands-on experience building multi-agent systems using LangGraph, CrewAI, or AutoGen.

· Tool Development: Proven ability to write "Tools" (API wrappers, Python functions, SQL executors) that allow agents to perform complex data operations.

· Prompt Engineering for DE: Experience in specialized prompting techniques like ReAct, Chain-of-Thought, and Few-Shot specifically for generating valid SQL and Spark code.

· Evaluation: Experience using frameworks like Ragas or TruLens to evaluate the reliability of agentic outputs.

2. Data Engineering & Cloud (The Core)

· Databricks on AWS: 4+ years of deep technical experience with Unity Catalog, Delta Live Tables (DLT), and Photon-accelerated clusters.

· Spark Mastery: Expert-level PySpark skills with a deep understanding of RDDs, DataFrames, and internal optimization (Shuffle tuning, predicate pushdown).

· Data Modeling: Advanced knowledge of Medallion Architecture and designing optimized Parquet/Delta storage layers for both batch and real-time streams.

· Python Expert: You write production-grade, asynchronous Python code.

· Architectural Vision: Ability to design systems that balance the "hallucination" risks of GenAI with the "strictness" of financial data requirements.

Month 1-2: Modernize existing Databricks pipelines and optimize Delta table performance.

Month 3-4: Deploy your first "Data Agent" that can autonomously generate documentation or SQL queries from our metadata catalog.

Month 6: Build a multi-agent system that monitors data quality across the lake and suggests/executes corrective ETL actions with human-in-the-loop triggers.

· The Scale: We don't play in a sandbox. You will be building agents that interact with petabytes of financial data.

· The Tech Stack: We are 100% committed to the AWS + Databricks ecosystem, giving you the best tools to succeed.

· Innovation First: You aren't just maintaining pipelines; you are defining what "Data Engineering" looks like in the age of AI.

As an Equal opportunity employer, we wholeheartedly welcome people from all backgrounds irrespective of caste, religion, gender, marital status, sexuality, disability, class or age to be part of our team. We believe that everyone's unique experiences and viewpoints make us stronger together. Come and be a part of #OneSpace*, where your individuality is celebrated and embraced.