Senior Data Engineer

Overview:

Responsibilities:

• Assembles large, complex data sets that meet functional and non-functional requirements, ensuring that the design and engineering approach is consistent across multiple systems
• Maintains, improves, cleans, and manipulates large data for operational and analytics data systems, builds complex processes supporting data transformation, data structures, metadata, data quality controls, dependency, and workload management, and communicates required information for deployment, maintenance, and support of business functionality
• Utilizes multiple architectural components in the design and development of client requirements and collaborates with development teams to understand data requirements and ensure the data architecture is feasible to implement
• Defines and builds data pipelines to enable data-informed decision making, ensuring adherence to release processes and risk management routines
• Contributes to existing test suites including integration, regression, and performance, analyzes test reports, identifies any test issues and errors, and leads triage of underlying causes
• Leads the identification of gaps in data management standards adherence and works with appropriate partners to develop plans to close gaps, leading concept testing and conducting research to prototype toolsets and improve existing processes
• Mentors Data Engineers in the delivery and release of continuous integration and continuous delivery events and defines key performance indicators and internal controls
• Required Qualifications:
• 12+ years of experience with Data Engineering
• Experience with large-scale analytical platforms, metadata-driven architectures, and distributed data processing frameworks is preferred.
• Design and implement cloud-native data platforms utilizing Object Storage, Data Lakehouse architecture, distributed query engines, and streaming technologies to support real-time, near real-time, and batch processing requirements.
• Hands on experience in Scala, Spark, Kafka Streaming
• Very good understanding of Hadoop components, architecture & tool sets: Spark and Scala, Informatica, Starburst, Object Storage in enterprise data platforms
• Must have Architected, developed & implemented solutions using real time streaming technologies
• Good knowledge of performance tuning
• Experience working with Exadata & Oracle SQL Tuning
• Experience working with Starburst and Object Storage
• Experience developing Streaming/Near Real time and Batch data integration solutions.
• Hands on experience in UNIX Shell Scripting
• Working experience with scheduling tools
• Desired Qualifications:
• CICD Build & Deployment tools
• API development experience
• Financial Services / Banking experience
• Experience in distributed computing
• Experience with open table formats such as Apache Iceberg, Delta Lake, or equivalent technologies
• Experience with enterprise metadata management, data lineage, governance, and data quality frameworks

Skills:

• Analytical Thinking
• Application Development
• Data Management
• Risk Management
• Solution Design
• Agile Practices
• Architecture
• Collaboration
• Decision Making
• DevOps Practices
• Business Acumen
• Data Quality Management
• Financial Management
• Solution Delivery Process
• Test Engineering
• Shift:
• Hours Per Week: