Data Engineer - Big Data, Python

The Applications Development Intermediate Programmer Analyst is an intermediate level position responsible for participation in the establishment and implementation of new or revised application systems and programs in coordination with the Technology team.

The overall objective of this role is to contribute to applications systems analysis and programming activities.

• Pipeline & Architecture Development: Utilize deep knowledge of data engineering principles, distributed systems, and ETL/ELT procedures to design, build, and optimize scalable data pipelines and architectures. Leverage scripting languages (e.g., Python, Bash) and SQL to automate workflows and analyze/optimize data processing code.
• Stakeholder Collaboration: Collaborate with business analysts, and cross-functional technology teams to understand data requirements, recommend robust data integration solutions, and deploy/support high-performance data platforms.
• Technical Design & Specifications: Apply strong programming skills (e.g., Python, Pyspark, Java, SQL) and software engineering best practices to design and implement scalable data models, schemas, and pipeline specifications.
• Data Quality & Security: Analyze data pipelines and storage systems to ensure data quality, integrity, and security. Conduct rigorous testing (unit, integration, and end-to-end) and debugging of data workflows, ensuring compliance with data governance and privacy standards.
• Performance Optimization: Identify data bottlenecks, analyze pipeline performance, and make evaluative judgments to recommend and implement optimizations for data ingestion, processing, and storage.
• Technical Problem Solving: Resolve complex data pipeline failures, data quality anomalies, and infrastructure bottlenecks by applying technical experience in big data technologies and cloud data warehouses.
• Autonomy & Execution: Operate with a high degree of autonomy and limited direct supervision in managing data engineering deliverables and pipeline deployments. Exercise independent judgment and autonomy in selecting appropriate data technologies, designing architectures, and resolving technical challenges.
• Risk & Compliance: Appropriately assess risk when making technical and business decisions, ensuring data privacy, security, and compliance with data protection regulations (e.g., GDPR, local laws).