- Lead evaluation sessions with external vendors, startups, and internal teams to probe architectural designs, technical credentials, and applicability within existing systems and information architecture.
- Drive team adoption of enterprise-authorized AI-assisted engineering practices to improve code quality, delivery speed, and operational outcomes, while establishing validation standards and promoting reuse of effective patterns.
- Apply knowledge of tools within the Software Development Life Cycle toolchain, including AI-assisted development and automation capabilities, to improve value realized by automation.
Lead architecture and engineering of large-scale data processing and platform solutions using Python, and Java.
Design and implement robust ETL/ELT pipelines, including ingestion, transformation, validation, reconciliation, and publishing across curated layers.
Build and operationalize Medallion architecture patterns for data quality, lineage, governance, and reuse.
Develop and optimize solutions on Data Lakes partitioning strategies.
Ensure engineering best practices: code quality, testing, CI/CD, observability, security-by-design, and operational readiness.
Drive performance optimization across Spark jobs (shuffle tuning, joins, caching, skew handling), storage layout, and Snowflake workloads.
Partner with product owners, architects, data governance, and downstream consumers to translate requirements into resilient technical solutions.
Required qualifications, skills, and capabilities:
- Formal training or certification on software engineering concepts and 5+ years of applied experience.
- Demonstrated experience leading effective use of approved AI-assisted software development tools, including setting expectations for validating outputs for correctness, performance, and security.
- Strong understanding of responsible AI use in engineering workflows, including data sensitivity, secure handling of inputs/outputs, and adherence to resiliency and security expectations; experience coaching engineers on safe, compliant adoption within delivery practices.
Strong hands-on development skills in Python and/or Java (ideally both).
Strong experience with Apache Spark and distributed data processing concepts.
Proven expertise building ETL/ELT pipelines and data integration frameworks.
Strong understanding of data storage/serialization and table/file formats, including Parquet and Avro.
Deep understanding of Big Data ecosystem fundamentals (distributed compute, fault tolerance, partitioning, data quality, metadata management).
Strong experience implementing Medallion architecture and Data Lake design principles.
Strong working knowledge of Snowflake including loading/unloading patterns and performance considerations.
Ability to lead technical decisions, drive alignment across teams, and communicate clearly with technical and non-technical stakeholders.
Preferred qualifications, skills, and capabilities:
Experience with data orchestration frameworks and pipeline automation.
