Consulting Partner - Data Engineering & Modern Data Platforms

• Serve as the senior technical advisor for complex data engineering and data platform transformation programs.
• Define enterprise data platform, lakehouse, and data engineering architectures.
• Lead architecture assessments and develop modernization strategies for legacy and cloud-based data ecosystems.
• Design scalable architectures covering data ingestion, transformation, storage, orchestration, governance, security, and consumption.
• Review and challenge architecture decisions made by engineering and solution teams.
• Establish technology standards, engineering principles, and reference architectures.
• Identify technical risks and scalability challenges early in the program lifecycle.
• Provide hands-on technical guidance for complex architecture and engineering issues.

The Consulting Partner must maintain strong hands-on technical depth and be capable of working directly with engineering teams when required.

include:

• Reviewing and optimizing PySpark and Spark applications.
• Analyzing query and pipeline performance issues.
• Reviewing complex SQL transformations.
• Designing and reviewing dbt models, transformation frameworks, and testing strategies.
• Troubleshooting complex data processing and distributed computing issues.
• Reviewing data partitioning, clustering, file sizing, and storage optimization strategies.
• Providing technical direction on Delta Lake, table design, and data lifecycle management.
• Reviewing Airflow DAGs and orchestration architecture.
• Helping teams resolve critical production issues and performance bottlenecks.
• Conducting architecture and code reviews for high-risk or complex components.

The role does not require day-to-day individual contributor coding but requires the ability to get hands-on with technology when necessary and independently validate engineering decisions.

• Lead the technical direction of large-scale data engineering and modernization programs.
• Oversee programs involving hundreds or thousands of pipelines, jobs, tables, and datasets.
• Define technical migration strategies for:

o Legacy ETL to modern ELT

o Stored procedures to dbt

o Legacy Spark platforms to Databricks

o Cloud-to-cloud data platform migration

o Autosys and legacy scheduling to Airflow

o Data warehouse modernization

o Data lake and lakehouse implementation

• Define migration patterns, factory approaches, automation strategies, and validation frameworks.
• Establish technical governance across multiple engineering teams and workstreams.
• Resolve cross-platform and cross-domain technical dependencies.
• Define technical KPIs for scalability, reliability, performance, data quality, and cost efficiency.

The candidate should have strong practical experience across several of the following technologies:

• Apache Spark
• PySpark
• SQL
• Python
• Distributed data processing
• Data pipeline design and optimization

• Databricks
• Delta Lake
• Snowflake
• Data lakehouse architectures
• Data warehouse modernization

• dbt
• Apache Airflow
• Astronomer
• Enterprise workload orchestration
• CI/CD for data engineering

• AWS, Azure and/or GCP
• Cloud storage and data services
• IAM and security architecture
• Networking for data platforms
• Infrastructure as Code
• Terraform

• Unity Catalog
• Data catalog and metadata management
• Data lineage
• Data quality
• Data observability
• DataOps

• Act as a trusted technical advisor to CIOs, CDOs, CTOs, and senior Data and Analytics executives.
• Translate complex technical issues into business implications and executive decisions.

•