L1 Data Engineer - Remote

Role overview

Key responsibilities

• Design, develop, and maintain scalable data pipelines and ETL/ELT workflows for business intelligence and analytics.
• Build and optimize ingestion processes with Azure Data Factory and Databricks while maintaining data quality.
• Transform large datasets with PySpark and Python using maintainable, performance-focused practices.
• Write and optimize SQL queries for reporting and data validation.
• Collaborate with data architects and senior engineers on organizationally aligned data models.
• Monitor and troubleshoot pipeline failures and data quality issues using root-cause analysis.
• Document pipelines, data dictionaries, and engineering standards.
• Explore tools and approaches that improve data infrastructure.

Requirements

• At least 3 years of professional experience in data engineering or a closely related role.
• Strong Python proficiency for data processing, transformation, and automation.
• Hands-on experience with Pandas, PySpark, Databricks, and Azure Data Factory.
• Working knowledge of Azure Synapse Analytics and Spark pool integration.
• Solid SQL skills, including query optimization and performance tuning.
• Knowledge of incremental loading, data lake architecture, Delta Lake, data governance, and cloud data security.

Preferred experience

• SQL Server migration experience, including schema conversion and data movement.
• Exposure to Terraform for Azure infrastructure provisioning and management.
• Familiarity with CI/CD practices for data engineering workflows.
• Experience with Delta Sharing or Lakehouse Federation concepts.

Certification requirement

• Candidates are expected to hold or be actively working toward the Databricks Certified Data Engineer Associate certification.
• The certification covers Lakehouse architecture, Spark SQL and PySpark ETL/ELT, incremental processing, structured streaming, orchestration, governance, and security.