L1 Data Engineer - Remote

Overview

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL/ELT workflows to support business intelligence and analytics use cases.
• Build and optimize data ingestion processes using Azure Data Factory and Databricks, ensuring data quality and consistency across all layers of the data platform.
• Transform and process large datasets using PySpark and Python, applying best practices for performance and maintainability.
• Write and optimize complex SQL queries to support analytical reporting and data validation requirements.
• Collaborate with data architects and senior engineers to implement and maintain data models aligned with organizational standards.
• Monitor, troubleshoot, and resolve pipeline failures and data quality issues, applying root-cause analysis to prevent recurrence.
• Contribute to documentation of data pipelines, data dictionaries, and engineering standards.
• Support the team in exploring and evaluating new tools and approaches to continuously improve the data infrastructure.

Requirements

• 2+ years of professional experience in a Data Engineering or closely related role.
• Strong proficiency in Python for data processing, transformation, and automation tasks.
• Hands-on experience with Pandas for data manipulation and PySpark for distributed data processing.
• Practical experience with Databricks, including notebook development, clusters, and job orchestration.
• Experience building and managing data pipelines with Azure Data Factory.
• Working knowledge of Azure Synapse Analytics, particularly Spark pool integration.
• Solid SQL skills, including query writing, optimization, and performance tuning.
• Familiarity with data engineering principles including incremental loading, data lake architecture, and Delta Lake.
• Understanding of data governance and security concepts within a cloud data platform.

Nice To Have

• Experience with SQL Server migration projects, including schema conversion and data movement.
• Exposure to Terraform for Azure infrastructure provisioning and management.
• Familiarity with CI/CD practices applied to data engineering workflows.
• Experience with Delta Sharing or Lakehouse Federation concepts.

Certification Requirement

• Candidates are expected to hold or be actively working toward the Databricks Certified Data Engineer Associate certification. This certification validates foundational knowledge across the following domains:
• Databricks Lakehouse Platform architecture and capabilities
• ETL and ELT workflows using Spark SQL and PySpark
• Incremental data processing and structured streaming
• Production pipeline development and orchestration
• Data governance and security within the Databricks environment