Data Engineer

ETL/ELT Pipeline Development: Build, and maintain scalable data pipelines using AWS. Implement both batch and incremental load patterns for BI reporting and application data needs.

Real-Time Data Streaming: Develop and manage real-time data ingestion pipelines using Kafka. Ensure low-latency, fault-tolerant data flow for critical business workflows.

Workflow Orchestration: Build, schedule, and monitor end-to-end data workflows using Apache Airflow. Manage dependencies, retries, and alerting for production DAGs.

Data Warehouse Management: Administer and optimize Amazon Redshift clusters including schema design, query performance tuning, distribution/sort keys, and vacuuming to ensure high availability and cost efficiency.

Data Quality & Observability: Implement automated data quality checks at ingestion and transformation stages. Define validation rules, build alerting for anomalies and discrepancies, and establish SLAs to ensure stakeholders can trust the data they use.

API Integrations: Integrate third-party and internal REST APIs into data pipelines to pull operational and product data into the warehouse.

Cloud Cost Optimization: Monitor and right-size data processing and storage resources across S3, EMR, Redshift, EC2, and Lambda. Proactively identify inefficiencies and propose cost-saving improvements.

BI & Analytics Collaboration: Partner with the BI team to align data models, preprocessing logic, and Redshift schema design with reporting and dashboard needs.

Bachelor’s degree in Computer Science or a related quantitative field.

2+ years of experience working as a Data Engineer

Good proficiency in Python and SQL for data transformation and pipeline development

Hands-on experience with Apache Spark (PySpark) for large-scale data processing