Data Engineering- SSEII

What You’ll Do:

Operationalize our Databricks Lakehouse: Develop and maintain a unified data platform using Azure Databricks, enabling efficient data storage, processing, and analysis across the organization. Build Data Pipelines: Design, develop, and maintain scalable data pipelines; establish API integrations for efficient data transfer. Implement ETL processes and ensure data integrity and quality.

Build Data Pipelines: Design, develop, and maintain scalable data pipelines; establish API integrations for efficient data transfer. Implement ETL processes and ensure data integrity and quality.

Deploy Data Technology: Utilize technologies such as Spark, Kafka, and Airflow to manage large-scale data processing. Test, monitor and troubleshoot systems to ensure smooth operation.

Collaborate: Work closely with data engineers, analysts, and scientists to understand their needs and optimize the performance of data workflows

What You’ll Bring:

Development Experience: 5+ years with Python, Scala, Java, or C#. Experience with Python is preferred.

Data Engineering Tools: 5+ years with Databricks, Snowflake, BigQuery, Apache Spark, HIVE, Hadoop, Cloudera, or RedShift. Experience with PySpark on Databricks is preferred. [TN1]

Data Architecture: Proficiency in high-performance data pipeline design and mastery of pipeline guarantees, such as Idempotency, At-Least-Once Processing, Exact-Once Processing, Fault Tolerance, Eventual Consistency, Streaming Consistency, Transactional Consistency, and Observability.

Containerization: Experience developing in a containerized environment like Docker, Rancher, or Kubernetes.

Data Orchestration: Experience with Apache Airflow (or similar tool) for orchestrating data processing jobs.

Education: Bachelor's degree in Computer Science (or related field) or equivalent combination of education and experience.

Educational Qualification and Experience:

Minimum of 15 years of formal education - Graduate / Post Graduate in Computer Science / Information Technology