Data Engineer

Data Engineer is responsible for creating efficient data pipelines, ensuring data quality, and implementing ETL (Extract, Transform, Load) processes to facilitate the seamless migration and deployment of data across systems. The Data Engineer works within the Agile frameworks to support the team's data-driven approach to application migration, collaborating closely with other team members to ensure data integrity and accessibility throughout the transition to cloud environments.

Key Responsibilities:

Design and develop scalable, efficient data architectures to support the migration process

Create data models and schemas that align with both source and target systems

Develop and maintain robust data pipelines for data generation, collection, and processing

Implement real-time and batch processing pipelines as required by the migration strategy

Design and implement ETL processes to migrate data between source and target systems

Develop scripts and workflows for data extraction, transformation, and loading

Develop and maintain data cleansing routines to address data inconsistencies

Create monitoring systems to track data quality metrics throughout the migration

Collaborate with the migration team to plan and execute data migration strategies

Provide expertise on data-related aspects of the migration process

Troubleshoot and resolve data-related issues during migration

Optimize data processing and storage for improved performance in cloud environments

Monitor and tune data pipelines for efficiency

Implement data security measures to protect sensitive information during migration

Ensure compliance with data protection regulations and organizational policies

Develop and maintain data access controls and audit trails