Data Engineer (m/f/d)
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
Halian seeks a Data Engineer to design and build scalable data infrastructure for autonomous systems, handling large-scale multimodal sensor data. The role requires 5+ years of experience in production-grade data infrastructure, strong Python skills, and expertise in cloud storage and dataset versioning tools.
Key Skills for This Role
Responsibilities
- Design and manage high throughput on vehicle data recording pipelines, including topic selection, multi GB/s write pipelines, and efficient data formats (MCAP/rosbag2)
- Oversee on platform storage and ensure reliable data transfer to cloud environments with integrity checks
- Ensure timestamp accuracy and synchronization across recorded data
- Design and maintain scalable data lake architecture across S3, FSx/Lustre, and GCS, including data organization, regional placement, caching strategies, retention policies, data lineage, and cost optimization
- Build dataset pipelines that convert raw sensor data into structured, training ready datasets with accurate time alignment across modalities
- Implement robust dataset versioning and discovery processes using tools such as DVC, LakeFS, Deep Lake, and FiftyOne
- Contribute to defining efficient on disk dataset formats, focusing on write performance and optimized I/O for large scale training workloads
- Develop and manage annotation pipelines, including defining vendor handoff formats, ingesting labeled data, performing quality control, handling schema evolution, and supporting iterative dataset improvements
Requirements
- 5+ years of experience building production grade data infrastructure, ideally involving large scale multimodal or sensor data
- Strong proficiency in Python, with the ability to work with C++ for ROS2 and pipeline related tooling
- Hands on experience with cloud storage and distributed systems (S3, GCS, FSx, Lustre), including performance and cost optimization
- Experience with dataset versioning and ML data tools such as DVC, LakeFS, Deep Lake, FiftyOne, or similar platforms
Full Job Posting
Role Overview
- This role focuses on designing and building scalable data infrastructure to support advanced autonomous systems.
- The position is responsible for transforming large scale multimodal sensor data into high quality, structured datasets that are ready for downstream processing and machine learning workflows.
- The work involves establishing foundational systems and architectural decisions that will support long term scalability, including how data is recorded, ingested, stored, versioned, labeled, and served.
- The environment handles hundreds of terabytes of data generated from LiDAR, cameras, IMU, GPS, and radar across multiple platforms.
Key Responsibilities
- On vehicle data recording pipeline: Design and manage high throughput recording systems, including topic selection, multi GB/s write pipelines, and efficient data formats (MCAP/rosbag2). Oversee on platform storage and ensure reliable data transfer to cloud environments with integrity checks. Ensure
- Data lake architecture: Design and maintain scalable storage solutions across S3, FSx/Lustre, and GCS. Define data organization, regional placement, caching strategies, retention policies, data lineage, and cost optimization.
- Dataset pipeline development: Build pipelines that convert raw sensor data into structured, training ready datasets. Ensure accurate time alignment across modalities, including ego pose, calibration metadata, and scenario tagging.
- Versioning and dataset management: Implement robust dataset versioning and discovery processes. Evaluate and deploy tools such as DVC, LakeFS, Deep Lake, and FiftyOne, ensuring datasets are reproducible, traceable, and easily accessible.
- Dataset format design: Contribute to defining efficient on disk dataset formats, focusing on write performance and optimized I/O for large scale training workloads.
- Annotation workflows: Develop and manage annotation pipelines, including defining vendor handoff formats, ingesting labeled data, performing quality control, handling schema evolution, and supporting iterative dataset improvements.
Required Experience
- 5+ years of experience building production grade data infrastructure, ideally involving large scale multimodal or sensor data (e.g., robotics, autonomous systems, geospatial, or scientific domains)
- Strong proficiency in Python, with the ability to work with C++ for ROS2 and pipeline related tooling
- Hands on experience with cloud storage and distributed systems (S3, GCS, FSx, Lustre), including performance and cost optimization
- Experience with dataset versioning and ML data tools such as DVC, LakeFS, Deep Lake, FiftyOne, or similar platforms
Preferred Qualifications
- Background in autonomous systems or mobile platforms, particularly in complex or unstructured environments
- Experience working with large scale annotation workflows and external labeling providers
- Familiarity with distributed training approaches (e.g., DDP, FSDP) to support efficient collaboration with machine learning infrastructure
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Halian | Managed Services, Recruitment Agency & Contract Staffing
Finance Data Analytics & Automation Consultant (m/f/d)
Dubai, UAE
A six-month, full-time project assignment for a hands-on Finance Data Analytics & Automation Consultant in Dubai. You will partner with the Group Finance Director to build an automated, data-led, AI-enabled reporting env
AI Business Analyst (m/f/d)
Abu Dhabi Emirate, UAE
A leading organization is seeking a Technical Business Systems Analyst to bridge business and technology. The role involves translating stakeholder requirements into technical deliverables, working with development and Q
Senior Frontend Designer (m/f/d)
Abu Dhabi Emirate, UAE
We are seeking a Senior Frontend Designer to design, build, and maintain modern, responsive web applications. The role combines deep frontend expertise with backend contributions using Node.js and Python. Requires 6+ yea
Conversational AI Engineer
Abu Dhabi Emirate, UAE
Halian is hiring a Conversational AI Engineer for a 1-year extendable contract in Abu Dhabi. The role requires 5+ years of Python experience, expertise in Prompt Engineering and RAG, and experience with Azure OpenAI Serv
Engineer - Navigation (m/f/d)
Abu Dhabi Emirate, UAE
Halian is seeking a Navigation Engineer to develop autonomous navigation capabilities for Unmanned Ground Vehicles (UGVs). The role involves designing planning and control algorithms, mathematical modeling, and integrati
Document Controller (m/f/d)
Abu Dhabi Emirate, UAE
Halian is seeking a Document Controller for a sustainable urban development project in Abu Dhabi. The role involves managing document control systems, ensuring compliance with standards, and supporting project teams. Can
Finance Manager UAE (m/f/d)
Abu Dhabi Emirate, UAE
Our client is seeking a Finance Manager to lead the finance function of the UAE entity, ensuring accurate financial reporting, statutory compliance, and effective treasury management. The role requires 8-12 years of expe
Engineer - Navigation (m/f/d)
Abu Dhabi Emirate, UAE
Halian is seeking a Navigation Engineer to develop autonomous navigation capabilities for Unmanned Ground Vehicles (UGVs) in Abu Dhabi. The role involves designing planning and control algorithms, mathematical modeling,
Finance Data Analytics & Automation Consultant (m/f/d)
Dubai, UAE
AI Business Analyst (m/f/d)
Abu Dhabi Emirate, UAE
Senior Frontend Designer (m/f/d)
Abu Dhabi Emirate, UAE
Conversational AI Engineer
Abu Dhabi Emirate, UAE
Engineer - Navigation (m/f/d)
Abu Dhabi Emirate, UAE
Document Controller (m/f/d)
Abu Dhabi Emirate, UAE
Finance Manager UAE (m/f/d)
Abu Dhabi Emirate, UAE
Engineer - Navigation (m/f/d)
Abu Dhabi Emirate, UAE